My partner has been granted a very well deserved year-long sabbatical starting in late May and that got me thinking about what my days would look like if I had a small sabbatical. Fortunately Mozilla’s offices will be closing over the winter holidays so I’ll have about 2 weeks to enact this plan.
Run in the morning (if rained out, yoga) every other morning
Take the dogs for a 3-4 mile hike on non-run days, otherwise the usual 45 min loop
In March of 2011 we shipped Firefox 4 and moved to a rapid release with 6 weeks on each of Nightly, Aurora, and Beta channels prior to shipping a new major version of Firefox Desktop and Mobile to our users. Both Nightly and Aurora channels were getting builds & updates nightly (breakage notwithstanding) while Beta builds were still a highly managed, hands-on release product that shipped once per week, giving 6 builds in all unless there were additional last-minute landings (typically critical security or 3rd party plugin/addon issues) requiring a beta 7 or, rarely, 8 prior to building our release candidate for that version.
This is the model we followed up until Firefox 23. Starting in Firefox 15 we had the ability to perform silent, background updating which meant that we could push more updates to releases without causing update fatigue. Release Management, Release Engineering, QA, Stability, Support hashed out what it would take to move to a system where Beta builds are done on a nightly, automated manner. We dubbed this a Rapid Beta model and as work from all teams has been done toward that goal we have managed to get a handle on where the bottlenecks are which impeding the complete automation of pushing out the most recent Beta code to our 10 million Beta users.
The reason it is to our advantage to get more builds to Beta users is because at 1/10th of our general release population, the faster we can get fixes (especially crash fixes or speculative fixes for compatibility and addon/plugin breakage) to our users, the sooner we can collect much-needed data that can verify the quality of our impending final build. With the previous model, fixes missing a beta train meant that much more risk was added to the landing and typically we throttled the landing of all but the most serious security and usability patches back after the 4th beta meaning sometimes developers (and release managers) would be forced to make more pressured decisions about whether something could make a release or have to wait 8 more weeks to be in the next train.
QA did work to pare down on the manual testing needed for sign-off, Release Engineering put together a fabulous Ship-It web interface that Release Management could use to request builds in a more hands-off way to make the processes around starting & monitoring a new beta build much less time intensive. Socorro work was done to make it possible to match crash data to build IDs so that we could technically support nightly Beta builds and see stability data in useful ways. Once all this work was in place we took a leap of faith and started releasing twice as many Beta builds in weeks 2-5 of the cycle for Firefox 23.
This new model has had two full releases now, Firefox 23 & 24. The feedback so far has been quite positive. Release Engineering has been minimally called upon when the shipping app interface hit glitches, but those are mostly ironed out now. QA is turning around their sign off of Firefox Desktop within approximately 24 hours and according to them their bug fix verification rates are going up with this new model in part because the smaller changes per Beta allow them to focus more. They’ve also had an intern and have had their remote testers team gain additional resources, but the switch to more frequent Betas has apparently gone quite smoothly for them. From a Release Management perspective, the tracking & landing of fixes on Beta is going much better since we now have less panic & stress on landings at the beginning of each week. With one Beta getting kicked off on Mondays we start the week with something to start evaluating mid-week and then we continue to pick up fixes as developers start their week in order to get another build for feedback gathered over the weekend.
Though the data is a little rough right now (I’m dreaming of a pushlog DB), the numbers so far look like we’re doing a good job of spreading out the landings over the course of the cycle, still tapering off at the end:
While at the same time, our overall tracking average remains stable and our tracked bugs fixed rate has been holding over 90% per release for the past 3 releases:
Along with these improvements to getting features, regression & crash fixes to our users sooner with more automation and hands-off processes, we’ve been getting a lot out of the fact that we now have people who are full time sheriffs of the tree. Ryan VanderMeulen and Ed Morley are doing a lot of the heavy lifting keeping uplifts in order and landing frequently as well as monitoring the trees for breakage. Having managed trees, as well as team trees for active development is likely responsible for our tracking+/fix ratio on mozilla-central improving over time.
Finally, what’s most important from this experiment and what we consider to be the biggest win so far is that this new beta model helps release drivers over the whole cycle make decisions about uplifts with less concern about timing, and more focus on overall risk to product. Having more Beta builds means not having to make rash decisions because of scarcity. We will continue to collect data and monitor our progress as well as work towards automated, nightly Beta builds since that would get us crash feedback on a more granular level but for now I see this current progress as a huge step forward for the stability and quality of our releases. Neither of the last two releases had to be followed by dot releases for anything we could have prevented. Our Beta audience size holds strong, confirming that background updates are doing their job. Next up we’ll be looking at potentially moving to a slightly longer, and overlapping Beta cycle while shortening time on Aurora – but that’s another post for another time.
I’ve been in Release Management for 1.8 years now and in that time we’ve grown from one overworked Release Manager to a team of 4 where we can start to split out responsibilities, cover more ground on a particular channel, and also…breathe a bit. With some of the team moving focus over to Firefox OS, we’ve opened up a great opportunity for a Mozillian to help Release Management drive Firefox Desktop & Mobile releases.
We’re looking for someone committed to learning the deepest, darkest secrets of release management who has a few hours a week consistently available to work with us by helping gather early feedback on our Nightly channel (aka mozilla-central or ‘trunk’). This very fabulous volunteer would get mentoring on tools, process, and build up awareness of risk needed for shipping software to 400 million users, starting at the earliest stage in development. On our Nightly/trunk channel there can be over 3000 changes in the 6 week development cycle and you’d be the primary person calling out potentially critical issues so they are less likely to cause pain to the user-facing release channels with larger audiences.
A long time back, in a post about developing community IT positions, mrz recalled a post where I stated that to have successful integration of community volunteers with paid staff in an organization there has to be time dedicated to working with that community member that is included in an employees hours so that the experience can be positive for both parties. It can’t just be “off the side of the desk” for the employee because that creates the risk of burnt out which can lead to communication irregularities with the volunteer and make them feel unneeded. For this community release manager position I will be able to put my time where my mouth is and dedicate hours in my week to actively shape and guide this community Release Manager in order to ensure they get the skills needed while we get the quality improvements in our product.
So here goes with an “official” call for help, come get in on the excitement with us.
Are familiar and interested in distributed development tools (version control, bug tracker) typically used in an open source project of size (remember when I said 400 million users? Ya, it’s not a small code base)
Want to learn (or already know) how to identify critical issues in a pool of bugs filed against a code base that branches every 6 weeks
Have worked in open source, or are extremely enthusiastic about learning how to do things in the open with a very diverse, global community of passionate contributors
Can demonstrate facility with public communications (do you blog, tweet, have a presence online with an audience?)
Will be part of the team that drives what goes in to final Firefox releases
Learn to coordinate across functional teams (security, support, engineering, quality assurance, marketing, localization)
Have an opportunity to develop tools & work with us to improve existing release processes and build your portfolio/resume
Mentor and guide your learning in how to ship a massive, open source software project under a brand that’s comparable to major for-profit technology companies (read: we’re competitive but we’re doing it for different end goals)
Teach you how to triage bugs and work with engineers to uncover issues and develop your intuition and decision making skills when weighing security/stability concerns with what’s best for our users
On-site time with Mozillians outside of Summits & work weeks – access to engineers, project managers, and other functional teams – get real world experience in how to work cross-functionally
Invitations to local work weeks where you can learn how to take leadership on ways to improve pre-release quality and stability that improve our Firefox Desktop/Mobile releases
provide references, t-shirts, and sometimes cupcakes
I’ll be posting this around and looking to chat with people either in person (if you’re in the Bay Area) or over vidyo. The best part is you can be anywhere in the world – we’ll figure out how to work with your schedule to ensure you get the guidance and mentoring you’re looking for.
Look forward to hearing from you! Let’s roll up our sleeves and make Firefox even better for our users!
If anyone else at Mozilla has a project that can be done in 3 months time (or at least give the contributor a sense of accomplishment and get them very engaged as a Mozilla contributor) feel free to add a project (and a mentor) to the wiki. Applications are being accepted via GNOME until May 1st.
One of my favourite things about this program is that it allows someone to ‘intern’ with Mozilla without the requirement of being a student. If you can help tweet/share the project, that would be much appreciated too. This project has been growing exponentially every session and is making a significant impact to FOSS community and culture.
Thanks to the GNOME Outreach Program for Women, we’ve got ourselves an awesome January intern who will be doing her first Open Source contributions all the way from Australia.
She did a great job of showing one of the things release managers do over the six weeks a Firefox version is in Beta. The spikes in the above graph align with our constant triaging of tracking-firefox17? flags and how the number of bugs flagged for tracking decreases after the first few betas have shipped. When we get to beta 4 we’re starting to get more reserved about what we’re willing to track (it usually has to be pretty critical, or a low-risk fix to a many-user-facing issue).
This next graph shows us what we already know – but it’s very nice to SEE: our bugs tracked for a particular release continually go down over time, gradually. Remember, this is while new bugs are being added to tracking regularly, so the fact that the trend keeps going down helps us know we are staying on top of our work and that engineers are continuing to fix tracked bugs as we close in on a 6 week ship date.
Now that we know Lianne has got what it takes, we’re going to set her on a more ambitious project – to create an engineering dashboard both for individuals and for teams, that would give them this sort of info on demand. Want to see where you’re at (or where your team is at) on a particular version? The engineering dashboard could show you in priority sequence what should be top on your list and also what bugs your team has unassigned that are tracked and should be assigned pronto (or communicate to RelMan that the bug should not be tracked).
This will be a huge improvement over email nagging (don’t worry, that’s still going to be around for many more months) because it will give us some quick, visual cues about how we’re doing with Firefox priorities and then we can also keep these measurements over time to compare release-to-release what the temperature of a particular version was. We hope this will allow us to keep fine tuning and working towards more stable beta cycles as we move forward.
Lianne will be with us from January 2 to April 2, 2013 and in her first week she’ll be evaluating a bunch of existing dashboards we know about to see what the pros and cons of each are and to do reconn on the technologies and visualizations people use. We’ll use that to help us develop the v1.0 of this project’s deliverable and make sure it’s left in a state that RelMan intern 2.0 can pick up next summer.
Please comment if you have dashboards you like, you loathe, or you just want us to know about.