Tagged: beta

Adding more Beta releases to the train

In March of 2011 we shipped Firefox 4 and moved to a rapid release with 6 weeks on each of Nightly, Aurora, and Beta channels prior to shipping a new major version of Firefox Desktop and Mobile to our users. Both Nightly and Aurora channels were getting builds & updates nightly (breakage notwithstanding) while Beta builds were still a highly managed, hands-on release product that shipped once per week, giving 6 builds in all unless there were additional last-minute landings  (typically critical security or 3rd party plugin/addon issues) requiring a beta 7 or, rarely, 8 prior to building our release candidate for that version.

Go to build by or before Tuesday EOD Pacific time, builds would be pushed to beta channel as soon as QA signed off which could be Friday morning or sometimes Thursday afternoons if done early.
Go to build by or before Tuesday EOD Pacific time, builds would be pushed to beta channel as soon as QA signed off which could be Friday morning or sometimes Thursday afternoons if done early.

This is the model we followed up until Firefox 23.  Starting in Firefox 15 we had the ability to perform silent, background updating which meant that we could push more updates to releases without causing update fatigue. Release Management, Release Engineering, QA, Stability, Support hashed out what it would take to move to a system where Beta builds are done on a nightly, automated manner.  We dubbed this a Rapid Beta model and as work from all teams has been done toward that goal we have managed to get a handle on where the bottlenecks are which impeding the complete automation of pushing out the most recent Beta code to our 10 million Beta users.

The reason it is to our advantage to get more builds to Beta users is because at 1/10th of our general release population, the faster we can get fixes (especially crash fixes or speculative fixes for compatibility and addon/plugin breakage) to our users, the sooner we can collect much-needed data that can verify the quality of our impending final build.  With the previous model, fixes missing a beta train meant that much more risk was added to the landing and typically we throttled the landing of all but the most serious security and usability patches back after the 4th beta meaning sometimes developers (and release managers) would be forced to make more pressured decisions about whether something could make a release or have to wait 8 more weeks to be in the next train.

QA did work to pare down on the manual testing needed for sign-off, Release Engineering put together a fabulous Ship-It web interface that Release Management could use to request builds in a more hands-off way to make the processes around starting & monitoring a new beta build much less time intensive.  Socorro work was done to make it possible to match crash data to build IDs so that we could technically support nightly Beta builds and see stability data in useful ways. Once all this work was in place we took a leap of faith and started releasing twice as many Beta builds in weeks 2-5 of the cycle for Firefox 23.

    First and last week still have one beta, weeks 2-5 have two builds per week where one is built on Monday shipping by Wednesday and the other build starts Thursday and ships by end of day Friday.
First and last week still have one beta, weeks 2-5 have two builds per week where one is built on Monday shipping by Wednesday and the other build starts Thursday and ships by end of day Friday.

This new model has had two full releases now, Firefox 23 & 24.  The feedback so far has been quite positive.  Release Engineering has been minimally called upon when the shipping app interface hit glitches, but those are mostly ironed out now.  QA is turning around their sign off of Firefox Desktop within approximately 24 hours and according to them their bug fix verification rates are going up with this new model in part because the smaller changes per Beta allow them to focus more.  They’ve also had an intern and have had their remote testers team gain additional resources, but the switch to more frequent Betas has apparently gone quite smoothly for them.  From a Release Management perspective, the tracking & landing of fixes on Beta is going much better since we now have less panic & stress on landings at the beginning of each week.  With one Beta getting kicked off on Mondays we start the week with something to start evaluating mid-week and then we continue to pick up fixes as developers start their week in order to get another build for feedback gathered over the weekend.

We're moving away from spikes of landings near the end of the Beta cycle now that we have more Betas for people to land in.
We’re moving away from spikes of landings near the end of the Beta cycle now that we have more Betas for people to land in.

Though the data is a little rough right now (I’m dreaming of a pushlog DB), the numbers so far look like we’re doing a good job of spreading out the landings over the course of the cycle, still tapering off at the end:

Landings are more evenly spread out in a week.
Landings are more evenly spread out in a week.

While at the same time, our overall tracking average remains stable and our tracked bugs fixed rate has been holding over 90% per release for the past 3 releases:

Tracking bugs fixed over unfixed Screen Shot 2013-10-17 at 5.55.12 PMScreen Shot 2013-10-17 at 5.57.07 PM Tracked to fixed percentage

 

 

 

 

 

 

 

 

Along with these improvements to getting features, regression & crash fixes to our users sooner with more automation and hands-off processes, we’ve been getting a lot out of the fact that we now have people who are full time sheriffs of the tree.  Ryan VanderMeulen and Ed Morley are doing a lot of the heavy lifting keeping uplifts in order and landing frequently as well as monitoring the trees for breakage.  Having managed trees, as well as team trees for active development is likely responsible for our tracking+/fix ratio on mozilla-central improving over time.

Finally, what’s most important from this experiment and what we consider to be the biggest win so far is that this new beta model helps release drivers over the whole cycle make decisions about uplifts with less concern about timing, and more focus on overall risk to product. Having more Beta builds means not having to make rash decisions because of scarcity.  We will continue to collect data and monitor our progress as well as work towards automated, nightly Beta builds since that would get us crash feedback on a more granular level but for now I see this current progress as a huge step forward for the stability and quality of our releases. Neither of the last two releases had to be followed by dot releases for anything we could have prevented.  Our Beta audience size holds strong, confirming that background updates are doing their job.  Next up we’ll be looking at potentially moving to a slightly longer, and overlapping Beta cycle while shortening time on Aurora – but that’s another post for another time.

 

Release-Mgmt: My First Beta from the ‘other’ side

Hello and welcome to my continued documentation off my learning curve in Release Management, something I’ve now been working at for 6 weeks.  Last time I was reeling from the new-to-me meeting/email/bugmail firehose.  Now I’ve got that more under control, having created many more filters and folders in Thunderbird as well as having a chance to do more work on the automated tracking emails script.

To continue to spread out the knowledge and tasks for release managers across more than just one Alex, last week I ran my first beta (Firefox 12 beta 4) and now I’m going to tell you what that involved because the bugzilla API is down.

Monday

Monday is about getting the queries down as much as possible.  This means making sure anything with Tracking?, approval-mozilla{beta,aurora,esr10}? is triaged and nudged further towards its final destination on the trains. We’re also needing to watch out at this point for riskier fixes, things that really need some bake time with actual users need to be landed for a beta4 since beta5 is more for low-risk regression back-outs and security fixes that need to land just in time for beta 6 (what is usually re-built as the appropriately branded final release).  At this point in the week there might be about 80-90 bugs we need to get sorted and at the end of the day only about 6 bugs were in the ‘really want this in beta 4′ list.

Tuesday

For this particular beta, we had been asked if it would be possible to give the ‘go’ to build earlier than usual because there was a fairly popular holiday weekend coming up (Easter) and our QA lead for this release was in Canada where Good Friday is a statutory holiday. The QA lead asked if we could go to build earlier so that a Thursday release of the beta would be possible. We confirmed with RelEng that this was do-able and agreed to do our best to get the ‘go’ out at an earlier time.  The plan was for everything to be landed by 2:30pm (Pacific) in order to have a changeset ready to fire off to Release Engineering by 5pm.

Going through the ‘burn list’ from Monday (6 bugs) mostly entailed tracking down people to land patches. There has to be a cutoff time for landings since it takes about 4 hours to get all the builds and tests for a push to the hg repo to report back completely. Note that improvements to build times are being worked on, case in point: faster Mac builds (newer hardware and using OS X 10.7)  take ~2.5hours off the normal Mac build times.

For one of the bugs we weren’t able to reach the dev and a volunteer committer found that the patch didn’t apply cleanly so that bug had to miss the train.  The others got in and I sent the ‘go’ to build Firefox 12 beta 4 at approximately 5:45pm (45 minutes later than desired).  All the results weren’t in yet for this changeset but I wanted to get a shot at making the request for earlier builds to QA so I took the risk that the builds/tests would be OK and we’d be already building when we confirmed that.  Had the builds/tests *not* turned out we’d have to scrap the chance at moving up the release window and one of our QA leads would have worked on a holiday to meet our ‘normal’ Friday release window.  So I took the leap (and this ended up being fine, though I wouldn’t do this again without good reason as it was a stressful call to make and it’s not a good practice to get into).

Right after the ‘go’ email was sent hg.mozilla.org went down and we lost 3 hours of build time.  This is not a normal result of giving the ‘go’ to build.  It was probably just because Hal was new to doing releases and I was new to running a beta so at least one thing had to come along and shake our confidence.

Wednesday

There’s not much (beta-running) to do on Wednesday except wait for Desktop & Mobile QA to do their thing.

Thursday

Mobile & Desktop QA send their results out – either signing off on the builds/updates or calling out issues. At this point QA signed off so at that point I could request the Release Engineer to push the updates to the beta channel (and upload a new beta apk to the Google Play Store). An hour or so later both QA leads signed off on the updates for the release that’s now live to our beta users.  After that, there’s just some product details to change for our websites to include beta 4.  We don’t do release notes per-beta which is good to know for when I run a beta 1.

Friday

Normally we’d be doing the push to beta channel on a Friday, so what would have been different was:

  • getting everything landed to mozilla-beta could have gone until later in the day
  • the ‘go’ to build email wouldn’t have resulted in immediate builds, they could start on Wednesday at the beginning of the Release Engineer’s day
  • QA needs two full nights of testing, so we’d get sign offs from QA on Friday morning instead – hence the push to beta channel (and store) on Friday afternoon

Nothing too crazy for my first beta.  I think beta 4 is a good one to start on – it’s not the “OMG last call!!1!” beta before a release and it’s not beta 1 where any fallout from our merge of mozilla-aurora -> mozilla-beta shake loose. Having done a beta release now,  I have a much more complete mental map of how the 6 week release cycle plays out for Release Management:

Multiply by the above by 6, sprinkle extra bug & meeting communication cycles to weeks 1 and 6, throw in twelve channel meetings, approx 30 more iterations of various queries triage to keep our tracking lists up to date and to know what’s really needing attention vs. what’s taking care of itself.

A ton of email/irc/automated notifications all with the goal of keeping tracked bugs moving forward and you’ve got your 6 week result: A fabulous new Firefox release.

Thanks for reading, more about automatic emails & wiki updates soon.