Adding more Beta releases to the train

In March of 2011 we shipped Firefox 4 and moved to a rapid release with 6 weeks on each of Nightly, Aurora, and Beta channels prior to shipping a new major version of Firefox Desktop and Mobile to our users. Both Nightly and Aurora channels were getting builds & updates nightly (breakage notwithstanding) while Beta builds were still a highly managed, hands-on release product that shipped once per week, giving 6 builds in all unless there were additional last-minute landings (typically critical security or 3rd party plugin/addon issues) requiring a beta 7 or, rarely, 8 prior to building our release candidate for that version.

Go to build by or before Tuesday EOD Pacific time, builds would be pushed to beta channel as soon as QA signed off which could be Friday morning or sometimes Thursday afternoons if done early.

This is the model we followed up until Firefox 23. Starting in Firefox 15 we had the ability to perform silent, background updating which meant that we could push more updates to releases without causing update fatigue. Release Management, Release Engineering, QA, Stability, Support hashed out what it would take to move to a system where Beta builds are done on a nightly, automated manner. We dubbed this a Rapid Beta model and as work from all teams has been done toward that goal we have managed to get a handle on where the bottlenecks are which impeding the complete automation of pushing out the most recent Beta code to our 10 million Beta users.

The reason it is to our advantage to get more builds to Beta users is because at 1/10th of our general release population, the faster we can get fixes (especially crash fixes or speculative fixes for compatibility and addon/plugin breakage) to our users, the sooner we can collect much-needed data that can verify the quality of our impending final build. With the previous model, fixes missing a beta train meant that much more risk was added to the landing and typically we throttled the landing of all but the most serious security and usability patches back after the 4th beta meaning sometimes developers (and release managers) would be forced to make more pressured decisions about whether something could make a release or have to wait 8 more weeks to be in the next train.

QA did work to pare down on the manual testing needed for sign-off, Release Engineering put together a fabulous Ship-It web interface that Release Management could use to request builds in a more hands-off way to make the processes around starting & monitoring a new beta build much less time intensive. Socorro work was done to make it possible to match crash data to build IDs so that we could technically support nightly Beta builds and see stability data in useful ways. Once all this work was in place we took a leap of faith and started releasing twice as many Beta builds in weeks 2-5 of the cycle for Firefox 23.

First and last week still have one beta, weeks 2-5 have two builds per week where one is built on Monday shipping by Wednesday and the other build starts Thursday and ships by end of day Friday.

This new model has had two full releases now, Firefox 23 & 24. The feedback so far has been quite positive. Release Engineering has been minimally called upon when the shipping app interface hit glitches, but those are mostly ironed out now. QA is turning around their sign off of Firefox Desktop within approximately 24 hours and according to them their bug fix verification rates are going up with this new model in part because the smaller changes per Beta allow them to focus more. They’ve also had an intern and have had their remote testers team gain additional resources, but the switch to more frequent Betas has apparently gone quite smoothly for them. From a Release Management perspective, the tracking & landing of fixes on Beta is going much better since we now have less panic & stress on landings at the beginning of each week. With one Beta getting kicked off on Mondays we start the week with something to start evaluating mid-week and then we continue to pick up fixes as developers start their week in order to get another build for feedback gathered over the weekend.

We're moving away from spikes of landings near the end of the Beta cycle now that we have more Betas for people to land in. — We’re moving away from spikes of landings near the end of the Beta cycle now that we have more Betas for people to land in.

Though the data is a little rough right now (I’m dreaming of a pushlog DB), the numbers so far look like we’re doing a good job of spreading out the landings over the course of the cycle, still tapering off at the end:

Landings are more evenly spread out in a week.

While at the same time, our overall tracking average remains stable and our tracked bugs fixed rate has been holding over 90% per release for the past 3 releases:

Along with these improvements to getting features, regression & crash fixes to our users sooner with more automation and hands-off processes, we’ve been getting a lot out of the fact that we now have people who are full time sheriffs of the tree. Ryan VanderMeulen and Ed Morley are doing a lot of the heavy lifting keeping uplifts in order and landing frequently as well as monitoring the trees for breakage. Having managed trees, as well as team trees for active development is likely responsible for our tracking+/fix ratio on mozilla-central improving over time.

Finally, what’s most important from this experiment and what we consider to be the biggest win so far is that this new beta model helps release drivers over the whole cycle make decisions about uplifts with less concern about timing, and more focus on overall risk to product. Having more Beta builds means not having to make rash decisions because of scarcity. We will continue to collect data and monitor our progress as well as work towards automated, nightly Beta builds since that would get us crash feedback on a more granular level but for now I see this current progress as a huge step forward for the stability and quality of our releases. Neither of the last two releases had to be followed by dot releases for anything we could have prevented. Our Beta audience size holds strong, confirming that background updates are doing their job. Next up we’ll be looking at potentially moving to a slightly longer, and overlapping Beta cycle while shortening time on Aurora – but that’s another post for another time.

Contribution opportunity: Early Feedback Community Release Manager

I’ve been in Release Management for 1.8 years now and in that time we’ve grown from one overworked Release Manager to a team of 4 where we can start to split out responsibilities, cover more ground on a particular channel, and also…breathe a bit. With some of the team moving focus over to Firefox OS, we’ve opened up a great opportunity for a Mozillian to help Release Management drive Firefox Desktop & Mobile releases.

We’re looking for someone committed to learning the deepest, darkest secrets of release management who has a few hours a week consistently available to work with us by helping gather early feedback on our Nightly channel (aka mozilla-central or ‘trunk’). This very fabulous volunteer would get mentoring on tools, process, and build up awareness of risk needed for shipping software to 400 million users, starting at the earliest stage in development. On our Nightly/trunk channel there can be over 3000 changes in the 6 week development cycle and you’d be the primary person calling out potentially critical issues so they are less likely to cause pain to the user-facing release channels with larger audiences.

A long time back, in a post about developing community IT positions, mrz recalled a post where I stated that to have successful integration of community volunteers with paid staff in an organization there has to be time dedicated to working with that community member that is included in an employees hours so that the experience can be positive for both parties. It can’t just be “off the side of the desk” for the employee because that creates the risk of burnt out which can lead to communication irregularities with the volunteer and make them feel unneeded. For this community release manager position I will be able to put my time where my mouth is and dedicate hours in my week to actively shape and guide this community Release Manager in order to ensure they get the skills needed while we get the quality improvements in our product.

So here goes with an “official” call for help, come get in on the excitement with us.

You

Are familiar and interested in distributed development tools (version control, bug tracker) typically used in an open source project of size (remember when I said 400 million users? Ya, it’s not a small code base)
Want to learn (or already know) how to identify critical issues in a pool of bugs filed against a code base that branches every 6 weeks
Have worked in open source, or are extremely enthusiastic about learning how to do things in the open with a very diverse, global community of passionate contributors
Can demonstrate facility with public communications (do you blog, tweet, have a presence online with an audience?)
Will be part of the team that drives what goes in to final Firefox releases
Learn to coordinate across functional teams (security, support, engineering, quality assurance, marketing, localization)
Have an opportunity to develop tools & work with us to improve existing release processes and build your portfolio/resume

We

Mentor and guide your learning in how to ship a massive, open source software project under a brand that’s comparable to major for-profit technology companies (read: we’re competitive but we’re doing it for different end goals)
Teach you how to triage bugs and work with engineers to uncover issues and develop your intuition and decision making skills when weighing security/stability concerns with what’s best for our users
On-site time with Mozillians outside of Summits & work weeks – access to engineers, project managers, and other functional teams – get real world experience in how to work cross-functionally
Invitations to local work weeks where you can learn how to take leadership on ways to improve pre-release quality and stability that improve our Firefox Desktop/Mobile releases
provide references, t-shirts, and sometimes cupcakes 🙂

I’ll be posting this around and looking to chat with people either in person (if you’re in the Bay Area) or over vidyo. The best part is you can be anywhere in the world – we’ll figure out how to work with your schedule to ensure you get the guidance and mentoring you’re looking for.

Look forward to hearing from you! Let’s roll up our sleeves and make Firefox even better for our users!

Planning a Summit is hard, let’s go shopping

Hello fellow Mozillians (and curious onlookers)!

In a couple of days you will, en-masse, get your first peek at the high level agenda of the 3 day MegaMozillaFest you’ll be cannon-balling into this coming October. As someone who feels very fortunate to have been part of the planning processes (more multiple threads than you can shake a stick at) I thought I’d take some time during my layover en-route to Toronto and fill you in on what got us here in the hopes that:

I can shine light on how much hard work and sincerity went into the organization so that participants can get a lot out of this experience
You can see how many different voices and teams come together to plan an event at this scale
You’ll become as excited as I am (or close) for the journey we’re about to go on together this fall

It started, for me, back in June when 70 ‘delegates’ were brought together at the Mozilla Paris office to begin the conversations that were intended to bubble up the big issues and tensions that our community needs to address at Summit.

Wait. It started before that.

For delegates going to the Paris Planning Assembly, we were asked to go out and interview people in our ‘functional areas’ and query them about the story of how they got engaged with Mozilla, what they perceived our challenges and opportunities to be, and where they see Mozilla going in the next 10-15 years. I reached out to our Homozilla list, WoMoz, and also to the Firefox Team – a nice, diverse group of people to try and drag up some feedback to bring.

Sidebar: We often get asked to seek feedback from others in preparation and most recently I did this for a TRIBE workshop. From doing those interviews in person I now plan to commit to doing all sorts of info-gatherings in person/vidyo rather than over email. It’s too easy to send out an email blast and then just pick up what comes back to you. I get so much more from talking with a person face-to-face (video chat counts) and as you’ll see later in the post about Summit planning, it’s important to go a little further trying to encourage feedback than just a one-way missive over email that can be easily ignored or missed.

I got less feedback from people than I would want or expect. It’s a part of our culture to be cycling rapidly on our work and that results in a general air of being overwhelmed and too busy to do anything out of the usual routine so I largely attribute the low response rate to that. For my part I could have framed the questions differently, reached out to individual people instead of group lists, held a vidyo ‘office hours’ to talk with people in person, and followed up more (ie: nagging, a RelMan’s #1 skill).

Minimal feedback aside, I went to Paris feeling very much like I could bring a certain voice to the discussions as someone who wears many hats in our organization and who’s been involved with the project since 2006. Also, as a former unpaid contributor, intern, then full time hire I can imagine some of the realities of what it’s like in the shoes of many Mozillians.

I’ve already written a bit about what happened at the Paris Summit Planning and how it impacted me so I’ll sum up by sharing that when we left Paris the “what’s next?” action was that we each put our names to one of 5 groups where we could be called upon later for ideas/session planning/accountability for that area: People, Process, Product, Strategy, Purpose

Those groups then got turned into three track themes and were assigned two track leads who would work with Steering Committee members on refining the sessions for each as well as with the other delegates from Paris who had signed up to be accountable for those topics. The entire breakdown of this process is handily in a wiki page, but here’s the high level of who’s involved so you know who I’ve been working with for the past month or so:

Purpose & Strategy – Track Leads: myself and Michelle Thorne, Steering Committee: Mitchell Baker and Mark Surman
Products & Technology – Track Leads: Dave Herman and David Ascher, Steering Committee: Jay Sullivan and Brendan Eich
People & Process – Track Leads: Michael Coates and Zbigniew Braniecki (Gandalf), Steering Committee: Debbie Cohen and Jim Cook

The above listed people, along with Dino Anderson and Dia Bondi (and then a whole host of people behind the scenes like Kate, Mardi, and outside contractors) shepherded us through the last month while we did 3 iterations on an agenda in order to get the results you’ll all see this Tuesday. We held a TON of meetings with each other, in our small Track groups, and with our body of delegates from Paris. In our group, Purpose and Strategy, we kept circling back to the Nature of Mozilla (NoM) and how to best transmit that message so it can be held as the core of every Mozillian’s understanding of who we are, why we are, and what we do. We’d have to ask ourselves: Do our sessions have NoM aspects to them? If not, this isn’t the time. The very core of what makes us Mozillian is the key to this experience because it’s at the heart of how we will move into the future with our new projects as well as continuing to properly steer our existing ships in the waters of the open web.

During this process of multiple hour-long or more meetings each week with some serious thinkers & planners at Mozilla, several things happened for me:

I worried sometimes that we had become too narrow, that as such a small group we weren’t getting enough input, yet at the same time I saw new, and very engaged people at the table taking on leadership roles and bringing up strong points. I had to trust that this part of the planning process (similar to with the Paris planning) was for the best and that we’d get the right pieces cobbled together – always with the larger group who will attend & participate in the final product foremost in our minds.
It was reinforced for me many times over how it is for me to participate in meetings that are over 30 minutes long. I’m really going to need to work on that for my own professional development and opportunities in the future. I’m most definitely open to suggestions on how I can ‘game’ my own self to be better at meetings. In TRIBE we learned about the forms of listening and I’ve been aware of/practiced active listening for 2 decades but there’s something about meetings – the group factor, the agendas, getting off topic. Perhaps some training in facilitation might help understand this area better.
At the beginning of our meetings Dino asked all the Track Leads to speak a little about how we work best and I stated that I always prefer *doing* and am at my best with a list of clear steps to take. While I still enjoy talking/dreaming big, at the end of a rousing brainstorming I need that list of “now what?” and it had better be clear what is expected from me next. This planning process was definitely done in a way that met & exceeded my needs in that area. I applaud all the people who are at the top of the accountability chain for the Summit on their skillful communications and guidance throughout the last few months.
I felt incredibly fortunate to be a part of the creation of this next milestone in our shared Mozilla experience. We were *creating* a journey for (almost) all Mozillians to share. Having these touchstones within a project is so important. Being at the table to dream up and then turn into actuality the plan for complete engagement, alignment, and inspiration of what will get us all to the next billion web users is a heady task and I truly believe we’re very much on the right track to providing a creative and engaging space in October that will change all of your lives in good ways.

At the end of our last agenda review, I was filled with excitement about our schedule: Science & Culture Fairs, lots of Open Sessions, something called ‘Speedgeeking’ which I suspect will be like lightning talks, and then the overall story we’re going to learn about but also build into over 3 days. I’m sad that I’m missing almost 2/3 of Summit due to a prior commitment to the Grace Hopper Conference Open Source Day but I’ll join many of you on Saturday evening in Toronto and go deep Sunday so I can get the most out of it.

I hope this explanation of the process helps you understand how many people put in a lot of time to create what you’ll be participating in. Please join us with an open heart to the goals of this gathering – whichever location you end up in – know that we brought in as many of your voices as we could, that we want more than anything for you to get as excited about Mozilla’s future as we’ve been while dreaming about how we’re all going to build it together, aligned and re-energized. See you in October!