Contribution opportunity: Early Feedback Community Release Manager

I’ve been in Release Management for 1.8 years now and in that time we’ve grown from one overworked Release Manager to a team of 4 where we can start to split out responsibilities, cover more ground on a particular channel, and also…breathe a bit. With some of the team moving focus over to Firefox OS, we’ve opened up a great opportunity for a Mozillian to help Release Management drive Firefox Desktop & Mobile releases.

We’re looking for someone committed to learning the deepest, darkest secrets of release management who has a few hours a week consistently available to work with us by helping gather early feedback on our Nightly channel (aka mozilla-central or ‘trunk’).  This very fabulous volunteer would get mentoring on tools, process, and build up awareness of risk needed for shipping software to 400 million users, starting at the earliest stage in development. On our Nightly/trunk channel there can be over 3000 changes in the 6 week development cycle and you’d be the primary person calling out potentially critical issues so they are less likely to cause pain to the user-facing release channels with larger audiences.

A long time back, in a post about developing community IT positions, mrz recalled a post where I stated that to have successful integration of community volunteers with paid staff in an organization there has to be time dedicated to working with that community member that is included in an employees hours so that the experience can be positive for both parties.  It can’t just be “off the side of the desk” for the employee because that creates the risk of burnt out which can lead to communication irregularities with the volunteer and make them feel unneeded.  For this community release manager position I will be able to put my time where my mouth is and dedicate hours in my week to actively shape and guide this community Release Manager in order to ensure they get the skills needed while we get the quality improvements in our product.

So here goes with an “official” call for help, come get in on the excitement with us.


  • Are familiar and interested in distributed development tools (version control, bug tracker) typically used in an open source project of size (remember when I said 400 million users? Ya, it’s not a small code base)
  • Want to learn (or already know) how to identify critical issues in a pool of bugs filed against a code base that branches every 6 weeks
  • Have worked in open source, or are extremely enthusiastic about learning how to do things in the open with a very diverse, global community of passionate contributors
  • Can demonstrate facility with public communications (do you blog, tweet, have a presence online with an audience?)
  • Will be part of the team that drives what goes in to final Firefox releases
  • Learn to coordinate across functional teams (security, support, engineering, quality assurance, marketing, localization)
  • Have an opportunity to develop tools & work with us to improve existing release processes and build your portfolio/resume


  • Mentor and guide your learning in how to ship a massive, open source software project under a brand that’s comparable to major for-profit technology companies (read: we’re competitive but we’re doing it for different end goals)
  • Teach you how to triage bugs and work with engineers to uncover issues and develop your intuition and decision making skills when weighing security/stability concerns with what’s best for our users
  • On-site time with Mozillians outside of Summits & work weeks – access to engineers, project managers, and other functional teams – get real world experience in how to work cross-functionally
  • Invitations to local work weeks where you can learn how to take leadership on ways to improve pre-release quality and stability that improve our Firefox Desktop/Mobile releases
  • provide references, t-shirts, and sometimes cupcakes 🙂

I’ll be posting this around and looking to chat with people either in person (if you’re in the Bay Area) or over vidyo. The best part is you can be anywhere in the world – we’ll figure out how to work with your schedule to ensure you get the guidance and mentoring you’re looking for.

Look forward to hearing from you! Let’s roll up our sleeves and make Firefox even better for our users!



Release-Mgmt: My First Beta from the ‘other’ side

Hello and welcome to my continued documentation off my learning curve in Release Management, something I’ve now been working at for 6 weeks.  Last time I was reeling from the new-to-me meeting/email/bugmail firehose.  Now I’ve got that more under control, having created many more filters and folders in Thunderbird as well as having a chance to do more work on the automated tracking emails script.

To continue to spread out the knowledge and tasks for release managers across more than just one Alex, last week I ran my first beta (Firefox 12 beta 4) and now I’m going to tell you what that involved because the bugzilla API is down.


Monday is about getting the queries down as much as possible.  This means making sure anything with Tracking?, approval-mozilla{beta,aurora,esr10}? is triaged and nudged further towards its final destination on the trains. We’re also needing to watch out at this point for riskier fixes, things that really need some bake time with actual users need to be landed for a beta4 since beta5 is more for low-risk regression back-outs and security fixes that need to land just in time for beta 6 (what is usually re-built as the appropriately branded final release).  At this point in the week there might be about 80-90 bugs we need to get sorted and at the end of the day only about 6 bugs were in the ‘really want this in beta 4’ list.


For this particular beta, we had been asked if it would be possible to give the ‘go’ to build earlier than usual because there was a fairly popular holiday weekend coming up (Easter) and our QA lead for this release was in Canada where Good Friday is a statutory holiday. The QA lead asked if we could go to build earlier so that a Thursday release of the beta would be possible. We confirmed with RelEng that this was do-able and agreed to do our best to get the ‘go’ out at an earlier time.  The plan was for everything to be landed by 2:30pm (Pacific) in order to have a changeset ready to fire off to Release Engineering by 5pm.

Going through the ‘burn list’ from Monday (6 bugs) mostly entailed tracking down people to land patches. There has to be a cutoff time for landings since it takes about 4 hours to get all the builds and tests for a push to the hg repo to report back completely. Note that improvements to build times are being worked on, case in point: faster Mac builds (newer hardware and using OS X 10.7)  take ~2.5hours off the normal Mac build times.

For one of the bugs we weren’t able to reach the dev and a volunteer committer found that the patch didn’t apply cleanly so that bug had to miss the train.  The others got in and I sent the ‘go’ to build Firefox 12 beta 4 at approximately 5:45pm (45 minutes later than desired).  All the results weren’t in yet for this changeset but I wanted to get a shot at making the request for earlier builds to QA so I took the risk that the builds/tests would be OK and we’d be already building when we confirmed that.  Had the builds/tests *not* turned out we’d have to scrap the chance at moving up the release window and one of our QA leads would have worked on a holiday to meet our ‘normal’ Friday release window.  So I took the leap (and this ended up being fine, though I wouldn’t do this again without good reason as it was a stressful call to make and it’s not a good practice to get into).

Right after the ‘go’ email was sent went down and we lost 3 hours of build time.  This is not a normal result of giving the ‘go’ to build.  It was probably just because Hal was new to doing releases and I was new to running a beta so at least one thing had to come along and shake our confidence.


There’s not much (beta-running) to do on Wednesday except wait for Desktop & Mobile QA to do their thing.


Mobile & Desktop QA send their results out – either signing off on the builds/updates or calling out issues. At this point QA signed off so at that point I could request the Release Engineer to push the updates to the beta channel (and upload a new beta apk to the Google Play Store). An hour or so later both QA leads signed off on the updates for the release that’s now live to our beta users.  After that, there’s just some product details to change for our websites to include beta 4.  We don’t do release notes per-beta which is good to know for when I run a beta 1.


Normally we’d be doing the push to beta channel on a Friday, so what would have been different was:

  • getting everything landed to mozilla-beta could have gone until later in the day
  • the ‘go’ to build email wouldn’t have resulted in immediate builds, they could start on Wednesday at the beginning of the Release Engineer’s day
  • QA needs two full nights of testing, so we’d get sign offs from QA on Friday morning instead – hence the push to beta channel (and store) on Friday afternoon

Nothing too crazy for my first beta.  I think beta 4 is a good one to start on – it’s not the “OMG last call!!1!” beta before a release and it’s not beta 1 where any fallout from our merge of mozilla-aurora -> mozilla-beta shake loose. Having done a beta release now,  I have a much more complete mental map of how the 6 week release cycle plays out for Release Management:

Multiply by the above by 6, sprinkle extra bug & meeting communication cycles to weeks 1 and 6, throw in twelve channel meetings, approx 30 more iterations of various queries triage to keep our tracking lists up to date and to know what’s really needing attention vs. what’s taking care of itself.

A ton of email/irc/automated notifications all with the goal of keeping tracked bugs moving forward and you’ve got your 6 week result: A fabulous new Firefox release.

Thanks for reading, more about automatic emails & wiki updates soon.

My first three weeks in Release Management

Three weeks and four days ago a request was floated out to me. Would I consider helping the release management team for a little bit? You see, we’ve had this unfortunate ‘trend’ in Release Management at Mozilla. Whenever a new hire is brought in to join in the fun of release management, the former team member would see their opening and often quite quickly head off to another gig. This 1 to 1 to 1 to 1 ‘team’ size leads to a single release manager taking on an incredible load and working alone, to boot. These two factors might very well lead someone to throw in the towel, no matter how awesome Mozilla and its mission and when you only have one release manager, that’s a terrible edge to ride. So, back to the question: Would I be willing to get in there and help out, attempt to lighten the load, learn how it’s done and let our 6-months-in release manager, Alex, perhaps start to ease off his 100 hour weeks?

Answer: Gladly. I can honestly say there is _nothing_ more motivating to me than being asked to help someone, especially if I think I can actually do it. I mean, I would certainly try to help someone beat off a streak of tigers but I would die pretty fast regardless of my high level of ‘motivation’, and being dead isn’t too helpful is it? In this case, being that I’m very familiar with how releases get out the door, I figured I had a good head start and could pick up the rest as we went along.

Three weeks ago I began a journey into learning how exactly the “other side” lives. Until then I had been blissfully ignorant of most things related to getting our product ready to ship. Sure, I remember before Firefox 4 went out and Shaver was promising people he would squeeze oranges for them if squeezed oranges would help them _get_stuff_done_ but mostly I was unaware of what went on at all those meetings, how bugs get chosen for different releases, and how we get a product to the point where a Sam/Beltzner/LegNeato/Akeybl could say to Release Engineering “go to build on Firefox Version N, build #1”.

No longer am I blind. Bonus? Now I know what the people I’ve known for 5 years _actually_do_.

Here’s what I have discovered so far:

  • Bug queries are my new best friends. We spend a LOT of time looking at various triangulations of bug data – approval required for {beta,aurora,esr10,1.9.2}? are we tracking it for {11,12,13}+? is it a security bug that needs to get into blocker? topcrash? regression? I now have at least 4 or 5 bug list tabs open at any given time.
  • There’s a lot of communication involved in getting a bug to the finish line. Comment in the bug, wait for response, no response? Send an email. Email response? Great, now back into the bug with you for more history. No response? Ping in IRC. We will do whatever it takes to get a status update, find out if we should really be tracking something, discover blockers, ask about security risks of landing, or otherwise get the information needed to make the most informed decision about including/excluding a patch from a particular release. In the past couple of weeks, when I wasn’t in meetings or looking at bug lists, I built on Christian‘s bztools a couple of scripts that allow us to enter a bugzilla query, sort the bugs into buckets by manager and then send out a single email to each manager of the bugs’ assignees asking them to take a look at what we’re tracking for Firefox Next. Let’s start those discussions early and keep the bugs alive with whatever steps are being taken next.
  • Meetings are much more plentiful now. Two Firefox related meetings per week, two channel meetings, one ESR/1.9.2 triage, and a couple other 1×1 meetings to do RelMan planning. Add to that my 3x/week meetings with Marc to continue project managing Autoland and my week’s windows for writing code are significantly diminished. We have been trying to brainstorm ways of structuring meetings, calendaring, automating, anything to help reduce meetings/make them more effective. Watch for more changes to roll out as we continue to iterate on what’s working and eliminate what’s not. This seems to be the area we should focus on improving more than anything. It’s also the area where I am weakest, having a background in feminist collective process but not much in the way of boardroom and software production meeting styles. However, I am positive we can keep trying to improve one thing per meeting (like setting up a template or using the fabulous new bugzilla extension for our wiki) and keep chipping our way to being a lean, action-focused meeting machine.

My teammate Aki and I were talking almost every evening during the first week. He’d be curious as to how it was going ‘over there’. On the first day I reported back how crazy it seemed and how we spent so much time on bug queries and asking developers to talk with us about how things were going. He said “it sounds like you need three people, one per release branch”. A couple night later, I again filled him in on how things were going and about the email nag tool and about ESR and he said “now it seems like a team of 7 would be more ideal. One per branch (now including ESR), two on tools, one on vacation – then they all rotate”. I think it’s hilarious, and currently seems far-fetched that this team could ever get that big. It sure would be nice though. There’s some interesting problems to solve here. In fact, today in a meeting with Axel and Armen about l10n & releasing we pretended engineering effort was not short in supply and imagined what a fully automated release could look like.

Total hand-waving of what we could do automatically: tracking bug queries of release blockers, landing approved patches and watching for results, requesting l10n milestones for a release, sending the go to build when the last blocker is landed and the tree is green on that build, requesting a push to mirrors once QA signs off.

There’s so much fun to be had here 🙂

My very first contribution in week one was to get us a ‘release-mgmt@’ email so that we could start to act (and be perceived of ) as a team. Seems like a small thing, but as we continue to use it, I hope that we’ll keep building a sense that there is more than just one person nagging you to get your bug landed/nominated/tracked/backed out. I hope it will start to feel like there’s a team, encouraging you, trying to help you whenever possible, and working to keep the trains moving on time.