My first three weeks in Release Management

Three weeks and four days ago a request was floated out to me. Would I consider helping the release management team for a little bit? You see, we’ve had this unfortunate ‘trend’ in Release Management at Mozilla. Whenever a new hire is brought in to join in the fun of release management, the former team member would see their opening and often quite quickly head off to another gig. This 1 to 1 to 1 to 1 ‘team’ size leads to a single release manager taking on an incredible load and working alone, to boot. These two factors might very well lead someone to throw in the towel, no matter how awesome Mozilla and its mission and when you only have one release manager, that’s a terrible edge to ride. So, back to the question: Would I be willing to get in there and help out, attempt to lighten the load, learn how it’s done and let our 6-months-in release manager, Alex, perhaps start to ease off his 100 hour weeks?

Answer: Gladly. I can honestly say there is _nothing_ more motivating to me than being asked to help someone, especially if I think I can actually do it. I mean, I would certainly try to help someone beat off a streak of tigers but I would die pretty fast regardless of my high level of ‘motivation’, and being dead isn’t too helpful is it? In this case, being that I’m very familiar with how releases get out the door, I figured I had a good head start and could pick up the rest as we went along.

Three weeks ago I began a journey into learning how exactly the “other side” lives. Until then I had been blissfully ignorant of most things related to getting our product ready to ship. Sure, I remember before Firefox 4 went out and Shaver was promising people he would squeeze oranges for them if squeezed oranges would help them _get_stuff_done_ but mostly I was unaware of what went on at all those meetings, how bugs get chosen for different releases, and how we get a product to the point where a Sam/Beltzner/LegNeato/Akeybl could say to Release Engineering “go to build on Firefox Version N, build #1”.

No longer am I blind. Bonus? Now I know what the people I’ve known for 5 years _actually_do_.

Here’s what I have discovered so far:

• Bug queries are my new best friends. We spend a LOT of time looking at various triangulations of bug data – approval required for {beta,aurora,esr10,1.9.2}? are we tracking it for {11,12,13}+? is it a security bug that needs to get into 3.6.next? blocker? topcrash? regression? I now have at least 4 or 5 bug list tabs open at any given time.
• There’s a lot of communication involved in getting a bug to the finish line. Comment in the bug, wait for response, no response? Send an email. Email response? Great, now back into the bug with you for more history. No response? Ping in IRC. We will do whatever it takes to get a status update, find out if we should really be tracking something, discover blockers, ask about security risks of landing, or otherwise get the information needed to make the most informed decision about including/excluding a patch from a particular release. In the past couple of weeks, when I wasn’t in meetings or looking at bug lists, I built on Christian‘s bztools a couple of scripts that allow us to enter a bugzilla query, sort the bugs into buckets by manager and then send out a single email to each manager of the bugs’ assignees asking them to take a look at what we’re tracking for Firefox Next. Let’s start those discussions early and keep the bugs alive with whatever steps are being taken next.
• Meetings are much more plentiful now. Two Firefox related meetings per week, two channel meetings, one ESR/1.9.2 triage, and a couple other 1×1 meetings to do RelMan planning. Add to that my 3x/week meetings with Marc to continue project managing Autoland and my week’s windows for writing code are significantly diminished. We have been trying to brainstorm ways of structuring meetings, calendaring, automating, anything to help reduce meetings/make them more effective. Watch for more changes to roll out as we continue to iterate on what’s working and eliminate what’s not. This seems to be the area we should focus on improving more than anything. It’s also the area where I am weakest, having a background in feminist collective process but not much in the way of boardroom and software production meeting styles. However, I am positive we can keep trying to improve one thing per meeting (like setting up a template or using the fabulous new bugzilla extension for our wiki) and keep chipping our way to being a lean, action-focused meeting machine.

My teammate Aki and I were talking almost every evening during the first week. He’d be curious as to how it was going ‘over there’. On the first day I reported back how crazy it seemed and how we spent so much time on bug queries and asking developers to talk with us about how things were going. He said “it sounds like you need three people, one per release branch”. A couple night later, I again filled him in on how things were going and about the email nag tool and about ESR and he said “now it seems like a team of 7 would be more ideal. One per branch (now including ESR), two on tools, one on vacation – then they all rotate”. I think it’s hilarious, and currently seems far-fetched that this team could ever get that big. It sure would be nice though. There’s some interesting problems to solve here. In fact, today in a meeting with Axel and Armen about l10n & releasing we pretended engineering effort was not short in supply and imagined what a fully automated release could look like.

Total hand-waving of what we could do automatically: tracking bug queries of release blockers, landing approved patches and watching for results, requesting l10n milestones for a release, sending the go to build when the last blocker is landed and the tree is green on that build, requesting a push to mirrors once QA signs off.

There’s so much fun to be had here 🙂

My very first contribution in week one was to get us a ‘release-mgmt@’ email so that we could start to act (and be perceived of ) as a team. Seems like a small thing, but as we continue to use it, I hope that we’ll keep building a sense that there is more than just one person nagging you to get your bug landed/nominated/tracked/backed out. I hope it will start to feel like there’s a team, encouraging you, trying to help you whenever possible, and working to keep the trains moving on time.