Tagged: info-sharing

Adding more Beta releases to the train

In March of 2011 we shipped Firefox 4 and moved to a rapid release with 6 weeks on each of Nightly, Aurora, and Beta channels prior to shipping a new major version of Firefox Desktop and Mobile to our users. Both Nightly and Aurora channels were getting builds & updates nightly (breakage notwithstanding) while Beta builds were still a highly managed, hands-on release product that shipped once per week, giving 6 builds in all unless there were additional last-minute landings  (typically critical security or 3rd party plugin/addon issues) requiring a beta 7 or, rarely, 8 prior to building our release candidate for that version.

Go to build by or before Tuesday EOD Pacific time, builds would be pushed to beta channel as soon as QA signed off which could be Friday morning or sometimes Thursday afternoons if done early.
Go to build by or before Tuesday EOD Pacific time, builds would be pushed to beta channel as soon as QA signed off which could be Friday morning or sometimes Thursday afternoons if done early.

This is the model we followed up until Firefox 23.  Starting in Firefox 15 we had the ability to perform silent, background updating which meant that we could push more updates to releases without causing update fatigue. Release Management, Release Engineering, QA, Stability, Support hashed out what it would take to move to a system where Beta builds are done on a nightly, automated manner.  We dubbed this a Rapid Beta model and as work from all teams has been done toward that goal we have managed to get a handle on where the bottlenecks are which impeding the complete automation of pushing out the most recent Beta code to our 10 million Beta users.

The reason it is to our advantage to get more builds to Beta users is because at 1/10th of our general release population, the faster we can get fixes (especially crash fixes or speculative fixes for compatibility and addon/plugin breakage) to our users, the sooner we can collect much-needed data that can verify the quality of our impending final build.  With the previous model, fixes missing a beta train meant that much more risk was added to the landing and typically we throttled the landing of all but the most serious security and usability patches back after the 4th beta meaning sometimes developers (and release managers) would be forced to make more pressured decisions about whether something could make a release or have to wait 8 more weeks to be in the next train.

QA did work to pare down on the manual testing needed for sign-off, Release Engineering put together a fabulous Ship-It web interface that Release Management could use to request builds in a more hands-off way to make the processes around starting & monitoring a new beta build much less time intensive.  Socorro work was done to make it possible to match crash data to build IDs so that we could technically support nightly Beta builds and see stability data in useful ways. Once all this work was in place we took a leap of faith and started releasing twice as many Beta builds in weeks 2-5 of the cycle for Firefox 23.

    First and last week still have one beta, weeks 2-5 have two builds per week where one is built on Monday shipping by Wednesday and the other build starts Thursday and ships by end of day Friday.
First and last week still have one beta, weeks 2-5 have two builds per week where one is built on Monday shipping by Wednesday and the other build starts Thursday and ships by end of day Friday.

This new model has had two full releases now, Firefox 23 & 24.  The feedback so far has been quite positive.  Release Engineering has been minimally called upon when the shipping app interface hit glitches, but those are mostly ironed out now.  QA is turning around their sign off of Firefox Desktop within approximately 24 hours and according to them their bug fix verification rates are going up with this new model in part because the smaller changes per Beta allow them to focus more.  They’ve also had an intern and have had their remote testers team gain additional resources, but the switch to more frequent Betas has apparently gone quite smoothly for them.  From a Release Management perspective, the tracking & landing of fixes on Beta is going much better since we now have less panic & stress on landings at the beginning of each week.  With one Beta getting kicked off on Mondays we start the week with something to start evaluating mid-week and then we continue to pick up fixes as developers start their week in order to get another build for feedback gathered over the weekend.

We're moving away from spikes of landings near the end of the Beta cycle now that we have more Betas for people to land in.
We’re moving away from spikes of landings near the end of the Beta cycle now that we have more Betas for people to land in.

Though the data is a little rough right now (I’m dreaming of a pushlog DB), the numbers so far look like we’re doing a good job of spreading out the landings over the course of the cycle, still tapering off at the end:

Landings are more evenly spread out in a week.
Landings are more evenly spread out in a week.

While at the same time, our overall tracking average remains stable and our tracked bugs fixed rate has been holding over 90% per release for the past 3 releases:

Tracking bugs fixed over unfixed Screen Shot 2013-10-17 at 5.55.12 PMScreen Shot 2013-10-17 at 5.57.07 PM Tracked to fixed percentage

 

 

 

 

 

 

 

 

Along with these improvements to getting features, regression & crash fixes to our users sooner with more automation and hands-off processes, we’ve been getting a lot out of the fact that we now have people who are full time sheriffs of the tree.  Ryan VanderMeulen and Ed Morley are doing a lot of the heavy lifting keeping uplifts in order and landing frequently as well as monitoring the trees for breakage.  Having managed trees, as well as team trees for active development is likely responsible for our tracking+/fix ratio on mozilla-central improving over time.

Finally, what’s most important from this experiment and what we consider to be the biggest win so far is that this new beta model helps release drivers over the whole cycle make decisions about uplifts with less concern about timing, and more focus on overall risk to product. Having more Beta builds means not having to make rash decisions because of scarcity.  We will continue to collect data and monitor our progress as well as work towards automated, nightly Beta builds since that would get us crash feedback on a more granular level but for now I see this current progress as a huge step forward for the stability and quality of our releases. Neither of the last two releases had to be followed by dot releases for anything we could have prevented.  Our Beta audience size holds strong, confirming that background updates are doing their job.  Next up we’ll be looking at potentially moving to a slightly longer, and overlapping Beta cycle while shortening time on Aurora – but that’s another post for another time.

 

Planning a Summit is hard, let’s go shopping

Hello fellow Mozillians (and curious onlookers)!

In a couple of days you will, en-masse, get your first peek at the high level agenda of the 3 day MegaMozillaFest you’ll be cannon-balling into this coming October. As someone who feels very fortunate to have been part of the planning processes (more multiple threads than you can shake a stick at) I thought I’d take some time during my layover en-route to Toronto and fill you in on what got us here in the hopes that:

  • I can shine light on how much hard work and sincerity went into the organization so that participants can get a lot out of this experience
  • You can see how many different voices and teams come together to plan an event at this scale
  • You’ll become as excited as I am (or close) for the journey we’re about to go on together this fall

It started, for me, back in June when 70 ‘delegates’ were brought together at the Mozilla Paris office to begin the conversations that were intended to bubble up the big issues and tensions that our community needs to address at Summit.

Wait.  It started before that.

For delegates going to the Paris Planning Assembly, we were asked to go out and interview people in our ‘functional areas’ and query them about the story of how they got engaged with Mozilla, what they perceived our challenges and opportunities to be, and where they see Mozilla going in the next 10-15 years.  I reached out to our Homozilla list, WoMoz, and also to the Firefox Team – a nice, diverse group of people to try and drag up some feedback to bring.

Sidebar:  We often get asked to seek feedback from others in preparation and most recently I did this for a TRIBE workshop. From doing those interviews in person I now plan to commit to doing all sorts of info-gatherings in person/vidyo rather than over email.  It’s too easy to send out an email blast and then just pick up what comes back to you. I get so much more from talking with a person face-to-face (video chat counts) and as you’ll see later in the post about Summit planning, it’s important to go a little further trying to encourage feedback than just a one-way missive over email that can be easily ignored or missed.

 

I got less feedback from people than I would want or expect.  It’s a part of our culture to be cycling rapidly on our work and that results in a general air of being overwhelmed and too busy to do anything out of the usual routine so I largely attribute the low response rate to that. For my part I could have framed the questions differently, reached out to individual people instead of group lists, held a vidyo ‘office hours’ to talk with people in person, and followed up more (ie: nagging, a RelMan’s #1 skill).

Minimal feedback aside, I went to Paris feeling very much like I could bring a certain voice to the discussions as someone who wears many hats in our organization and who’s been involved with the project since 2006.  Also, as a former unpaid contributor, intern, then full time hire I can imagine some of the realities of what it’s like in the shoes of many  Mozillians.

I’ve already written a bit about what happened at the Paris Summit Planning and how it impacted me so I’ll sum up by sharing that when we left Paris the “what’s next?” action was that we each put our names to one of 5 groups where we could be called upon later for ideas/session planning/accountability for that area: People, Process, Product, Strategy, Purpose

Those groups then got turned into three track themes and were assigned two track leads who would work with Steering Committee members on refining the sessions for each as well as with the other delegates from Paris who had signed up to be accountable for those topics.  The entire breakdown of this process is handily in a wiki page, but here’s the high level of who’s involved so you know who I’ve been working with for the past month or so:

  • Purpose & Strategy – Track Leads: myself and Michelle Thorne, Steering Committee: Mitchell Baker and Mark Surman
  • Products & Technology – Track Leads: Dave Herman and David Ascher, Steering Committee: Jay Sullivan and Brendan Eich
  • People & Process – Track Leads: Michael Coates and Zbigniew Braniecki (Gandalf), Steering Committee: Debbie Cohen and Jim Cook

The above listed people, along with Dino Anderson and Dia Bondi (and then a whole host of people behind the scenes like Kate, Mardi, and outside contractors) shepherded us through the last month while we did 3 iterations on an agenda in order to get the results you’ll all see this Tuesday.  We held a TON of meetings with each other, in our small Track groups, and with our body of delegates from Paris. In our group, Purpose and Strategy, we kept circling back to the Nature of Mozilla (NoM) and how to best transmit that message so it can be held as the core of every Mozillian’s understanding of who we are, why we are, and what we do.  We’d have to ask ourselves: Do our sessions have NoM aspects to them?  If not, this isn’t the time.  The very core of what makes us Mozillian is the key to this experience because it’s at the heart of how we will move into the future with our new projects as well as continuing to properly steer our existing ships in the waters of the open web.

During this process of multiple hour-long or more meetings each week with some serious thinkers & planners at Mozilla, several things happened for me:

  • I worried sometimes that we had become too narrow, that as such a small group we weren’t getting enough input, yet at the same time I saw new, and very engaged people at the table taking on leadership roles and bringing up strong points. I had to trust that this part of the planning process (similar to with the Paris planning) was for the best and that we’d get the right pieces cobbled together – always with the larger group who will attend & participate in the final product foremost in our minds.
  • It was reinforced for me many times over how it is for me to participate in meetings that are over 30 minutes long. I’m really going to need to work on that for my own professional development and opportunities in the future.  I’m most definitely open to suggestions on how I can ‘game’ my own self to be better at meetings. In TRIBE we learned about the forms of listening and I’ve been aware of/practiced active listening for 2 decades but there’s something about meetings – the group factor, the agendas, getting off topic.  Perhaps some training in facilitation might help understand this area better.
  • At the beginning of our meetings Dino asked all the Track Leads to speak a little about how we work best and I stated that I always prefer *doing* and am at my best with a list of clear steps to take. While I still enjoy talking/dreaming big, at the end of a rousing brainstorming I need that list of “now what?” and it had better be clear what is expected from me next. This planning process was definitely done in a way that met & exceeded my needs in that area. I applaud all the people who are at the top of the accountability chain for the Summit on their skillful communications and guidance throughout the last few months.
  • I felt incredibly fortunate to be a part of the creation of this next milestone in our shared Mozilla experience.  We were *creating* a journey for (almost) all Mozillians to share.  Having these touchstones within a project is so important. Being at the table to dream up and then turn into actuality the plan for complete engagement, alignment, and inspiration of what will get us all to the next billion web users is a heady task and I truly believe we’re very much on the right track to providing a creative and engaging space in October that will change all of your lives in good ways.

At the end of our last agenda review, I was filled with excitement about our schedule:  Science & Culture Fairs, lots of Open Sessions, something called ‘Speedgeeking’ which I suspect will be like lightning talks, and then the overall story we’re going to learn about but also build into over 3 days. I’m sad that I’m missing almost 2/3 of Summit due to a prior commitment to the Grace Hopper Conference Open Source Day but I’ll join many of you on Saturday evening in Toronto and go deep Sunday so I can get the most out of it.

I hope this explanation of the process helps you understand how many people put in a lot of time to create what you’ll be participating in.  Please join us with an open heart to the goals of this gathering – whichever location you end up in – know that we brought in as many of your voices as we could, that we want more than anything for you to get as excited about Mozilla’s future as we’ve been while dreaming about how we’re all going to build it together, aligned and re-energized.  See you in October!

 

Women Hacking Glass – First SF community meetup

I’ve created an event for the first meeting of Women Hacking Glass in SF at the Mozilla public space.

Since I posted in G+ a few weeks ago things got busy and I didn’t have time to lean on Google like I’d planned to ask for hardware but then a pair of Glass practically fell in my lap when a coworker decided he didn’t want to be an Explorer any more so I wrangled a ‘donation’ to get his Glass in order to use them for community hacking with other women in the Bay Area.  I’m curious to see how the first meetup goes – what will we be able to create?  What kinds of feedback will we provide to the GDK developers who are working on the first version of a release?  What kinds of barriers will we hit with Mirror API?  I look forward to learning about everyone’s hopes and dreams for this exciting hardware and finding ways to hack our way to making them a reality.

Copy from the event invite:

Are you interested in learning how to make apps for Google Glass?  Don’t have the access to the hardware?

Come out to Mozilla SF and meet with other Glass Hacking gals to experiment with Android Studio, creating simple apps, getting access to Mirror API, and trying out your hacks on an actual pair of Glass that will be made available during WHG meetups for testing on.  Since there are very few people out there with the hardware, and few of those early adopter/explorers are women let’s work together to increase the numbers of women getting in on the ground floor for development (as well as being able to provide feedback to Google GDK developers) on this revolutionary new hardware.

There is a small (non-refundable) fee to prevent no-shows from taking up space – all money generated from this event will be donated to Mozilla Foundation via http://www.mozilla.org/donate

Prepare ahead of time:

* Have a google account

* Read https://developers.google.com/glass/quickstart/index and do as much of the pre-installation of tools/IDE that you can

* Think about your first app and what you want to learn to build

* Dream big, show up

For people who are interested in applying pressure to Google and showing them there are women interested in developing for Glass (the current Glass Developers group is easily 95% male) – go to http://www.google.com/glass/start/how-to-get-one/ and submit your request anyway, even though they say the waitlist is full.  My coworker can’t be the only person returning his pair and I trust Google will open more spots when they see a lot of interest.

Where Are My $project-branch Nightly Builds?

Did you know that we don’t build a fresh nightly on a branch unless there’s fresh code?  Well, now you do!  In the interest of saving even _more_ resources and network bandwidth so that we can accommodate even  _more_ project branches we have added this little bit of logic to our nightly build scheduler.  It makes sense, right?  I mean, why have a nightly that is just like last night’s nightly?

In other news, Ben just added 4 more twigs (aka Disposable Project Branches) for side project work and one of them is still available for booking.