Tagged: process

Adding more Beta releases to the train

In March of 2011 we shipped Firefox 4 and moved to a rapid release with 6 weeks on each of Nightly, Aurora, and Beta channels prior to shipping a new major version of Firefox Desktop and Mobile to our users. Both Nightly and Aurora channels were getting builds & updates nightly (breakage notwithstanding) while Beta builds were still a highly managed, hands-on release product that shipped once per week, giving 6 builds in all unless there were additional last-minute landings  (typically critical security or 3rd party plugin/addon issues) requiring a beta 7 or, rarely, 8 prior to building our release candidate for that version.

Go to build by or before Tuesday EOD Pacific time, builds would be pushed to beta channel as soon as QA signed off which could be Friday morning or sometimes Thursday afternoons if done early.
Go to build by or before Tuesday EOD Pacific time, builds would be pushed to beta channel as soon as QA signed off which could be Friday morning or sometimes Thursday afternoons if done early.

This is the model we followed up until Firefox 23.  Starting in Firefox 15 we had the ability to perform silent, background updating which meant that we could push more updates to releases without causing update fatigue. Release Management, Release Engineering, QA, Stability, Support hashed out what it would take to move to a system where Beta builds are done on a nightly, automated manner.  We dubbed this a Rapid Beta model and as work from all teams has been done toward that goal we have managed to get a handle on where the bottlenecks are which impeding the complete automation of pushing out the most recent Beta code to our 10 million Beta users.

The reason it is to our advantage to get more builds to Beta users is because at 1/10th of our general release population, the faster we can get fixes (especially crash fixes or speculative fixes for compatibility and addon/plugin breakage) to our users, the sooner we can collect much-needed data that can verify the quality of our impending final build.  With the previous model, fixes missing a beta train meant that much more risk was added to the landing and typically we throttled the landing of all but the most serious security and usability patches back after the 4th beta meaning sometimes developers (and release managers) would be forced to make more pressured decisions about whether something could make a release or have to wait 8 more weeks to be in the next train.

QA did work to pare down on the manual testing needed for sign-off, Release Engineering put together a fabulous Ship-It web interface that Release Management could use to request builds in a more hands-off way to make the processes around starting & monitoring a new beta build much less time intensive.  Socorro work was done to make it possible to match crash data to build IDs so that we could technically support nightly Beta builds and see stability data in useful ways. Once all this work was in place we took a leap of faith and started releasing twice as many Beta builds in weeks 2-5 of the cycle for Firefox 23.

    First and last week still have one beta, weeks 2-5 have two builds per week where one is built on Monday shipping by Wednesday and the other build starts Thursday and ships by end of day Friday.
First and last week still have one beta, weeks 2-5 have two builds per week where one is built on Monday shipping by Wednesday and the other build starts Thursday and ships by end of day Friday.

This new model has had two full releases now, Firefox 23 & 24.  The feedback so far has been quite positive.  Release Engineering has been minimally called upon when the shipping app interface hit glitches, but those are mostly ironed out now.  QA is turning around their sign off of Firefox Desktop within approximately 24 hours and according to them their bug fix verification rates are going up with this new model in part because the smaller changes per Beta allow them to focus more.  They’ve also had an intern and have had their remote testers team gain additional resources, but the switch to more frequent Betas has apparently gone quite smoothly for them.  From a Release Management perspective, the tracking & landing of fixes on Beta is going much better since we now have less panic & stress on landings at the beginning of each week.  With one Beta getting kicked off on Mondays we start the week with something to start evaluating mid-week and then we continue to pick up fixes as developers start their week in order to get another build for feedback gathered over the weekend.

We're moving away from spikes of landings near the end of the Beta cycle now that we have more Betas for people to land in.
We’re moving away from spikes of landings near the end of the Beta cycle now that we have more Betas for people to land in.

Though the data is a little rough right now (I’m dreaming of a pushlog DB), the numbers so far look like we’re doing a good job of spreading out the landings over the course of the cycle, still tapering off at the end:

Landings are more evenly spread out in a week.
Landings are more evenly spread out in a week.

While at the same time, our overall tracking average remains stable and our tracked bugs fixed rate has been holding over 90% per release for the past 3 releases:

Tracking bugs fixed over unfixed Screen Shot 2013-10-17 at 5.55.12 PMScreen Shot 2013-10-17 at 5.57.07 PM Tracked to fixed percentage

 

 

 

 

 

 

 

 

Along with these improvements to getting features, regression & crash fixes to our users sooner with more automation and hands-off processes, we’ve been getting a lot out of the fact that we now have people who are full time sheriffs of the tree.  Ryan VanderMeulen and Ed Morley are doing a lot of the heavy lifting keeping uplifts in order and landing frequently as well as monitoring the trees for breakage.  Having managed trees, as well as team trees for active development is likely responsible for our tracking+/fix ratio on mozilla-central improving over time.

Finally, what’s most important from this experiment and what we consider to be the biggest win so far is that this new beta model helps release drivers over the whole cycle make decisions about uplifts with less concern about timing, and more focus on overall risk to product. Having more Beta builds means not having to make rash decisions because of scarcity.  We will continue to collect data and monitor our progress as well as work towards automated, nightly Beta builds since that would get us crash feedback on a more granular level but for now I see this current progress as a huge step forward for the stability and quality of our releases. Neither of the last two releases had to be followed by dot releases for anything we could have prevented.  Our Beta audience size holds strong, confirming that background updates are doing their job.  Next up we’ll be looking at potentially moving to a slightly longer, and overlapping Beta cycle while shortening time on Aurora – but that’s another post for another time.

 

Contribution opportunity: Early Feedback Community Release Manager

I’ve been in Release Management for 1.8 years now and in that time we’ve grown from one overworked Release Manager to a team of 4 where we can start to split out responsibilities, cover more ground on a particular channel, and also…breathe a bit. With some of the team moving focus over to Firefox OS, we’ve opened up a great opportunity for a Mozillian to help Release Management drive Firefox Desktop & Mobile releases.

We’re looking for someone committed to learning the deepest, darkest secrets of release management who has a few hours a week consistently available to work with us by helping gather early feedback on our Nightly channel (aka mozilla-central or ‘trunk’).  This very fabulous volunteer would get mentoring on tools, process, and build up awareness of risk needed for shipping software to 400 million users, starting at the earliest stage in development. On our Nightly/trunk channel there can be over 3000 changes in the 6 week development cycle and you’d be the primary person calling out potentially critical issues so they are less likely to cause pain to the user-facing release channels with larger audiences.

A long time back, in a post about developing community IT positions, mrz recalled a post where I stated that to have successful integration of community volunteers with paid staff in an organization there has to be time dedicated to working with that community member that is included in an employees hours so that the experience can be positive for both parties.  It can’t just be “off the side of the desk” for the employee because that creates the risk of burnt out which can lead to communication irregularities with the volunteer and make them feel unneeded.  For this community release manager position I will be able to put my time where my mouth is and dedicate hours in my week to actively shape and guide this community Release Manager in order to ensure they get the skills needed while we get the quality improvements in our product.

So here goes with an “official” call for help, come get in on the excitement with us.

You

  • Are familiar and interested in distributed development tools (version control, bug tracker) typically used in an open source project of size (remember when I said 400 million users? Ya, it’s not a small code base)
  • Want to learn (or already know) how to identify critical issues in a pool of bugs filed against a code base that branches every 6 weeks
  • Have worked in open source, or are extremely enthusiastic about learning how to do things in the open with a very diverse, global community of passionate contributors
  • Can demonstrate facility with public communications (do you blog, tweet, have a presence online with an audience?)
  • Will be part of the team that drives what goes in to final Firefox releases
  • Learn to coordinate across functional teams (security, support, engineering, quality assurance, marketing, localization)
  • Have an opportunity to develop tools & work with us to improve existing release processes and build your portfolio/resume

We

  • Mentor and guide your learning in how to ship a massive, open source software project under a brand that’s comparable to major for-profit technology companies (read: we’re competitive but we’re doing it for different end goals)
  • Teach you how to triage bugs and work with engineers to uncover issues and develop your intuition and decision making skills when weighing security/stability concerns with what’s best for our users
  • On-site time with Mozillians outside of Summits & work weeks – access to engineers, project managers, and other functional teams – get real world experience in how to work cross-functionally
  • Invitations to local work weeks where you can learn how to take leadership on ways to improve pre-release quality and stability that improve our Firefox Desktop/Mobile releases
  • provide references, t-shirts, and sometimes cupcakes :)

I’ll be posting this around and looking to chat with people either in person (if you’re in the Bay Area) or over vidyo. The best part is you can be anywhere in the world – we’ll figure out how to work with your schedule to ensure you get the guidance and mentoring you’re looking for.

Look forward to hearing from you! Let’s roll up our sleeves and make Firefox even better for our users!

 

 

Planning a Summit is hard, let’s go shopping

Hello fellow Mozillians (and curious onlookers)!

In a couple of days you will, en-masse, get your first peek at the high level agenda of the 3 day MegaMozillaFest you’ll be cannon-balling into this coming October. As someone who feels very fortunate to have been part of the planning processes (more multiple threads than you can shake a stick at) I thought I’d take some time during my layover en-route to Toronto and fill you in on what got us here in the hopes that:

  • I can shine light on how much hard work and sincerity went into the organization so that participants can get a lot out of this experience
  • You can see how many different voices and teams come together to plan an event at this scale
  • You’ll become as excited as I am (or close) for the journey we’re about to go on together this fall

It started, for me, back in June when 70 ‘delegates’ were brought together at the Mozilla Paris office to begin the conversations that were intended to bubble up the big issues and tensions that our community needs to address at Summit.

Wait.  It started before that.

For delegates going to the Paris Planning Assembly, we were asked to go out and interview people in our ‘functional areas’ and query them about the story of how they got engaged with Mozilla, what they perceived our challenges and opportunities to be, and where they see Mozilla going in the next 10-15 years.  I reached out to our Homozilla list, WoMoz, and also to the Firefox Team – a nice, diverse group of people to try and drag up some feedback to bring.

Sidebar:  We often get asked to seek feedback from others in preparation and most recently I did this for a TRIBE workshop. From doing those interviews in person I now plan to commit to doing all sorts of info-gatherings in person/vidyo rather than over email.  It’s too easy to send out an email blast and then just pick up what comes back to you. I get so much more from talking with a person face-to-face (video chat counts) and as you’ll see later in the post about Summit planning, it’s important to go a little further trying to encourage feedback than just a one-way missive over email that can be easily ignored or missed.

 

I got less feedback from people than I would want or expect.  It’s a part of our culture to be cycling rapidly on our work and that results in a general air of being overwhelmed and too busy to do anything out of the usual routine so I largely attribute the low response rate to that. For my part I could have framed the questions differently, reached out to individual people instead of group lists, held a vidyo ‘office hours’ to talk with people in person, and followed up more (ie: nagging, a RelMan’s #1 skill).

Minimal feedback aside, I went to Paris feeling very much like I could bring a certain voice to the discussions as someone who wears many hats in our organization and who’s been involved with the project since 2006.  Also, as a former unpaid contributor, intern, then full time hire I can imagine some of the realities of what it’s like in the shoes of many  Mozillians.

I’ve already written a bit about what happened at the Paris Summit Planning and how it impacted me so I’ll sum up by sharing that when we left Paris the “what’s next?” action was that we each put our names to one of 5 groups where we could be called upon later for ideas/session planning/accountability for that area: People, Process, Product, Strategy, Purpose

Those groups then got turned into three track themes and were assigned two track leads who would work with Steering Committee members on refining the sessions for each as well as with the other delegates from Paris who had signed up to be accountable for those topics.  The entire breakdown of this process is handily in a wiki page, but here’s the high level of who’s involved so you know who I’ve been working with for the past month or so:

  • Purpose & Strategy – Track Leads: myself and Michelle Thorne, Steering Committee: Mitchell Baker and Mark Surman
  • Products & Technology – Track Leads: Dave Herman and David Ascher, Steering Committee: Jay Sullivan and Brendan Eich
  • People & Process – Track Leads: Michael Coates and Zbigniew Braniecki (Gandalf), Steering Committee: Debbie Cohen and Jim Cook

The above listed people, along with Dino Anderson and Dia Bondi (and then a whole host of people behind the scenes like Kate, Mardi, and outside contractors) shepherded us through the last month while we did 3 iterations on an agenda in order to get the results you’ll all see this Tuesday.  We held a TON of meetings with each other, in our small Track groups, and with our body of delegates from Paris. In our group, Purpose and Strategy, we kept circling back to the Nature of Mozilla (NoM) and how to best transmit that message so it can be held as the core of every Mozillian’s understanding of who we are, why we are, and what we do.  We’d have to ask ourselves: Do our sessions have NoM aspects to them?  If not, this isn’t the time.  The very core of what makes us Mozillian is the key to this experience because it’s at the heart of how we will move into the future with our new projects as well as continuing to properly steer our existing ships in the waters of the open web.

During this process of multiple hour-long or more meetings each week with some serious thinkers & planners at Mozilla, several things happened for me:

  • I worried sometimes that we had become too narrow, that as such a small group we weren’t getting enough input, yet at the same time I saw new, and very engaged people at the table taking on leadership roles and bringing up strong points. I had to trust that this part of the planning process (similar to with the Paris planning) was for the best and that we’d get the right pieces cobbled together – always with the larger group who will attend & participate in the final product foremost in our minds.
  • It was reinforced for me many times over how it is for me to participate in meetings that are over 30 minutes long. I’m really going to need to work on that for my own professional development and opportunities in the future.  I’m most definitely open to suggestions on how I can ‘game’ my own self to be better at meetings. In TRIBE we learned about the forms of listening and I’ve been aware of/practiced active listening for 2 decades but there’s something about meetings – the group factor, the agendas, getting off topic.  Perhaps some training in facilitation might help understand this area better.
  • At the beginning of our meetings Dino asked all the Track Leads to speak a little about how we work best and I stated that I always prefer *doing* and am at my best with a list of clear steps to take. While I still enjoy talking/dreaming big, at the end of a rousing brainstorming I need that list of “now what?” and it had better be clear what is expected from me next. This planning process was definitely done in a way that met & exceeded my needs in that area. I applaud all the people who are at the top of the accountability chain for the Summit on their skillful communications and guidance throughout the last few months.
  • I felt incredibly fortunate to be a part of the creation of this next milestone in our shared Mozilla experience.  We were *creating* a journey for (almost) all Mozillians to share.  Having these touchstones within a project is so important. Being at the table to dream up and then turn into actuality the plan for complete engagement, alignment, and inspiration of what will get us all to the next billion web users is a heady task and I truly believe we’re very much on the right track to providing a creative and engaging space in October that will change all of your lives in good ways.

At the end of our last agenda review, I was filled with excitement about our schedule:  Science & Culture Fairs, lots of Open Sessions, something called ‘Speedgeeking’ which I suspect will be like lightning talks, and then the overall story we’re going to learn about but also build into over 3 days. I’m sad that I’m missing almost 2/3 of Summit due to a prior commitment to the Grace Hopper Conference Open Source Day but I’ll join many of you on Saturday evening in Toronto and go deep Sunday so I can get the most out of it.

I hope this explanation of the process helps you understand how many people put in a lot of time to create what you’ll be participating in.  Please join us with an open heart to the goals of this gathering – whichever location you end up in – know that we brought in as many of your voices as we could, that we want more than anything for you to get as excited about Mozilla’s future as we’ve been while dreaming about how we’re all going to build it together, aligned and re-energized.  See you in October!

 

My Big Shift

This is the second in a series of blog posts that will summarize my experience and takeaways from the Mozilla Summit 2013 Planning Assembly that took place in our Paris, France office on June 14-17, 2013.

✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈

Stereotype:  something that may be true about some members of a group, but not all, and yet is applied to all members of the group without regard for the individual. While stereotype might be a scary word to some people, worry not – they are within your control to manipulate, eliminate, and examine thoroughly for holes in logic.

Debunk your internally held stereotypes, and not just once.

Unpacking a stereotype you’ve held for any length of time is not a ‘set it and forget it’ process.  It came as quite a shock to me this past weekend when I became aware of the deep levels of distrust I had been holding toward new hires within the last 2 years. These are people who were in the ranks of our explosive growth spurt* and when I examined this feeling I saw that I didn’t believe this new bunch of Mozillians could be as hooked, committed, and passionate as myself.

It dawned on me over the course of our intensive 2 days of plenary activities that I held this belief while still going day to day believing what I care most about is bring new people into the community.  And I do care about this, but I see now how my stereotyping might also be holding me back.  When I bring someone into the community or get a chance to passionately wax on how Mozilla has changed my life I like to think I can tell if they are also catching the fever. For me, it’s been such a good pairing of challenging technical work and new areas to practice social justice activism in, so I’ll evangelize to anyone who’s interested while also inquiring with them to discover what area of Mozilla contribution might be a good starting place for them. I expect to see a similar spark of what is possible, working with Mozilla, for their passion. When I’m able, I try to help them network and make connections within the project to help clear barriers to their goals and find a mentor.  Again, that’s based on my experience of coming on board with the help of a strong mentor as well as having social connections to leaders within the organization.

If it’s not apparent yet, I am trying to recreate for others what I saw as the perfect ‘hook’. I believe I have a strong set of core values that align with being Mozillian and I want to install them into others. Here’s the catch: becoming a Mozillian isn’t close to how, in a perfect world, one might install a binary package containing certain values into a person and we have no reliable test of whether it patches correctly and provides a guaranteed shared core with each other.

After this weekend, and talking deeply and at great lengths with people who have joined the community in the last two years, who have obvious passion & commitment to the Mozilla mission, I have started unpacked my stereotype of a what makes a new Mozillian passionate and what hooks them.  It’s something I will have to keep reminding myself of.  The reminder will look like asking people what their hook was instead of looking for mine in them.  Knowing logistically that it was impossible for everyone to have my exact experience but trusting they had something to activate them in the ways I felt mattered were not aligning at all until I got to spend this time having what sometimes seemed like the same conversation over and over.  I was with 60 people where at least half of them were brought on in the last two years and yet, the discussions were passionate, committed, and inspiring. I am thankful for our process that it allowed me to have this revelation that will help me be a better Mozillian and community leader in the future.

In the actions I have leading up to the Summit, one is to look closer at the ‘hooks’ of others and to work on how we might abstract & synthesize those into options for new Mozillians to have an activity to engage in where we can be as close as 100% certain as possible that they’ve been activated as Mozillians with the core values we’re wanting to trust each other has. A follow-up post will talk about the core values and their importance to several other big topics for Summit 2013.

* ~300 employees to ~600 employees over the course of 2011 and continuing to grow towards 1000 in 2012-13

Trust Process

This is the first in a series of blog posts that will summarize my experience and takeaways from the Mozilla Summit 2013 Planning Assembly that took place in our Paris, France office on June 14-17, 2013.

✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈✈

We are a group of very smart people. Don’t ever doubt that for a second. We are a hit-the-ground-running bunch of doers who really like actionable items and clear instructions as well as accountability and deliverables. Oh and throw in some metrics and a post-mortem so we can iterate, please. This makes us very challenging for outsiders to work with, especially when trying to engage a sizable group in open process activities. Turns out we’ll ask a lot of questions to clarify instructions, search high and low for actionable items, and demand deliverables. We’ll interrupt process, we’ll doubt the value of the process, and we’ll assume even before attempting that it is too open-ended to be of value or to generate the tangible items we’re craving as the proof our time was well spent.

If we go too far down that path in this unconference style of meetup, we will lose out tremendously on letting ourselves be changed and engaged by each other.

Love it or hate it, asking each other “is this making sense to you?” or “are we doing it right?”,  that uncertainty IS a big part of process. Asking each other questions, being confused together, not knowing what’s coming next together is a form of bonding.

Bonding – which I shall refer to from now on as friendship – typically strengthens over equal parts of time spent together + common interests + shared experience. Sometimes you can fast forward a friendship by just overloading one of those areas. Having a particularly intense experience with someone (eg: riding a roller coaster), spending a lot of consecutive time (eg: traveling together), and connecting intensely on a common interest (eg: hacking) are all examples of ways to ramp up the development of a strong bond with another human. When attending a Summit we are provided a multitude of opportunities to have many of those three with a variety of Mozillians we normally might not interact with, and sometimes with ones we do – deepening connections is the ‘bonus’ to the work we get done when we assemble en masse. My most remembered experiences at Summits, All-Hands’, Moz Camp, and other larger gatherings with Mozillians have always been about the micro interactions which are not  part of the schedule. Walking to dinner, sitting on the bus, playing Rock Band – those are opportunities to look at the person or people around you and to try to make sure you’re getting to know even a little bit about them.

In the very earliest part of our process for the weekend of discovery and digging into the meat of what should drive our Summit planning a question came up about whether we could affect the decision to have the Summit in 3 different geo-locations.  Just that one little detail was something many people were tripping over and wanted to discuss and it was clear our dive into the weekend’s process would be held up until there were answers. What ended up happening was Mardi explained how logistically they had looked at previous Summits & large gatherings and had determined that 600 people seemed to be the largest number of Mozillians in one place where there still could be a ‘cozy’ feeling that allows for the kind of bonding previously mentioned.  Once Mardi said this, I was changed.  I, too, had wondered about the dispersal but now it made sense to me that the importance of connection between Mozillians had been placed before the need to put us all in one location and  I appreciated getting this new perspective on what the process had accomplished already.

Mitchell then spoke about how sometimes we get hung up on comparing one thing to another and are quite vocal if we find it wanting.  It’s important for us to have this gathering, this alignment exercise for the project as a whole, and if we don’t want to call it a ‘Summit’ because it’s not everyone in one physical space then call it something else in your head.  I found her point to be very inspiring and concluded that it’s important to go to this 2013 gathering – at whichever location – and BE there.  Do not waste time comparing it to other events, attend the event you’re at and reflect later. For me, this was when I knew I could trust the weekend’s activities would lead to great things and more changes of my tightly held beliefs I came into the weekend with.  There will be more about this in future posts.

Thanks, Process.

 

 

More reasons to support the Ada Initiative

This week there’s been a tremendous amount of tech community churn with companies being called out for blatant sexism/women-as-sex-objects in their company promotional material.  Sqoot organized a hackathon in Boston and made a very big mistake in their call for participation which kicks off with the assumption that hackers would all be men, then continues with a misguided attempt at an apology that only suggests they are sorry I don’t have the same sense of ‘fun’ that they do (they have updated the apology to this, which I still find lacking).  This morning I woke up to the delightful twit-splosion about Geeklist.  I notice that I had never heard of either of these companies prior to their exposure from feminists calling them out which leads me to think about the long term impact for these kinds of internet altercations.  Much like how having what goes into a MacDonald’s burger exposed or seeing video of how WalMart treats its employees has shaped my physical world consumer habits, I suspect that hearing about/experiencing sexism (or a multitude of other poor behaviours) from a particular company will help steer my internet participation whether I’m already familiar with them or not.

What these events should remind us of is that there are people working on this stuff. Individuals, to be sure, along with bloggers and the tweet-verse but also actual companies like, for example The Ada Initiative.  They are experts at working alongside organizations, tech conference organizers, and open-source communities to help set up training, hiring processes, and organizational policies that would have helped both Sqoot and Geeklist avoid this kind of publicity in the first place by addressing their assumptions at a lower level.

If your company hasn’t got a Code of Conduct (and Mozilla is currently hard at work on creating ours this week after our own conflict a couple of weeks ago), if things are just being brushed aside right now or your employees are told to ‘lighten up’, then trust me: you’ve got a ticking time bomb in your organization’s future.  Not having something in place is not the way to deal with the tricky details that come with the admirable goal of a diverse workplace/community.  Sure, getting those things in place, making sure policies have teeth, and organizing some sensitivity training may not end all possibilities of people getting hurt or ending up in confrontations but I believe that setting the tone and getting a few ducks in a row is a wise undertaking for most companies with more than 2 employees and it most certainly won’t HURT. Once you have something in place, consider future occurrences of conflict opportunities to iterate.

Get in touch with The Ada Initiative today and figure out what your company doesn’t have in place yet that will give it the future you really want.  I’m pretty sure avoiding having your brand dragged through the mud in the eyes of approximately half your potential market isn’t in your business plan.