I’ve been in Release Management for 1.8 years now and in that time we’ve grown from one overworked Release Manager to a team of 4 where we can start to split out responsibilities, cover more ground on a particular channel, and also…breathe a bit. With some of the team moving focus over to Firefox OS, we’ve opened up a great opportunity for a Mozillian to help Release Management drive Firefox Desktop & Mobile releases.
We’re looking for someone committed to learning the deepest, darkest secrets of release management who has a few hours a week consistently available to work with us by helping gather early feedback on our Nightly channel (aka mozilla-central or ‘trunk’). This very fabulous volunteer would get mentoring on tools, process, and build up awareness of risk needed for shipping software to 400 million users, starting at the earliest stage in development. On our Nightly/trunk channel there can be over 3000 changes in the 6 week development cycle and you’d be the primary person calling out potentially critical issues so they are less likely to cause pain to the user-facing release channels with larger audiences.
A long time back, in a post about developing community IT positions, mrz recalled a post where I stated that to have successful integration of community volunteers with paid staff in an organization there has to be time dedicated to working with that community member that is included in an employees hours so that the experience can be positive for both parties. It can’t just be “off the side of the desk” for the employee because that creates the risk of burnt out which can lead to communication irregularities with the volunteer and make them feel unneeded. For this community release manager position I will be able to put my time where my mouth is and dedicate hours in my week to actively shape and guide this community Release Manager in order to ensure they get the skills needed while we get the quality improvements in our product.
So here goes with an “official” call for help, come get in on the excitement with us.
- Are familiar and interested in distributed development tools (version control, bug tracker) typically used in an open source project of size (remember when I said 400 million users? Ya, it’s not a small code base)
- Want to learn (or already know) how to identify critical issues in a pool of bugs filed against a code base that branches every 6 weeks
- Have worked in open source, or are extremely enthusiastic about learning how to do things in the open with a very diverse, global community of passionate contributors
- Can demonstrate facility with public communications (do you blog, tweet, have a presence online with an audience?)
- Will be part of the team that drives what goes in to final Firefox releases
- Learn to coordinate across functional teams (security, support, engineering, quality assurance, marketing, localization)
- Have an opportunity to develop tools & work with us to improve existing release processes and build your portfolio/resume
- Mentor and guide your learning in how to ship a massive, open source software project under a brand that’s comparable to major for-profit technology companies (read: we’re competitive but we’re doing it for different end goals)
- Teach you how to triage bugs and work with engineers to uncover issues and develop your intuition and decision making skills when weighing security/stability concerns with what’s best for our users
- On-site time with Mozillians outside of Summits & work weeks – access to engineers, project managers, and other functional teams – get real world experience in how to work cross-functionally
- Invitations to local work weeks where you can learn how to take leadership on ways to improve pre-release quality and stability that improve our Firefox Desktop/Mobile releases
- provide references, t-shirts, and sometimes cupcakes
I’ll be posting this around and looking to chat with people either in person (if you’re in the Bay Area) or over vidyo. The best part is you can be anywhere in the world – we’ll figure out how to work with your schedule to ensure you get the guidance and mentoring you’re looking for.
Look forward to hearing from you! Let’s roll up our sleeves and make Firefox even better for our users!
We’ve got 2 projects right now for GNOME Outreach Project for Women to apply to: https://wiki.mozilla.org/GNOME_Outreach_Summer2013 thanks to Liz Henry and Selena Deckelmann
If anyone else at Mozilla has a project that can be done in 3 months time (or at least give the contributor a sense of accomplishment and get them very engaged as a Mozilla contributor) feel free to add a project (and a mentor) to the wiki. Applications are being accepted via GNOME until May 1st.
One of my favourite things about this program is that it allows someone to ‘intern’ with Mozilla without the requirement of being a student. If you can help tweet/share the project, that would be much appreciated too. This project has been growing exponentially every session and is making a significant impact to FOSS community and culture.
This question comes up enough that I figured a quick blog post/status update would be helpful.
Poll individual Bugzilla bugs for an autoland token (currently using whiteboard tags, future Autoland has an extension & webservice that makes polling the entirety of bugzilla no longer needed). When an autoland request is found, the serviced does an automated landing to try for you of all non-obsolete patches attached to the bug if they can be landed cleanly on tip of mozilla-central and either the patch author or a feedback provider have appropriate hg permissions, otherwise it reports back to the bug what the issues were. Upon completion of the try run, a comment is left in the bug stating the results and if a final repo destination (or destinations) was specified (the hg permissions must match up between requester/reviewer and the destination repo(s)) the service can continue on to autolanding the patch(es) to the destination repo. A comment would be left on the bug when the push to final destination(s) is done. There would be no reporting back of final build results, that would be handed back over to human eyes on TBPL.
What is Autoland’s status now?
In April of 2012, right before Marc’s second internship ended, we launched a very experimental public-facing version of Autoland and announced it a bit so we could get more people testing it. This had varying degrees of success. We got more bugs ironed out but also discovered that Autoland’s daemons for hgpushing and bugzilla polling tend to fall over a bit too often. When we moved Autoland off it’s staging VM to a more permanent home we lost the status page that would tell us (and developers) what the modules were doing and that really made the workings of Autoland quite opaque. With Marc leaving at the end of April and my switch over to help with Release Management in Feb 2012 I had kept meeting regularly with Marc and driving the project to completion as much as possible but hadn’t been able to pull my weight on coding for the last 3 months of our time together. This left us with an Autoland that stopped working and no one available to continue to massage it into being the robust system we needed it to be. I took mention of Autoland out of our trychooser page, try server docs, and have generally tried to downplay it’s existence while still keeping a plan on the back burner for how we will resurrect it as soon as there is some time.
What does Autoland need to be publicly usable again?
- A status page that can show what modules are running or down, display what’s in the queue, and give a quick visual to users if Autoland is up or down as a result.
- Nagios alerts on the Autoland modules that let me (and other people interested in helping to maintain Autoland) know when things fall over.
- At least one person, if not several, who can access the Autoland master VM and ‘kick’ it as needed
This is what’s needed for a short term solution. I know that we have some bugs with our hg pusher module as well as some trickiness in our message queue that, once fixed, would make the overall system more robust. We need people using the system to be able to catch more of those bugs though, so in the meantime having as close to on-click restoring of the system would be a huge win here.
What does Autoland need to be truly ‘production’ ready?
- Security review on the code and the BMO extension so that we can move away from whiteboard tags and let people use the BMO extension instead — this gives much cleaner input to Autoland of what is being requested.
- More VMs to run hgpusher modules on so that Autoland can handle a larger load. Each VM can run 2 hgpushers max so we’d want to be able to grow our pushing farm as the usage of the system increases.
- Being able to push to repos other than Try.
There is no clear plan to be able to get the system beyond Try landings, but I see automated try landings as still being a huge help so I’d be super happy just to see that part get back to a working state. This project is no longer a RelEng priority now that I’ve permanently moved to Release Management and Marc has gone on to other internships and more schooling. I can’t promise anything time-wise, but I wanted to provide some clarity into what is needed and put out the “patches welcome” call. I see Autoland as being a great option for a community-managed project and I want to keep working on it when time permits. If you are looking to become a Mozilla contributor and are interested in automation and web APIs – this might be a good starter project for you. Please get in touch.
In a timely confluence with Mozilla’s new Steward initiative, I’m preparing to get some community contributors engaged with some of the projects we work on in Release Engineering. A fair amount of our production infrastructure has to be locked behind VPN and sekrit passwords (we have 400+ million users to protect) but there are more and more RelEng side projects. We provide tools to the larger developer community and solve interesting scalability challenges with our unique (and massive) automation systems that can be worked on by any interested person in their own local test environment and then integrated into our /build repos. My personal goal is to try and get 2 or 3 regular community contributors to come work with us on tackling these.
In order to solicit contributions I have been working with David Boswell. We added Release Engineering to the mozilla.org/contribute ‘areas of interest’ page and I have created the beginnings of a RelEng-specific contribution page. The first two areas that I think would be a great introduction to working with RelEng code & tools are the TryChooser and our upcoming Autoland system. For the latter, our intern Marc Jessome is sticking around this fall as a contributor to carry on the amazing work he put into this system over the summer. He’ll be continuing to debug the code and improve the portability of it so that we can get it into a beta testing stage by the end of October. As that work is being done we also need someone to help us write the API functionality that will allow sheriffs and developers to write tools that utilize this new hands-off landing queue. We’d also be happy to have people work on the issues that come up when we take Autoland to the next level – auto-landing on a production branch. To do this we’ll want some automated backing out, bisection, and the ability to wait on getting patches reviewed before continuing.
Another great area for someone interested in helping out Firefox developers is working on the TryChooser syntax and features. There is a whole tracking bug dedicated to try_enhancements and most of those bugs are ones that can be worked on in a local staging environment. It’s a chance to get your feet wet with buildbot and our custom scheduling setup. Some of these smaller bugs would be short on time commitment and high on developer appreciation if you fix them. That can be a winning combination for a new contributor, I speak from experience on that
So, if you’re reading this post and you or someone you know is interested in dipping their toes into becoming a Mozilla contributor and these projects make you curious then come find me and we’ll get you set up with a staging environment so that you can start fixing real world tools and automation bugs in no time.