Category: all

Catchall for posts.

Release Management Tooling: Past, Present, and Future

Release Management Tooling: Past, Present, and Future

As I was interviewing a potential intern for the summer of 2015 I realized I had outlined all our major tools and what the next enhancement for each could be but that this wasn’t well documented anywhere else yet.

By coming to Release Management from my beginnings as a Release Engineer, I’ve been a part of seeing our overall release automation improve across the whole spectrum of what it takes to put out packaged software for multiple platforms and we’ve come a long way so this post is also intended to capture how the main tools we use have gotten to their current state as well as share where they are heading.

Ship-It

Past: Release Manager on point for a release sent an email to the Release-Drivers mailing list with an hg changeset, a version, build number, and this was the “go” to build for Release Engineering to take over and execute a combination of automated/manual steps (there was even a time when it was only said in IRC, email became the constant when Joduinn pushed for consistency and a traceable trail of events). Release Engineers would update a config files & locale changes, get them attached to a bug, approved, uplifted, then go reconfigure the build machines so they could kick off the release build automation.

Present: Ship-It is an app developed by Release Engineering (bhearsum) that allows a Release Manager to input the configurations needed (changeset, version, build number, partials to be created, l10n changesets) all in one place, and on submit the build automation picks up this change from a db, reconfigures the build machine, and triggers builds. When all goes well, there are zero human hands between the “go” and the availability of builds to QA.

Future: In two parts:
1. To have a simple app that can take a list of bug numbers and check them for landing to {branch} (where branch is Beta, Release, or ESR), once all the bug numbers listed have landed, check tree herder for green status on that last changeset, submit to Ship-It if builds are successful. Benefits: hands off even sooner, knowing that all the important fixes are on the branch in question, and that the tree is totally green prior to build (sometimes we “go” without all the results because of human timing needs).
2. Complete End-To-End Release Checklist, dynamically updated to show what stage a release job is at and who’s got the ball in their court. This should track from buglist added (for the final landings a RM is waiting on) all the way until the release notes are live and QA signs off on updates for the general release being in the wild.

Nucleus (aka Release Note App)

Past: Oh dear, you probably don’t even want to know how our release notes used to be made. It’s worse than sausage. There was a sqlite db file, a script that pulled from that db and generated html based on templates and then the Release Manager had to manually re-order the html to get the desired appearance on final pages, all this was then committed to SVN and with that comes the power to completely break mozilla.org properties. Fun stuff. Really. Also once Release Management was more than just one person we shared this sqlite db over Dropbox which had some fun quirks, like clobbering your changes if two people had the file open at the same time. Nowhere to go but up from here!

Present: Thanks to the web production team (jgmize, hoosteeno, craigcook, jbertsch) we got a new Django app in place that gives us a proper databse that’s redundant, production quality, and not in our hands. We add in release notes as well as releases and can publish notes to both staging and production without any more commits to SVN. There’s also an API that can be scripted to.

Future: The future’s so bright in this area, let me get my shades. We have a flag in Bugzilla for relnote-firefox where it can get set to ? when something is nominated and then when we decide to take on that bug as a release note we can set it to {versionNum}+. With a little tweaking on the Bugzilla side of things we could either have a dedicated field for “release-note text” or we could parse it out of a syntax in a comment (though that’s gonna be more prone to user error, so I prefer the former) and then automatically grab all the release notes for a version, create the release in Nucleus, add the notes, publish to staging, and email the link around for feedback without any manual interference. This also means we can dynamically adjust release notes using Bugzilla (and yes, this will need to be really cautiously done), and it makes sure that our recent convention of having every release note connect to a bug persist and become the standard.

Release Dash

Past: Our only way to visualize the work we were doing was a spreadsheet, and graphs generated from it, of how many crasher bugs were tracked for a version, how many bugs tracked/fixed over the course of 18 weeks for a version, and not much else. We also pay attention to the crash rate at ship time, whether we had to do a dot release or chemspill, and any other release-version-specific issues are sort of lost in the fray after we’re a couple of weeks out from a release. This means we don’t have a great sense of our own history, what we’re doing that works in generating a more stable/successful release, and whether a release is in fact ready to go out the door. It’s a gamble, and we take it every 6 weeks.

Present: We have in place a dashboard that is supposed to allow us to view the current crash data, select Talos (performance) data, custom bug queries, and be able to compare a current release coming down the pipe to previous releases. We do not use this dashboard yet because it’s been a side project for the past year and a half, primarily being created and improved upon by fabulous – yet short-term – interns at Mozilla. The dashboard relies on Elastic Search for Bugzilla data and the cluster it points to is not always up. The dash is written in php and that’s no one’s strong suit on our current team, our last intern did his work by creating a Python Flask app that would work into the current dash. The present situation is basically: we need to work on this.

Future: In the future, this dashboard will be robust, reliable, production-quality (and supported), and it will be able to go up on Mozilla office screens in the dashboard rotation where it will make clear to any viewer:
* Where we are in the current release cycle
* What blockers remain for releas
* How our stability is (over/under acceptable rates)
* If we’re meeting performance expectations
And hopefully more. We have to find more ways to get visibility into issues a release might hit once it’s with the larger population. I’d love to see us get more of our Beta user’s feedback by asking for it on specific features/fixes, get a broader Beta audience that is more reflective of our overall release population (by hardware, location, language, user types) and then grow their ability to report issues well. Then we can find ways to get that front and center too – including to developers because they are great at confirming if something unusual is happening.

What Else?

Well, we used to have an automated script that reminded teams of their open & tracked bugs on Beta/Aurora/Nightly in order to provide a priority order that was visible to devs & their managers. It’s a finicky script that breaks often. I’d like to see that replaced with something that’s not just a cronjob on my personal VPS. We’re also this close to not needed to update product-details (still in SVN) on every release. The fact that the Release Management team has the ability to accidentally take down all mozilla.org properties when a mistake is made submitting svn propedits is not desireable or necessary. We should get the heck away from that asap.

We’ll have more discussions of this in Portland, especially with the teams we work closely with and Sylvestre and I will be talking up our process & future goals at FOSDEM in 2015 as well as following it with a work week in Paris where we can put our heads down and code. Next summer we get an intern again and so we’ll have another set of skilled hands to put on tooling & web service improvements.

Always improving. Always automating. These are the things that make me excited for the next year of Release Management.

New to Bugzilla

I believe it was a few years ago, possibly more, when someone (was it Josh Matthews? David Eaves) added a feature to Bugzilla that indicated when a person was “New to Bugzilla”. It was a visual cue next to their username and its purpose was to help others remember that not everyone in the Bugzilla soup is a veteran, accustomed to our jargon, customs, and best practices. This visual cue came in handy three weeks ago when I encouraged 20 new contributors to sign up for Bugzilla. 20 people who have only recently begun their journey towards becoming Mozilla contributors, and open source mavens. In setting them loose upon our bug tracker I’ve observed two things:

ONE: The “New to Bugzilla” flag does not stay up long enough. I’ll file a bug on this and look into how long it currently does stay up, and recommend that if possible we should have it stay up until the following criteria are met:
* The person has made at least 10 comments
* The person has put up at least one attachment
* The person has either reported, resolved, been assigned to, or verified at least one bug

TWO: This one is a little harder – it involves more social engineering. Sometimes people are might be immune to the “New to Bugzilla” cue or overlook it which has resulted in some cases there have been responses to bugs filed by my cohort of Ascenders where the commenter was neither helpful nor forwarding the issue raised. I’ve been fortunate to be in-person with the Ascend folks and can tell them that if this happens they should let me know, but I can’t fight everyone’s fights for them over the long haul. So instead we should build into the system a way to make sure that when someone who is not New to Bugzilla replies immediately after a “New to Bugzilla” user there is a reminder in the comment field – something along the lines of “You’re about to respond to someone who’s new around here so please remember to be helpful”. Off to file the bugs!

Why I was part of creating a thing called TransTech(SF)

Last night I helped hold the third local meetup of trans and genderqueer people who are interested in getting together to hack on our projects. This is the third event since the amazing Trans*H4CK  Hackathon (the first one of its kind!) that took place in October 2013.

When I heard through the grapevine that there was going to be a trans-focused, trans-organized, local hackathon I was beside myself.  Finally I would get the combination of identity and technology in one space that I have been dreaming of since going back to school as a mature student in 2004-5 in order to get a degree in Software Development.  My lofty goals in going back to school can easily be summarized in two ways: 1. I wanted to get a better job than being a brunch chef or a non-profit artist-run center, contract employee with no job security or dental benefits and 2. I hoped that whatever I could learn about technology and software would become tools I could bring back to my queer/trans/genderqueer/arts community and help anyone who wanted it at whatever level they ask for.  This has come to mean hosting websites for people, helping build websites for people and small orgs, and also doing some random consulting. It has also led, funny enough, to a certain amount of fixing people’s computers.  That’s nothing I learned in school, it’s just something I seem to have the right combination of lucky and chutzpah to try — ratio of success lies at about 85-90% so far (some things are unfixable, by me at least).

Anyway, I got into tech and got through a grueling 4 years of school with all its associated stresses.  No need to go into details because I finished that thing AND I got a pretty amazing job right out the gate.  Health benefits (specifically the much-needed dental)?  Check.  Ability to help my communities with tech stuff?  Yes….ish.

Turns out it’s not that easy to still be people’s go-to for tech help when you start working more-than-fulltime at a fast-paced company. Also when you move across the country from the majority of your closest friends and aquaintance community.  I made that decision though because I wanted to take the opportunity that existed for me to come here, to SF, and be in the heart of it all.  Little did I know at the time I was walking on stage at a very precarious time (that’s a whole other post).

Skip forward.  I’m lucky enough to have one woman (self-identified to me as a feminist on our first meeting) on my team of 8 people.  So I think tech is pretty awesome. Then she leaves, goes to another team.  Then I go to my first PyCon and swim in a sea of homogenous men I don’t know.  Then I meet a handful of really exciting people who put the words “Queer” and “Tech” on a scrap of paper for a DIY session in the basement of that hotel in Atlanta, 2009.  That is one of the *big* moments.  It’s when I started, with that group of people, to promote teaching Python to women. Based on the success of RailsBridge‘s workshops teaching Ruby on Rails to women (I went to one, Ruby and Rails are not my thing for now) and also the Boston Python Workshop which did a lot of the groundwork for adapting the RailsBridge-style weekend material to a Python version, I ended up starting PyStar with a few people. Our goal was similar:  make it possible for more women to feel comfortable learning Python in a non alpha-male space.  We had a few workshops in SF and Mountain View (Mozilla office space was easy to get and use for weekend workshops). There were groups creating workshops in Philadelphia (still, to this day, the most active city for this project) and in Minneapolis.  I even threw one together in Paris for a night when I was living there for 3 weeks and working from the Paris Mozilla office.

I’m sharing this with you, reader, because this is the backdrop for what comes next.  For me, teaching women to code was…OK.  Don’t get me wrong – it’s important work and I’m glad it’s happening.  For me though, I still felt kind of lonely.  Since the majority of attendees were middle/upper class, young, heterosexual women I wasn’t really connecting on the levels that I do with people who share other similar identities and life experiences with me.  Or at least ones I am more familiar with, if not my own.

I got a bit disheartened by organizing events that I no longer felt I truly connected with and I took some time to try and figure out how to best spend my time and how to be the activist I wanted to be.  Enter GeekFeminism, having several of the people I met through that hired on at Mozilla, PyCon drastically improving its ratio of women attendees (speakers, still being improved), and meeting Leanne.  Over the past couple of years, again keeping it short, I’ve managed to go to conferences and meet lots of people and dream big with them. The LGBT lunches at Grace Hopper have been fascinating.  Getting to attend AdaCamp – huge boost.  Finding people and connecting with them on all the things that are big problems we want to be a part of solving — the one is often comes down to is “How do we get more diversity in tech?”.

At Mozilla I end up making that a second full-time job that is not what I am paid to do (working on that).  Instead I got discretionary funding that I could use to help leverage women in open source which eventually grew into supporting anything with diverse populations that were not prominent or numerically well-represented in open source.  This looked like me being able to offer up the Mozilla office(s) if and when I could be there to hold the space.  It looked like being able to sponsor a variety of events with amounts less than $2k who were promoting, retaining, or otherwise engaging people in tech & open source more specifically.

One of the events I ended up directing money towards was Trans*H4CK because, as I mentioned at the start – I was so frickin’ excited that this was going to happen and I wanted to help it happen.  I reached out to Kortney and offered money, asked if (while I was planning to attend regardless) I could talk for 5 minutes about open source. I did that.  I went and talked and gave my, now usual, talk about why open source is important (think: View Source on webpages and how you might have benefited from being able to do that).  Then I hacked on stuff and met people and generally soaked up what it felt like to be in a space where I felt seen, whole, and just surrounded by people who were also being seen.  We exist, in numbers more than the one or two at your day job! What a fun weekend.

Then I wondered – why just one weekend?  This was in the Bay Area and while a handful of people traveled to the event most were local.  In the style of user groups and meetups, why not continue to have a monthly get-together of this particular blend of nerdgeektransgenderqueer fabulousness?  I proposed this to Kortney, offered up the Mozilla SF space to hold it, got food for the attendees, and we had one in November that was probably attended by about 20 people.  Pretty sweet.  There was a lot more socializing than actual hacking but that happens when people are still so new to getting to be around each other.  With the success of that, I wanted to hold more but I think the holiday period got in the way so the next one I helped put on, this time at the Double Union maker space, was February 18th, 2014.  It being closer to my house and not at the office I rarely go to anyway was awesome, btw. Also I bought snacks with my own money to bring. That night I thought: we need a regular night – say, the third Tuesday of every month, so that it can be something people just *know* is on and can show up to. No need for invites, mailouts, or headcount – keep it simple. I proposed this to the people at the meetup, and thought this was generally agreed upon.

With that in mind, and with a pretty busy schedule coming up, I wanted to get the March date on the Double Union calendar sooner rather than later so a couple of weeks ago I reached out to the membership list asking if the Tuesday night circuit class would be up for sharing the space with another trans hacking night (as that was the best date for me, as the organizer).  They were open to it and all seemed fine but another email thread started up about whether or not the trans hacking nights would be trans-only or not (if they should, even).  This is something, btw, that the TransTech community participants will be chatting more about so as to determine what guidelines and priorities to have when choosing space or setting up meetings. In that thread Kortney expressed concern about not being able to make the date and I said that I was trying to set up the every-third-Tuesday nights, as discussed at the February meeting.  No further response came and I went ahead with booking it (circuit group moved to another night to give us the whole space, thanks!) and setting up an invite.  Double Union still doesn’t publicize their address so the invite was needed to have a way to reach people who said they would attend.  My goal was to use this next meetup to set up open communication tools that could be used by any member of this nascent “we get together and hack on stuff with other trans/genderqueer people” community.  So we did that.

Last night I was at Double Union with some others and I set up a mailing list, a wiki, and IRC channel, and a Twitter account.  I shared all the admin/login details with the others because this isn’t *my* thing. This is, to the best of my knowledge and experience in open source, how we start communities that are as free of bottlenecks on getting things done as possible.  Then I put that information out there.  The goals, again, are to have spaces where trans and genderqueer people who are in technology (or want to learn to be) can meetup to work on their projects or even work on each other’s projects.  I hope we’ll also get to work on projects that help our communities, for example I’m trying to figure out how to help Homomobiles get an app as it drives me nuts to see people using Uber and Lyft when we have this opportunity to put even a sliver of our collective queer and trans money back to a queer and trans organization and help grow it.

All that to say:  There’s always room for lots of ways to bring people together. This is my expression of how I do it.  Mozilla, while being my employer, does not dictate what I do with my spare time and yet I am fortunate in being able to book our office space(s) when available for events.

I’m being told on Facebook and Twitter that this is stealing, that I’m stepping on Kortney’s toes and please believe me, I tried to do everything I could to avoid that. I intentionally sought another name out of respect for the way that I see Kortney branding and promoting Trans*H4CK and how he is doing great things with that.  I see TransCode (teaching coding), Trans*H4CK (hackathons and speaker series), and then TransTech (local community building, mentor/mentee finding, getting things done in a room with people you have that certain connection with) as all being part of the building blocks for continuing a trajectory of making and taking up space in technology for those of us who are outsiders, underrepresented, or just plain want to put our efforts into working with other social justice activists in building trans & genderqueer focused space.

If you’re in the Bay Area and want to know about meetups, be part of shaping this community, or otherwise lurk about to see what might grab your interest down the road then please follow us on Twitter and/or join our mailing list: https://mailman-mail5.webfaction.com/listinfo/transtech-discuss

Let’s support each other in always leveling up.