Lesbians Who Tech 2019 recap

I’ve been to every Lesbians Who Tech summit that has taken place in SF (6 total) at the beautiful Castro Theater and this year was the BEST. I love that this conference is unapologetically Political, that they are committed to modeling a world where 50% of the speakers are women of colour, that they highlight issues that are current and affect more than *just* tech, and that the level of technical acumen in the talks continues to rise year over year.

Also, the conference is just plain FUN. There’s always dancing and video montages and high fives. It’s very collegiate in ways and that’s light and fresh and certainly more engaging than some of the technical conferences I’ve attended where it’s faces in laptops all day long. This year there was a Queer Jeopardy opportunity and I wasted no time in grabbing a spot as a contestant.

When they said they needed 3 players, I worked my way to the front of the stage, which meant going against a stream of 1000 lesbians trying to make a break for the bathrooms! They said to come back in 10 minutes, at 11:40 am, and then I was on stage with 2 other contestants managing to clean up during the first round that day. I was then told to return the next for a second round which ended up being against 2 new players where I continue my winning streak, becoming (as far as I know) the first Queer Jeopardy champ at Lesbians Who Tech. I loved the “Name the drone footage” category – one of them was of a house and I guessed it was Edie Windsor’s but it was Mark Zuckerberg’s which I worried meant someone from LWT flew a drone over his house to capture but a simple Google search turned up the image used:

In the Tech Pavilion, GoDaddy had a photo booth where you could make your own sign to finish the sentence “Make Tech More”. It’s a cute idea and I see that GoDaddy is *trying* but looking at the options they provided, I found myself quickly wishing for other words like “intergenerational”, “accessible”, “authentic”, “socialist”, “anti-capitalist”, or perhaps “socially responsible”. I had to settle for “weird” from their pre-printed options. If someone else does this, I hope they might make a dry erase option for people to write in their own. The best swag was this poofy dog – @buttonspom on Instagram – which got me wondering if Snapchat could make it easier for people to have pet accounts on our platform?? I’ve created an account for Shortstack (@corgishortstack) but I don’t end up posting much to it since switching accounts is not possible in the app (that I know of anyway).

As I mentioned earlier – this conference is unapologetically political and centers the voices of black and brown queer women which meant the excellent work Alicia Garza is doing with the Black Futures Lab got center-stage with a packed house near the end of the day on Friday.

Another talk I enjoyed was “How to Prevent the Robopocalypse” where Cynthia Yeung flipped the script of “robots will take over our jobs” and pointed out how robots & humans will need to work together on certain areas long into the future, and more importantly how universal healthcare, revamped education, and basic income could pair with a welcoming of human/robot society being built. The robopocalyse is a lie, she said, the current systems are already broken and causing harm & risk to humanity. The robots are not the enemy in this scenario, the policies of dropping people from social safety nets, quality education and care, are the real culprits we should be fighting.

My talk went well. I was presenting at 10:30am on Saturday at Badlands with several other amazing speakers. The two folks who went before me both shared some serious knowledge and comfortable public speaking skills about Agile development and then geo motion capture. It was a packed house, impressive for an early, rainy morning. I presented “Ship Fast & Leave No Engineer Behind” which is a slimmed down version of an internal roadshow I’ve been touring to small groups within Engineering to familiarize them with Release; what we do, how we can help them get features ready to launch, and what some best practices and key tools are for being able to make trains on time. I saw a lot of heads nodding in the crowd as I explained our process of moving quickly through development, to stabilization, and then to monitoring releases post-launch. My friend Marcy, who is not a tech worker, was the best barometer of my success – when I was done she said she fully understood what I do now 🙂

Basking in the glow of two days of being immersed with queers who are empowered and vastly knowledgeable about many many things like AI, geo motion capture, Pixar animation lighting, and so much more has left me a little sad to be back to reality, however on the bright side – my laptop is now delightfully bedazzled and provides a daily reminder of the queerest event in tech. I’m super glad that Snap had a strong presence at the conference and that I met several coworkers that I hadn’t before. Looking forward to growing our internal initiatives over the course of the next year with all the awesome women @ Snap that I’ve met in the last month!

Contribution opportunity: Early Feedback Community Release Manager

Did you make a New Year’s resolution to build out your software development skillset with a focused contribution to an open source project? Do you want to work with a small team where you can have a big impact?

We’re once again looking for someone committed to learning the deepest, darkest secrets of release management when shipping a big, open source software project to half a billion users. Our small team of 3 employees already has one contributor who has been a consistent volunteer for Release Management since 2013 and has worked his way from these tasks to taking on coordination and growth strategy for the Extended Support releases. Our commitment to contributors is that we will do our best to keep you engaged and learning, this is not grunt work, it’s deep participation that matters to the organization and to the products we ship.

You’ll need to consistently commit a 1-3 hours a week to helping analyze our Nightly channel (aka mozilla-central or ‘trunk’) and raise issues that need prompt attention. The very fabulous volunteer who takes on this task will get mentoring on tools, process, and build up awareness around risks in shipping software, starting at the earliest stage in development. On our Nightly/trunk channel there can be over 3000 changes in a 6 week development cycle and you’d be the primary person calling out potentially critical issues so they are less likely to cause pain to the user-facing release channels with larger audiences.

A long time back, in a post about developing community IT positions, mrz recalled a post where I stated that to have successful integration of community volunteers with paid staff in an organization there has to be time dedicated to working with that community member that is included in an employees hours so that the experience can be positive for both parties. It can’t just be “off the side of the desk” for the employee because that creates the risk of burnout which can lead to communication irregularities with the volunteer and make them feel unappreciated. For this community release manager position I will dedicate 1-3 hours each week to actively on-board and guide this community Release Manager contributor in order to ensure they get the skills needed while we get the quality improvements in our product.

Here is the “official” call for help, come get in on the excitement with us!

You

Are familiar and interested in distributed development tools (version control, bug tracker) typically used in an open source project of size (remember when I said half a billion users? Ya, it’s not a small code base)
Want to learn (or already know) how to identify critical issues in a pool of bugs filed against a code base that branches every 6 weeks
Have worked in open source, or are extremely enthusiastic about learning how to do things in the open with a very diverse, global community of passionate contributors
Can demonstrate facility with public communications (do you blog, tweet, have a presence online with an audience?)
Will be part of the team that drives what goes in to final Firefox releases
Learn to coordinate across functional teams (security, support, engineering, quality assurance, marketing, localization)
Have an opportunity to develop tools & work with us to improve existing release processes and build your portfolio/resume
Can commit to at least 6 months (longer is even better) of regular participation – this will benefit you by giving you time to really get hands-on experience and understanding of release cycles

We

Mentor and guide your learning in how to ship a massive, open source software project under a brand that’s comparable to major for-profit technology companies (read: we’re competitive but we’re doing it for a mission-driven org)
Teach you how to triage bugs and work with engineers to uncover issues and develop your intuition and decision making skills when weighing security/stability concerns with what’s best for our users
On-site time with Mozillians: select attendance at team & company work weeks – access to engineers, project managers, and other functional teams – get real world experience in how to work cross-functionally
Provide work references about how awesome you are, various swag, and sometimes cupcakes 🙂

I’ll be posting this around and looking to chat with people either in person (if you’re in the Bay Area) or over video chat. The best part is you can be anywhere in the world – we can figure out how to work a schedule that ensures you get the guidance and mentoring you’re looking for. Reach out to me in IRC (lsblakk), on Twitter (@lsblakk) or email (lsblakk at mozilla.com).

Look forward to hearing from you! Let’s roll up our sleeves and make Firefox even better for our users!

Release Management Tooling: Past, Present, and Future

As I was interviewing a potential intern for the summer of 2015 I realized I had outlined all our major tools and what the next enhancement for each could be but that this wasn’t well documented anywhere else yet.

By coming to Release Management from my beginnings as a Release Engineer, I’ve been a part of seeing our overall release automation improve across the whole spectrum of what it takes to put out packaged software for multiple platforms and we’ve come a long way so this post is also intended to capture how the main tools we use have gotten to their current state as well as share where they are heading.

Ship-It

Past: Release Manager on point for a release sent an email to the Release-Drivers mailing list with an hg changeset, a version, build number, and this was the “go” to build for Release Engineering to take over and execute a combination of automated/manual steps (there was even a time when it was only said in IRC, email became the constant when Joduinn pushed for consistency and a traceable trail of events). Release Engineers would update a config files & locale changes, get them attached to a bug, approved, uplifted, then go reconfigure the build machines so they could kick off the release build automation.

Present: Ship-It is an app developed by Release Engineering (bhearsum) that allows a Release Manager to input the configurations needed (changeset, version, build number, partials to be created, l10n changesets) all in one place, and on submit the build automation picks up this change from a db, reconfigures the build machine, and triggers builds. When all goes well, there are zero human hands between the “go” and the availability of builds to QA.

Future: In two parts:
1. To have a simple app that can take a list of bug numbers and check them for landing to {branch} (where branch is Beta, Release, or ESR), once all the bug numbers listed have landed, check tree herder for green status on that last changeset, submit to Ship-It if builds are successful. Benefits: hands off even sooner, knowing that all the important fixes are on the branch in question, and that the tree is totally green prior to build (sometimes we “go” without all the results because of human timing needs).
2. Complete End-To-End Release Checklist, dynamically updated to show what stage a release job is at and who’s got the ball in their court. This should track from buglist added (for the final landings a RM is waiting on) all the way until the release notes are live and QA signs off on updates for the general release being in the wild.

Nucleus (aka Release Note App)

Past: Oh dear, you probably don’t even want to know how our release notes used to be made. It’s worse than sausage. There was a sqlite db file, a script that pulled from that db and generated html based on templates and then the Release Manager had to manually re-order the html to get the desired appearance on final pages, all this was then committed to SVN and with that comes the power to completely break mozilla.org properties. Fun stuff. Really. Also once Release Management was more than just one person we shared this sqlite db over Dropbox which had some fun quirks, like clobbering your changes if two people had the file open at the same time. Nowhere to go but up from here!

Present: Thanks to the web production team (jgmize, hoosteeno, craigcook, jbertsch) we got a new Django app in place that gives us a proper databse that’s redundant, production quality, and not in our hands. We add in release notes as well as releases and can publish notes to both staging and production without any more commits to SVN. There’s also an API that can be scripted to.

Future: The future’s so bright in this area, let me get my shades. We have a flag in Bugzilla for relnote-firefox where it can get set to ? when something is nominated and then when we decide to take on that bug as a release note we can set it to {versionNum}+. With a little tweaking on the Bugzilla side of things we could either have a dedicated field for “release-note text” or we could parse it out of a syntax in a comment (though that’s gonna be more prone to user error, so I prefer the former) and then automatically grab all the release notes for a version, create the release in Nucleus, add the notes, publish to staging, and email the link around for feedback without any manual interference. This also means we can dynamically adjust release notes using Bugzilla (and yes, this will need to be really cautiously done), and it makes sure that our recent convention of having every release note connect to a bug persist and become the standard.

Release Dash

Past: Our only way to visualize the work we were doing was a spreadsheet, and graphs generated from it, of how many crasher bugs were tracked for a version, how many bugs tracked/fixed over the course of 18 weeks for a version, and not much else. We also pay attention to the crash rate at ship time, whether we had to do a dot release or chemspill, and any other release-version-specific issues are sort of lost in the fray after we’re a couple of weeks out from a release. This means we don’t have a great sense of our own history, what we’re doing that works in generating a more stable/successful release, and whether a release is in fact ready to go out the door. It’s a gamble, and we take it every 6 weeks.

Present: We have in place a dashboard that is supposed to allow us to view the current crash data, select Talos (performance) data, custom bug queries, and be able to compare a current release coming down the pipe to previous releases. We do not use this dashboard yet because it’s been a side project for the past year and a half, primarily being created and improved upon by fabulous – yet short-term – interns at Mozilla. The dashboard relies on Elastic Search for Bugzilla data and the cluster it points to is not always up. The dash is written in php and that’s no one’s strong suit on our current team, our last intern did his work by creating a Python Flask app that would work into the current dash. The present situation is basically: we need to work on this.

Future: In the future, this dashboard will be robust, reliable, production-quality (and supported), and it will be able to go up on Mozilla office screens in the dashboard rotation where it will make clear to any viewer:
* Where we are in the current release cycle
* What blockers remain for releas
* How our stability is (over/under acceptable rates)
* If we’re meeting performance expectations
And hopefully more. We have to find more ways to get visibility into issues a release might hit once it’s with the larger population. I’d love to see us get more of our Beta user’s feedback by asking for it on specific features/fixes, get a broader Beta audience that is more reflective of our overall release population (by hardware, location, language, user types) and then grow their ability to report issues well. Then we can find ways to get that front and center too – including to developers because they are great at confirming if something unusual is happening.

What Else?

Well, we used to have an automated script that reminded teams of their open & tracked bugs on Beta/Aurora/Nightly in order to provide a priority order that was visible to devs & their managers. It’s a finicky script that breaks often. I’d like to see that replaced with something that’s not just a cronjob on my personal VPS. We’re also this close to not needed to update product-details (still in SVN) on every release. The fact that the Release Management team has the ability to accidentally take down all mozilla.org properties when a mistake is made submitting svn propedits is not desireable or necessary. We should get the heck away from that asap.

We’ll have more discussions of this in Portland, especially with the teams we work closely with and Sylvestre and I will be talking up our process & future goals at FOSDEM in 2015 as well as following it with a work week in Paris where we can put our heads down and code. Next summer we get an intern again and so we’ll have another set of skilled hands to put on tooling & web service improvements.

Always improving. Always automating. These are the things that make me excited for the next year of Release Management.