Continuing along with the education of future Build and Release engineers, Armen and I were introduced to automation with Robert Helmer. Rob’s leaving Mozilla in a couple of weeks and it’s too bad we won’t get to work with him this summer. He’s really got this all figured out and most of his presentation was way over my head. I said to Ben Hearsum later that it feels like once I’m actually touching the build system it will probably make more sense, but just talking about it? That’s a bit confusing. In any case, Rob presented for about an hour on the automation process from past to present and then there was a lively discussion about what’s coming next.

The next day Ben Hearsum talked about Try server and showed us the current set up. I’m really excited to learn more about Try server, and also to having permissions to use it myself. Considering that my recent patch had to be backed out for breaking buildsymbols on the Mac and Linux platforms the Try server could really help me out. I’ll be talking with Ben about how I can get in on it. I have an LDAP account now…

Following Ben, Nick Thomas showed us the flow of releases from tinderbox to virus scanning to their final resting place on the ftp servers. In his diagram you can see pre Sept 2007, how it is currently and then on the right – what we’re aiming for.

The last presentation of the day was Rob Campbell with his presentation on unit tests. Again we were treated to a thorough explanation of how things were done, how they’re being done now and where things are heading.

The most exciting part of the week was realising that there is still a lot of room in Mozilla to have an impact on how things are done. On Wednesday morning we went for breakfast and Armen and I were lucky enough to be sitting next to Mike Schroepfer – VP of Engineering for Mozilla. He shared a lot of information with us and also pointed out how every single person at the table (the Build, Release and Automation team) has contributed significantly. That Mozilla has been able to grow as quickly as it has without falling down is a testament to all their hard work (and that of many others) and I am inspired to be part of that.

Thanks to Dave and to John for helping organize this opportunity. I think that my transition to interning this summer will be much smoother as I have met many of the people with whom I’ll be working, plus the overviews of each area of Build were invaluable.

Armen and I are in California attending Build and Release team meetings this week. Over the next two days we’ll be introduced to the many facets of the Build and Release workflow.

Todays first session was about Talos with Alice.

Here is the diagram of Talos (copied from the diagram Alice drew – yes Talos is a robot):

Alice walked us through how Talos gets its information from browser builds by having buildbot read for new builds from quickparse which is a text file. Buildbot has a script that knows what to look for in order to find new builds and there is a 5 minute delay before Talos is deployed because the information can get into quickparse before the build is finished and so therefore does not technically exist.

Currently there are 30 Production and 20 Stage Talos machines running, this past December there was only 1 Production machine and the stage machines.

This huge increase of Talos machines has led to an insane amount of data being gathered and a database which is in serious need of some help.

After Alice’s presentation we all tried a standalone Talos so we could see the tests at work.

Anyone can try them, just follow these directions. If you are using a recent nightly you might need to add security.fileuri.strict_origin_policy : false to your sample.config file preferences because of new security features. Also, you can comment out the tjss tests because those are kind of boring – the fun test is the svg since you’ll see a lot of graphics tests running on your browser. This standalone runs on a new profile so it’s okay if you already have Firefox running when you run this script.

So the information that’s generated is good for recognizing regression like in this bug, where if you look at the graph you can see how the build was chugging along, something got checked in that affected performance and then it was backed out and the performance went back to normal.

Pic here, see bug #425941 for more info:

More information on Talos Machines.

Things to remember:
1. When you post your patch you select review ? and not review + (I had been confused about why I couldn’t put my reviewer’s email address in next to the +)
2. When your previously working code stops working suddenly and print statements galore are not helping, and you know what part isn’t working but not why…Stop poking at the code and go to MXR — Thank you MXR for helping me catch (through a line by line comparison) that I had accidentally deleted a key line thinking it was my own addition. I think this calls for an editor that does coloring on text not for syntax but for diff’d text.

Tomorrow should be a big day. I’ll be in a little box office at the Royal Cinema all day and night working for Cinéfranco – if anyone wants to see a French movie, I can hook you up – and I’ll be anxiously awaiting a review result because this is it people, these are the tweaks that should net me a test version of a nightly build.

Fingers crossed that one particular Build guy will be working on Saturday…