Lately I’ve been working on a a script which can check your try syntax for a bug number and a setting asking for –post-to-bugzilla.  If you’ve provided both, your try server results can be posted directly to the bug.  This is just part of a larger project to have patches submitted on a bug get automatically tried out, results posted, and at some point down the road they could even be pushed to trunk after a successful try run so “look Ma! No hands”.

Today I have a script running in staging, polling for completed try runs and doing dry runs of posting to bugzilla in log format only so I can keep an eye out for unusual output.  Already this has shown me a couple of bugs to sort out, and I anticipate having them ironed out very soon.  However, before this lands I would like to get some feedback/ideas/suggestions on what the output to the bug should look like and there could be a couple of options even, with a setting in your try syntax.

In my early testing I posted to our landfill bugzilla and here’s what a couple of results looked like:

Lots of success looked like too much info

So I took it out, and only printed the warnings and failures

Which works on small runs

From what you see here, I’m sure you can imagine what it would look like if 145 builds all had warnings/failure combos.

So – what do you want to know in the bug? Let’s keep it simple, ok? We can add more later and it’s important not to create a bug-spammer here that folks will clamor to turn off soon after it goes live.

Off the top of my head, and after talking with Catlee today about it, I think it should look like this:

Try run for $revision with the following comment:
$try_syntax line
S:# W:# F:# (results total) builds complete from N total requests
S:# W:# F:# (results total) tests complete from N total requests
For more information please see http://tbpl.mozilla.org/?tree=MozillaTry&rev;=$revision

This gets you a quick glance at total builds/total requests so you can see that everything is accounted for, and where things might have gone wrong in builds vs. tests but doesn’t list the failed/warning builder names so you have to follow the link to get them.  Maybe there would be interest in printing what your try syntax request triggered but I’m not sure that’s useful in the bug reporting even though it’s requested for when a developer is pushing to try. What do you think?

It would be possible to break down the results further by platform instead of or as well as build/test.  Any ideas on how to get that much information across without making a bug comment too verbose? All input appreciated and considered, I’ll be trying to land this in the next couple of weeks so comment here, ping me in IRC (lsblakk), or comment in the bug.

The other day this post by Google with slides detailing their Chrome release cycle speed up was going around and it mentioned how try and CI were key to their success. It got me thinking that it’s time for another update about the upcoming improvements our try server automation. Most of my Q1 work will be on the try server, with some time on Fennec Beta releases, and a bit of time also working on making it much easier to spin up disposable project branches.

The road map image above shows how there are three areas of focus and here they are now in a more detailed list with bug number attached:

Improving Current Automation

• Bug 617321 tracks adding two new try buildbot master instances to our other buildbot-masters in the Santa Clara colo.  This gives us flexibility to have rolling downtimes on try (as we already have on the other masters) where we can update things behind the scenes and it also helps by adding redundancy to the try automation in case of a colo outage.
• Bug 580346 is almost done and it adds xserves to try which gives us some faster macosx building power, that along with about 40 ix builders for win/linux builds will crank out more try builds faster.
• Bug 594236 is key to getting TryChooser syntax turned on as a default.  With an interactive hg prompt on push to try, you should be able to select what you want and/or have your syntax validated.  Khuey started something to do this and if anyone is up for taking it to the next level before I can get around to it, please please do.

Add Features

• Bug 430942 is what I am actively working on today and the rest of this week, I’m about to have a second draft ready for review and I expect you can watch for this to land in the next few weeks. With or without try syntax, you will be able to specify the bug number in your push comment and have the results of your try run posted as a comment on that bug.  Hopefully this will help out development in letting people know where something is at if you are away when the results come in. It’s also part of our old bug (pre-2010) smack down goal so finishing it will be one more step toward that carried-over goal being met.
• Bug 615705 is tracking a few more tweaks to the try syntax that will give users more flexibility in the syntax and choice about what to build.
• Bug 421895 is another old bug and Chris Atlee is close to getting it up for poking at. It provides a way for users to cancel their own try build requests without having to ping RelEng.
• Bug 621681 addresses having better threading/headers since the current headers only help with threading for some clients. When I first wrote it I was testing with Thunderbird, where it works as intended but apparently Gmail and other clients need some help.

Future

Looking to the future right now we have bug 625464 which talks about setting up something to scan bugzilla for a flag on attachments that will trigger an automatic try run with that attachment and either tip of trunk or perhaps a user repo, which would require the other main future bug, 625463. With the ability to poll and run try on hg.m.o user repos we can have project branches (temporary branches that are loaned out to devs or teams to work on a particular project) toggle a setting that would have their pushes to the repo get run through try instead of the main mozilla-central automation.  This could be handy when you want to limit the machines you are building/testing on with the TryChooser syntax.

I hope anyone reading this will find the upcoming try work as exciting as I do.  Reading the Google slides, I couldn’t help but sit up straighter at the mention of try being one of the reasons they were able to speed up their release cycle. I’m hoping we can get there too and that our try server will be more robust and ready to handle our soon-to-be-speedier release process too.