Assisted/Automated Landing - Designing the Systems

Ehsan's blog post wishing for assisted landings on mozilla-central started a lot of people talking about this being a very desirable and useful tool for developers, where they could set a flag in Bugzilla and then be free to do other work until the results of their push were posted back to the bug. As part of enhancing the Tryserver I was already working on a way for users to signify in their try-syntax that the results of the push should go to the bug and these two ideas started to fuse into a dreamworld where someone could attach a patch to Bugzilla and have it be tried and pushed to trunk all with some magical bot automation.

After doing a very short survey of developers and their try usage I have observed that there are two very different stakeholders here and both of them need separate-but-related tools:

Developers	The Bot (automation)
ease of use better reporting (less email anyone?) option to post to bug(s) *after* a try run has indicated success	queuing of patches culled from a flag in Bugzilla automatically apply to tip of repo push, and report back with results

After soaking in the survey feedback and a first attempt with a whiteboard yesterday, I woke up this morning with some clearer ideas on how to take a first run at creating this system.  It involves creating several new tools, one new database, and enhancing our existing buildapi.

New tools for Developers:

• Adding more Try syntax options:
• include list of the bug(s) that you would like your try results posted to (however many make for a complete run on your push, this can be one linux build or a complete ~186 builder try: -a buildset) 
• turn off email notifications
• Adding functionality to the self-serve api view for a revision (eg: https://build.mozilla.org/buildapi/self-serve/try/rev/de8ea75bc48e) that will better show your results for that push and provide a button which will post the patch(es) to a specified bug
• Auto-landing from a bug in Bugzilla using the [autoland-try] whiteboard tag where any attached patches which are not obsolete, and have nothing set for 'r' are applied to the current tip of mozilla-central, pushed to try and those results are returned to a comment in the bug

New tools to Automate landings (bot or script):

• Crawl Bugzilla for bugs where [autoland-$branchname] is in the whiteboard and automatically push to tip of named branch, get the results, and return them to a comment in the bug (stripping out the whiteboard tag on completion)
• bot will grab all non-obsolete, r+ patches (if $branchname != 'try') 
• interdependent bugs will not be handled in this first swipe at a working system
• pushes will have autoland-$bugnumber as the reason for the build in schedulerdb so that the results can be watched for, aggregated, and reposted to the bug on completion
• Watch results coming back for one or two oranges (we can set a threshold) and re-triggers those, watching for the second set of results - to attempt catching intermittent oranges
• Backout patches where even with a rebuild on an orange, there still remain orange results
• LDAP authentication checking for bugzilla patch author -> hg commit permissions and being able to ensure that only people with the right credentials can trigger automatic landings. This may mean checking the reviewer too before allowing a patch to be applied & pushed.

The next step is to get this design organized into bugs so that we can parcel out the work involved and start testing/completing segments and features as we work towards the whole. We have a RelEng intern this summer, Marc Jessome (Another Canadian in RelEng!), who will be doing a lot of the work between now and the end of August. Stop by anytime to say "Hi" to Marc and to chat with either of us about the project - feedback is always appreciated.  I'm happy to say that 52 people filled out the Try Usage survey just from posting it on Yammer. That was super helpful, thank you.

Automated Try Results Posted to Bugzilla - A request for input on what the comment should contain

Lately I've been working on a a script which can check your try syntax for a bug number and a setting asking for --post-to-bugzilla.  If you've provided both, your try server results can be posted directly to the bug.  This is just part of a larger project to have patches submitted on a bug get automatically tried out, results posted, and at some point down the road they could even be pushed to trunk after a successful try run so "look Ma! No hands".

Today I have a script running in staging, polling for completed try runs and doing dry runs of posting to bugzilla in log format only so I can keep an eye out for unusual output.  Already this has shown me a couple of bugs to sort out, and I anticipate having them ironed out very soon.  However, before this lands I would like to get some feedback/ideas/suggestions on what the output to the bug should look like and there could be a couple of options even, with a setting in your try syntax.

In my early testing I posted to our landfill bugzilla and here's what a couple of results looked like:

Lots of success looked like too much info

So I took it out, and only printed the warnings and failures

Which works on small runs

From what you see here, I'm sure you can imagine what it would look like if 145 builds all had warnings/failure combos.

So - what do you want to know in the bug? Let's keep it simple, ok? We can add more later and it's important not to create a bug-spammer here that folks will clamor to turn off soon after it goes live.

Off the top of my head, and after talking with Catlee today about it, I think it should look like this:

Try run for $revision with the following comment:
$try_syntax line
S:# W:# F:# (results total) builds complete from N total requests
S:# W:# F:# (results total) tests complete from N total requests
For more information please see http://tbpl.mozilla.org/?tree=MozillaTry&rev=$revision

This gets you a quick glance at total builds/total requests so you can see that everything is accounted for, and where things might have gone wrong in builds vs. tests but doesn't list the failed/warning builder names so you have to follow the link to get them.  Maybe there would be interest in printing what your try syntax request triggered but I'm not sure that's useful in the bug reporting even though it's requested for when a developer is pushing to try. What do you think?

It would be possible to break down the results further by platform instead of or as well as build/test.  Any ideas on how to get that much information across without making a bug comment too verbose? All input appreciated and considered, I'll be trying to land this in the next couple of weeks so comment here, ping me in IRC (lsblakk), or comment in the bug.

Learning advanced Bugzilla

In our first year of Mozilla development at Seneca - we learned how to file basic bugs, how to upload patches and we followed a module owner or the like so we could see just how much bugmail a person can handle. Now that I'm a Build intern I am learning to use bugzilla on a few more levels.

This is just one (small) example:



These bugs will allow me to go through the process of creating, setting up and deploying 4 new VMs so that I'll be able to see if they can handle both build and unittest builds. In a perfect world, they will be able to do this and that means we will set up a pool of buildslaves for each platform and delegate work from both build and unittest as needed. The goal is to be scalable and to use as little hardware as possible.

The reason I have to test this is because it's possible that VMs cannot successfully run unittests as they are written now. We need to know that the unittests can and will run before choosing this method.

More on this soon. Back to bringing WinXP vms up to date with mozilla-build instead of ye olde Cygwin.

Source Server Tweaks - Now with refactoring and a clean_root function

Things to remember:
1. When you post your patch you select review ? and not review + (I had been confused about why I couldn't put my reviewer's email address in next to the +)
2. When your previously working code stops working suddenly and print statements galore are not helping, and you know what part isn't working but not why...Stop poking at the code and go to MXR -- Thank you MXR for helping me catch (through a line by line comparison) that I had accidentally deleted a key line thinking it was my own addition. I think this calls for an editor that does coloring on text not for syntax but for diff'd text.

Tomorrow should be a big day. I'll be in a little box office at the Royal Cinema all day and night working for Cinéfranco - if anyone wants to see a French movie, I can hook you up - and I'll be anxiously awaiting a review result because this is it people, these are the tweaks that should net me a test version of a nightly build.

Fingers crossed that one particular Build guy will be working on Saturday...

Just a blip...

Well, I've broken the ice that formed on my project over the holidays. There's something that really intimidated me about going back to that code, that work flow. I don't know how to explain it except that I felt afraid of my own code and lack of experience, felt like I wasn't going to remember anything. So I hemmed and hawed but tonight I got back into it for a solid 3 hours and it feels good to remember what I was doing and why.

One small change that has made my life easier - I added srctool.exe to my path so I could call it from anywhere and this makes life good because when I want to test if my code is working I have to call make on the new symbolstore.py, then call make buildsymbols in my objdir, then go into the command window and call srctool.exe -r (a raw dump of the indexing info in a pdb) to see if it all worked. The path to a pdb file is something like C:\ff_clean\mozilla\objdir_debugInfo\dist\crashreporter-symbols\2007120422\accessiblemarshal.pdb\E82E8047412045539226FBBC0BE301974\accessiblemarshal.pdb
and so being able to go to that directory and just call srctool -r accessiblemarshal.pdb is way easier. So I'm happy about that.

I've submitted my changes so far back to the bug and am now awaiting feedback. If I can successfully handle at least one of the points that led to a - review, then I am moving forward.

Now it's time for some (light) reading about windows proprietary debug database formats.