Assisted/Automated Landing - Designing the Systems

Ehsan's blog post wishing for assisted landings on mozilla-central started a lot of people talking about this being a very desirable and useful tool for developers, where they could set a flag in Bugzilla and then be free to do other work until the results of their push were posted back to the bug. As part of enhancing the Tryserver I was already working on a way for users to signify in their try-syntax that the results of the push should go to the bug and these two ideas started to fuse into a dreamworld where someone could attach a patch to Bugzilla and have it be tried and pushed to trunk all with some magical bot automation.

After doing a very short survey of developers and their try usage I have observed that there are two very different stakeholders here and both of them need separate-but-related tools:

Developers	The Bot (automation)
ease of use better reporting (less email anyone?) option to post to bug(s) *after* a try run has indicated success	queuing of patches culled from a flag in Bugzilla automatically apply to tip of repo push, and report back with results

After soaking in the survey feedback and a first attempt with a whiteboard yesterday, I woke up this morning with some clearer ideas on how to take a first run at creating this system.  It involves creating several new tools, one new database, and enhancing our existing buildapi.

New tools for Developers:

• Adding more Try syntax options:
• include list of the bug(s) that you would like your try results posted to (however many make for a complete run on your push, this can be one linux build or a complete ~186 builder try: -a buildset) 
• turn off email notifications
• Adding functionality to the self-serve api view for a revision (eg: https://build.mozilla.org/buildapi/self-serve/try/rev/de8ea75bc48e) that will better show your results for that push and provide a button which will post the patch(es) to a specified bug
• Auto-landing from a bug in Bugzilla using the [autoland-try] whiteboard tag where any attached patches which are not obsolete, and have nothing set for 'r' are applied to the current tip of mozilla-central, pushed to try and those results are returned to a comment in the bug

New tools to Automate landings (bot or script):

• Crawl Bugzilla for bugs where [autoland-$branchname] is in the whiteboard and automatically push to tip of named branch, get the results, and return them to a comment in the bug (stripping out the whiteboard tag on completion)
• bot will grab all non-obsolete, r+ patches (if $branchname != 'try') 
• interdependent bugs will not be handled in this first swipe at a working system
• pushes will have autoland-$bugnumber as the reason for the build in schedulerdb so that the results can be watched for, aggregated, and reposted to the bug on completion
• Watch results coming back for one or two oranges (we can set a threshold) and re-triggers those, watching for the second set of results - to attempt catching intermittent oranges
• Backout patches where even with a rebuild on an orange, there still remain orange results
• LDAP authentication checking for bugzilla patch author -> hg commit permissions and being able to ensure that only people with the right credentials can trigger automatic landings. This may mean checking the reviewer too before allowing a patch to be applied & pushed.

The next step is to get this design organized into bugs so that we can parcel out the work involved and start testing/completing segments and features as we work towards the whole. We have a RelEng intern this summer, Marc Jessome (Another Canadian in RelEng!), who will be doing a lot of the work between now and the end of August. Stop by anytime to say "Hi" to Marc and to chat with either of us about the project - feedback is always appreciated.  I'm happy to say that 52 people filled out the Try Usage survey just from posting it on Yammer. That was super helpful, thank you.

Try What You Want

Last week bug 473184 saw some progress when we landed the try_parser which allows devs to put in their hg commit comments what try builds they would like on a particular push. This is a first step towards a two-part goal: 1) giving devs more self-serve tools for try and 2) lowering try wait-times.

Now you can try what you want instead of just pushing and getting the default set (currently default is all opt & debug builds, all unittests, no talos).  You will get less emails and only the results you're looking for and everyone wins when you only ask for what you need since it will make more machines available for other builds.

To use this fabulous new TryChooser in your commit message (soon to be available in a committed config file as well) you do the following:

• You've got your patch ready to try
• Set the commit comment to what you would like to run on try
• In this case, let's say you want a linux opt build, no mobile, no unittests, and only talos tp4 results

hg commit -m "Patch to fix a pixel placement try: --build o --p linux --m none --u none --t tp4"

• Push to try
• What happens? 'Linux tryserver build' builder is created and runs your opt linux build. If the build completes successfully then the talos tests are triggered and once the test-master gets the notice of the change it will read the hg commit comments and only kick off the one builder for tp4 'Rev3 Fedora 12 tryserver talos tp4' 
• You get a total of 3 emails and have used very little compute time - you rock!

please go to http://wiki.mozilla.org/Build:TryChooser for complete info and options as well as some answers to questions you make have and take the TryChooser out for a spin.