Try results to the bug(s) of your choice upon completion

The TrySyntax helper and TryChooser wiki docs have both been updated to reflect the new option when pushing to try where you can now ask to have your complete summary of results (and a link to the tbpl page for your revision) posted as a comment to the bug on completion.  Here's a live example to check out:

Sample comment in a bug when using --post-to-bugzilla in your syntax.

Now you have more control over how you get your try results and how noisy a try push is.

Please send comments and issues to the bug tracking this work.  Thanks for trying it out!

Use Try? Read this.

Two updates to Try are about to go into effect which enforce asking for what you want using the try syntax and configuring how much email you want to get with your results.  Read more below.

Bug 661409 - Now that this has landed, a push to try only generates email about a particular try builder's results if it does not succeed.  You can adjust this to be more verbose by adding a -e/--all-emails to your try syntax if you miss getting over all those emails, or you can just shut off the emails completely with a -n/--no-emails in your commit syntax. Note that you must be using the "try: " syntax for these email flags to be picked up which leads quite handily to...

Bug 649402 - Try syntax use is about to be mandatory as soon as this bug is fixed and the hg hook is enabled on the try repo. We're doing this to encourage developers who use try to take an extra moment and request only the resources they absolutely need on their push.  This should reduce the test/talos load that has been increasing wait times across all branches during busy periods.  One additional psychological change is that the "try: -a" syntax has been removed and in order to ask for a mozilla-central matching run you must be more explicit: "try: -b do -p all -u all -t all". I've updated the docs to reflect this change as well as the TryChooser syntax helper webpage. We're really not trying to make your life harder with this change, approximately 50-60% of pushes to try currently use the try syntax and if you push to try without it you will get a helpful message pointing you to docs and syntax builder.  Check with #developers for tips and tricks from the folks who've been using this since the beginning, I know they have many including using the newly-minted Mozilla-Inbound repo where a push will get the complete set of tests/talos if you'd like to let your patch bake for a bit after doing a selective try run.

Update on the Auto/Assisted Landing System

Almost a week since the post introducing the design attempt for auto/assisted branch landings via Bugzilla and Try and guess what? We re-wrote everything!

The details are in the wiki, bugs have been filed, code is being written.  We are working on making this system use a message queue and also see if we can work with mozillapulse to get information on bug changes from Bugzilla.

I'd love to tell you more about it but you can read the wiki and I'm excited to get back to my SchedulerDBPoller component.

Assisted/Automated Landing - Designing the Systems

Ehsan's blog post wishing for assisted landings on mozilla-central started a lot of people talking about this being a very desirable and useful tool for developers, where they could set a flag in Bugzilla and then be free to do other work until the results of their push were posted back to the bug. As part of enhancing the Tryserver I was already working on a way for users to signify in their try-syntax that the results of the push should go to the bug and these two ideas started to fuse into a dreamworld where someone could attach a patch to Bugzilla and have it be tried and pushed to trunk all with some magical bot automation.

After doing a very short survey of developers and their try usage I have observed that there are two very different stakeholders here and both of them need separate-but-related tools:

Developers	The Bot (automation)
ease of use better reporting (less email anyone?) option to post to bug(s) *after* a try run has indicated success	queuing of patches culled from a flag in Bugzilla automatically apply to tip of repo push, and report back with results

After soaking in the survey feedback and a first attempt with a whiteboard yesterday, I woke up this morning with some clearer ideas on how to take a first run at creating this system.  It involves creating several new tools, one new database, and enhancing our existing buildapi.

New tools for Developers:

• Adding more Try syntax options:
• include list of the bug(s) that you would like your try results posted to (however many make for a complete run on your push, this can be one linux build or a complete ~186 builder try: -a buildset) 
• turn off email notifications
• Adding functionality to the self-serve api view for a revision (eg: https://build.mozilla.org/buildapi/self-serve/try/rev/de8ea75bc48e) that will better show your results for that push and provide a button which will post the patch(es) to a specified bug
• Auto-landing from a bug in Bugzilla using the [autoland-try] whiteboard tag where any attached patches which are not obsolete, and have nothing set for 'r' are applied to the current tip of mozilla-central, pushed to try and those results are returned to a comment in the bug

New tools to Automate landings (bot or script):

• Crawl Bugzilla for bugs where [autoland-$branchname] is in the whiteboard and automatically push to tip of named branch, get the results, and return them to a comment in the bug (stripping out the whiteboard tag on completion)
• bot will grab all non-obsolete, r+ patches (if $branchname != 'try') 
• interdependent bugs will not be handled in this first swipe at a working system
• pushes will have autoland-$bugnumber as the reason for the build in schedulerdb so that the results can be watched for, aggregated, and reposted to the bug on completion
• Watch results coming back for one or two oranges (we can set a threshold) and re-triggers those, watching for the second set of results - to attempt catching intermittent oranges
• Backout patches where even with a rebuild on an orange, there still remain orange results
• LDAP authentication checking for bugzilla patch author -> hg commit permissions and being able to ensure that only people with the right credentials can trigger automatic landings. This may mean checking the reviewer too before allowing a patch to be applied & pushed.

The next step is to get this design organized into bugs so that we can parcel out the work involved and start testing/completing segments and features as we work towards the whole. We have a RelEng intern this summer, Marc Jessome (Another Canadian in RelEng!), who will be doing a lot of the work between now and the end of August. Stop by anytime to say "Hi" to Marc and to chat with either of us about the project - feedback is always appreciated.  I'm happy to say that 52 people filled out the Try Usage survey just from posting it on Yammer. That was super helpful, thank you.

Captain Destructo Breaks Everything

Alternate title ideas: "It's not all s/Tryserver/Try"  or "What I should have done, and didn't"

I bet you get the point by now. Today I caused a fairly lengthy, unnecessary downtime on Try.  Now that I'm writing this, things are under control again and there's a few small niggly bits left but nothing that will keep me up at night.

It all started with a bug about graphserver posts from tests not getting through because they were looking for MozillaTry (the tinderbox name for Try) but instead the graphserver only knew about Tryserver (the branch name for Try) and nothing was using Try (except the repo for Try) which is what it ought to have been doing in the first place.

Now that I've been adding a lot of project branches in a short amount of time, certain things have become more streamlined and so I felt that the best option was to go through and rename Tryserver/MozillaTry to Try everywhere so that from the repo going forward, everything was the same. This has been working extremely well for our project branches and helps make setup a snap.

Here's where it gets all broken. I approached this bug with a quick swipe at this problem was superficial and ended up causing some preventable burning.  I shall now list for you (and future me) what I did and what I should have done:

Did:

• hg rename on configs for desktop
• branch configs for s/tryserver/try
• updated graphserver branch name to Try
• a quick downtime window from 10am - 12pm in order to prevent builds from getting split into two different upload dirs

Should have done:

• hg rename on configs for mobile
• grep of buildbotcustom for "tryserver" as we have special casing for it in several files
• log uploader and post_upload scripts to make sure everything about the try build was going to the right place
• updated the dir permissions on ftp for the new upload location and ensure that the archive is on nfs mount
• edited cronjobs on staging to catch the new try builds
• updated graphserver machines table for each try platform's builder name
• more notice for downtime, with a 4 hour window that would have allowed a test push to make sure everything was wired up correctly  
• updated the treeclosure hook to include the new tinderbox page

Some of the things I should have done didn't have an impact on the burning/try closure but it's fair to say that if I had done a staging round of all my plans first I would have caught more of the obvious things that I missed. I would have then planned the downtime better and been prepared to ensure the disturbance would have been minimal since this was, after all, a really low priority bug.

Aki told me that he had a manager who said "you don't learn til you break something".  Well I broke everything try-related today and here's hoping that I have learned something because the stress of this whole day is not something I want to experience often. It's that feeling you get when you realize you've started something that you can't back out of and there's no way to go but forward, even though everything in front of you now appears hopeless and messy.


So here's some lessons to take away:

1. Staging is not to be underestimated even for just renaming things that are already working
2. Taking the time to search with grep/mxr and find the terms you are replacing before starting the upgrade in production will help find wiring you might have overlooked in your preparations
3. Prepare more thoroughly and have a clear idea of the env. you started in and what it will take to have that env. back when you're done. Leaving dangly bits is not ideal.

Happy Friday.
(and many thanks to Aki)

Automated Try Results Posted to Bugzilla - A request for input on what the comment should contain

Lately I've been working on a a script which can check your try syntax for a bug number and a setting asking for --post-to-bugzilla.  If you've provided both, your try server results can be posted directly to the bug.  This is just part of a larger project to have patches submitted on a bug get automatically tried out, results posted, and at some point down the road they could even be pushed to trunk after a successful try run so "look Ma! No hands".

Today I have a script running in staging, polling for completed try runs and doing dry runs of posting to bugzilla in log format only so I can keep an eye out for unusual output.  Already this has shown me a couple of bugs to sort out, and I anticipate having them ironed out very soon.  However, before this lands I would like to get some feedback/ideas/suggestions on what the output to the bug should look like and there could be a couple of options even, with a setting in your try syntax.

In my early testing I posted to our landfill bugzilla and here's what a couple of results looked like:

Lots of success looked like too much info

So I took it out, and only printed the warnings and failures

Which works on small runs

From what you see here, I'm sure you can imagine what it would look like if 145 builds all had warnings/failure combos.

So - what do you want to know in the bug? Let's keep it simple, ok? We can add more later and it's important not to create a bug-spammer here that folks will clamor to turn off soon after it goes live.

Off the top of my head, and after talking with Catlee today about it, I think it should look like this:

Try run for $revision with the following comment:
$try_syntax line
S:# W:# F:# (results total) builds complete from N total requests
S:# W:# F:# (results total) tests complete from N total requests
For more information please see http://tbpl.mozilla.org/?tree=MozillaTry&rev=$revision

This gets you a quick glance at total builds/total requests so you can see that everything is accounted for, and where things might have gone wrong in builds vs. tests but doesn't list the failed/warning builder names so you have to follow the link to get them.  Maybe there would be interest in printing what your try syntax request triggered but I'm not sure that's useful in the bug reporting even though it's requested for when a developer is pushing to try. What do you think?

It would be possible to break down the results further by platform instead of or as well as build/test.  Any ideas on how to get that much information across without making a bug comment too verbose? All input appreciated and considered, I'll be trying to land this in the next couple of weeks so comment here, ping me in IRC (lsblakk), or comment in the bug.

Try Server Road Map - Q1 2010

The other day this post by Google with slides detailing their Chrome release cycle speed up was going around and it mentioned how try and CI were key to their success. It got me thinking that it's time for another update about the upcoming improvements our try server automation. Most of my Q1 work will be on the try server, with some time on Fennec Beta releases, and a bit of time also working on making it much easier to spin up disposable project branches.

The road map image above shows how there are three areas of focus and here they are now in a more detailed list with bug number attached:

Improving Current Automation

• Bug 617321 tracks adding two new try buildbot master instances to our other buildbot-masters in the Santa Clara colo.  This gives us flexibility to have rolling downtimes on try (as we already have on the other masters) where we can update things behind the scenes and it also helps by adding redundancy to the try automation in case of a colo outage.
• Bug 580346 is almost done and it adds xserves to try which gives us some faster macosx building power, that along with about 40 ix builders for win/linux builds will crank out more try builds faster.
• Bug 594236 is key to getting TryChooser syntax turned on as a default.  With an interactive hg prompt on push to try, you should be able to select what you want and/or have your syntax validated.  Khuey started something to do this and if anyone is up for taking it to the next level before I can get around to it, please please do.

Add Features

• Bug 430942 is what I am actively working on today and the rest of this week, I'm about to have a second draft ready for review and I expect you can watch for this to land in the next few weeks. With or without try syntax, you will be able to specify the bug number in your push comment and have the results of your try run posted as a comment on that bug.  Hopefully this will help out development in letting people know where something is at if you are away when the results come in. It's also part of our old bug (pre-2010) smack down goal so finishing it will be one more step toward that carried-over goal being met.
• Bug 615705 is tracking a few more tweaks to the try syntax that will give users more flexibility in the syntax and choice about what to build.
• Bug 421895 is another old bug and Chris Atlee is close to getting it up for poking at. It provides a way for users to cancel their own try build requests without having to ping RelEng.
• Bug 621681 addresses having better threading/headers since the current headers only help with threading for some clients. When I first wrote it I was testing with Thunderbird, where it works as intended but apparently Gmail and other clients need some help.

Future

Looking to the future right now we have bug 625464 which talks about setting up something to scan bugzilla for a flag on attachments that will trigger an automatic try run with that attachment and either tip of trunk or perhaps a user repo, which would require the other main future bug, 625463. With the ability to poll and run try on hg.m.o user repos we can have project branches (temporary branches that are loaned out to devs or teams to work on a particular project) toggle a setting that would have their pushes to the repo get run through try instead of the main mozilla-central automation.  This could be handy when you want to limit the machines you are building/testing on with the TryChooser syntax.

I hope anyone reading this will find the upcoming try work as exciting as I do.  Reading the Google slides, I couldn't help but sit up straighter at the mention of try being one of the reasons they were able to speed up their release cycle. I'm hoping we can get there too and that our try server will be more robust and ready to handle our soon-to-be-speedier release process too.

Please use TryChooser

Recently there were some improvements to the trychooser and the landing of those changes led to a couple of bugs[1][2] being discovered and quickly fixed.  It is thanks to those who are regularly using the trychooser that we are able to find bugs quickly and also continue to improve the tryserver.

Right now there are over 350 backed up test/talos requests for the tryserver and when I checked our report for trychooser usage it shows that the average number of users pushing with a try syntax has fallen below 50% where it used to be closer to 60%.

I encourage you to please use the trychooser syntax as much as possible. If you do not need every single try result for your patch, do not just push to try and use up all the resources needlessly. Take a moment to insert some try syntax into your commit message.

See https://wiki.mozilla.org/Build:TryChooser for details and http://people.mozilla.org/~lsblakk/trychooser/ for a simple try syntax builder.

Thanks in advance.

Impending TryChooser change you should know about

On Monday November 29th there will be a bug fix landed on the TryChooser that will change how you use it:

Removed: -m,--mobile argument for requesting mobile platforms.
ALL platform selections will use the -p,--platform argument (as requested by developers)

This change is part of a larger back-end change which will improve try by making new platforms or test/talos suites added to mozilla-central dynamically available to tryserver. This also means the Mozmill test suite will be enabled for try when this patch lands on Monday.

I will update the TryChooser wiki and the trychooser syntax web helper. If anyone is using other tools to help with using trychooser, please update accordingly.

What will be useful once this lands is a way to give a user-friendly, and dynamically-generated, list of what platforms/talos/test options are available to try. Any ideas?

Get more green with TryChooser!

Our fabulous intern Anamaria Stoica has been working hard the past few months helping to pull data from buildbot's scheduler DB that allows us to learn all sorts of new information about how the release infrastructure is being used, and how it could be optimized. Among other things, she's generated an end-to-end report so we can see at a glance how long the complete build cycle for any push to mozilla-central took.  Now, on top of that, she's added a way to see who is using the awesome new TryChooser syntax.

In the coming weeks I'll be emailing folks who are using try but not the TryChooser syntax and encouraging them to give it a shot. In my stock email letter I will say something along the lines of:

When you use the TryChooser syntax to make requests for only what you need, you allow precious CPU cycle time to be used elsewhere by those who need it.

This is helpful for two reasons:

1. Those testing fixes to blockers of the next beta/rc/release can get try results quickly and get their code landed on trunk
2. Overall wait times for try results are reduced

What's interesting in looking at the reports though is that more of the pushes to try that use TryChooser are green. Here's a snapshot of those who are using TryChooser:

The number of build requests is an aggregate of all the builders triggered so all build types, platforms, test and talos builders run on that push.

Now for comparison, here's a snapshot of folks not using the TryChooser:

I have no way of knowing from this snapshot which of those pushes were intentionally not using the TryChooser and perhaps trying to get all 176 possible builds. Some of them may be using custom mozconfigs as a way to kill off unwanted builds (p.s. you get burning with that).

If you haven't tried it yet, please give TryChooser a go and see if getting a smaller set of builds (ie: what you need, not just the default) and perhaps getting more green on you try pushes gives you a boost.

Try What You Want

Last week bug 473184 saw some progress when we landed the try_parser which allows devs to put in their hg commit comments what try builds they would like on a particular push. This is a first step towards a two-part goal: 1) giving devs more self-serve tools for try and 2) lowering try wait-times.

Now you can try what you want instead of just pushing and getting the default set (currently default is all opt & debug builds, all unittests, no talos).  You will get less emails and only the results you're looking for and everyone wins when you only ask for what you need since it will make more machines available for other builds.

To use this fabulous new TryChooser in your commit message (soon to be available in a committed config file as well) you do the following:

• You've got your patch ready to try
• Set the commit comment to what you would like to run on try
• In this case, let's say you want a linux opt build, no mobile, no unittests, and only talos tp4 results

hg commit -m "Patch to fix a pixel placement try: --build o --p linux --m none --u none --t tp4"

• Push to try
• What happens? 'Linux tryserver build' builder is created and runs your opt linux build. If the build completes successfully then the talos tests are triggered and once the test-master gets the notice of the change it will read the hg commit comments and only kick off the one builder for tp4 'Rev3 Fedora 12 tryserver talos tp4' 
• You get a total of 3 emails and have used very little compute time - you rock!

please go to http://wiki.mozilla.org/Build:TryChooser for complete info and options as well as some answers to questions you make have and take the TryChooser out for a spin.

No really, it *is* easy to test your Python code

This week is all about getting custom selection for tryserver submissions (bug 473184) ready for public consumption. The plan is to have a custom scheduler that calls a function which parses a commit comment (and later, the option of an .info file instead) and then only schedules the builders which have been requested. This will be awesome for our build machine resources since it will mean that if you don't want a particular platform/test/talos suite it will never even get triggered.

In today's work on the parser I wrote python unittests for the first time. I remember doing some unittests in school but that was way back when we used Java. The python unittest module is really awesome and easy to get up and running. Once I had my tests passing I then ran the coverage tool on both the test suite and the parser to see how I did in covering all my bases:

This tool is Frickin' Awesome! I can see at a glance that I need another test to see what happens if someone wants a subset of mobile build platforms.

Tomorrow I'll be getting my coverage even higher and then starting to stage this scheduler and see how it handles our large amount of possible builders to choose from. Keep your ears open - selecting your try build configuration is right around the corner.

Disposable Project Branches - aka Twigs

Hey Firefox Developers - Want a place that's kind of like Tryserver but all yours for a short period of time?  Release Engineering now has 3 'disposable' branches set up in our build automation that you can use for testing a mozilla-central clone or a clone of your own m-c-based repo.  I call them Twigs because they are for temporary, small projects.

The Twigs (maple, cedar, and birch) each have their own tinderbox pages, run the full suite of unittests, and give you talos results by default.  Like Tryserver, they also allow you to override the mozconfig with a custom mozconfig so that you could kill off platforms you are not interested in results for.  Doing so will give you a cleaner tree and will allow our build and test resources to be used where they are most needed.

To use a branch simply grab a twig from the bookings page, file an IT bug to have your repo cloned over to your selected twig repo and then start pushing to the twig's repo to see your builds start.  At the moment one small glitch is that the first push doesn't get you any builds (bug 562026) - so you need to push twice on a fresh repo clone and the second set will get picked up by the hg poller.

I look forward to hearing if this setup helps you out.

Tryserver 2.0 - Fine tuning and learning the hard way

I really thought the try server as a branch was ready to roll out when I did it.  Seriously.  It took a couple of months to get it to the point where I felt it was ready for the public.  So I pushed it live last week - detail here.

Immediately a few issues came up that ruined my vision of a smooth transition:

* Emails were going to the changeset author and not the person pushing the change. Turns out I had changed this default behaviour and didn't know that there was a reason why we used to grab the email address from buildbot sendchange info instead of from hg author.

* Packaged unittests were being merged.  This was quite a big deal for the first full day of being live because people were getting all kinds of strange emails about results that weren't theirs.  The reason this wasn't caught in testing was that there is no 1:1 mapping between a build and its packaged unit test results (except when you go look at tbpl). We should get one set up for MozillaTest where our staging results go in order to make sure something like this doesn't happen again.

* The URL included in an email regarding test results didn't contain a changeset - only the build and leak test builders were setting the information needed to grab this for the emails.  Thanks to dbaron for catching that quickly and bringing it to my attention.

* The try-mac slaves got delayed on their way to the new tryserver's slave pool because of some glitches with puppet and so a huge backlog formed for OS X builds and people thought they didn't even exist.  Because the backlog got so large (80+ full-length builds) I opted to restart the master, wiping those out of the queue in order to get the tryserver back to decent turnaround for all platforms.

I'm still ironing out some issues, tweaking the email results, and I've temporarily disabled the web interface as I work on getting it to use hg push so that all the inputs to tryserver arrive in the same way.  I'm also trying to get more slaves to add to the builder pool since the packaged unittest builders are builder hogs. 

Let me know if I've missed anything.  It's my goal this quarter to make tryserver as helpful as possible in keeping mozilla-central green. 

Tryserver 2.0 is Live

This morning the quietly running in the background try-as-branch became the new try server and the try you've known for a few years now is about to be turned off.

You will now be getting opt and debug builds, as well as packaged unit tests for all current try platforms (64 bit are on the way). If you would like to read more about how the new try server works you can look to https://wiki.mozilla.org/Build:TryServerAsBranch where how to add a custom mozconfig and where to file bugs on tryserver issues is covered, as well as other info.

Now that this new tryserver is up and running, the remaining try slaves from the current tryserver are quickly being moved over and we will be back to full speed on try builds in the next few days.

Upcoming work on try will see 64 bit builds, sending the unit tests over to talos user desktop platforms for testing (just like mozilla-central has now) and also allowing you to select with more granularity which try builds/platforms you would like your patch to be done on. Also there will be a try branch for 1.9.2-based builds coming soon - tracking bug for that is 563822

If you have questions, come find me (lsblakk) in #build or #developers or you can file bugs on tryserver.

Speaking of bugs, this new try-as-branch will allow RelEng to close the following pretty much right away:

• https://bugzilla.mozilla.org/show_bug.cgi?id=486123
• https://bugzilla.mozilla.org/show_bug.cgi?id=486756
• https://bugzilla.mozilla.org/show_bug.cgi?id=487203
• https://bugzilla.mozilla.org/show_bug.cgi?id=497380
• https://bugzilla.mozilla.org/show_bug.cgi?id=532174
• https://bugzilla.mozilla.org/show_bug.cgi?id=550114

I really appreciate everyone's patience during this transition, I hope that the new tryserver gets us better results and helps keep trunk building green.

Adding choices to Try Server web interface

Just put in my patches on bug 473184 which will allow folks who submit patches through the try server web interface to select if they want a build or unittest run and what platforms it should be run on.

Looks like this:


I've tested it successfully in the staging environment and I hope to get this rolled out in production before the end of the quarter.

Tryserver now has unittests on all 3 platforms

Bug 445611 is now resolved fixed, you can see the unittest results on all three platforms here: MozillaTry.

There are hopefully more machines coming that will be allotted to tryserver, with the likelihood that more developers will now use the tryserver. I'm also hoping to start working on setting up the web interface to allow for the selection of which platforms you want to try your patch on and whether you want unittests run or not since at the moment they are turned on for all patch submissions by default.

This patch also required the landing of a patch for bug 479225 where we now call the reftest/crashtest suites and the mochitest suites with make from the top level directory which helped us get rid of a bunch of extra workarounds we had to do for Mac OS since it needed to know which .app file to look for and we're changing that name so often (Minefield, Shiretoko, etc). Now all three platforms are even closer to being alike. Many thanks to Ted for making this possible.

Unittests on Try Server - Linux HG Builds is Go!

So today we deployed a unittest builder for the Linux platform on Try Server. You can see on the waterfall that there is now a column for Linux unittest results. The builds resulting from that builder will not be uploaded anywhere.

The Mac unittest builder is close and I hope it will be ready to go later today, if not then very early next week. Windows is of course being a pain and isn't production ready yet.

This is a big win in that developers can now test their patches in the Try Server and see the unittest results before committing, it's unfortunate though that the current implementation of this functionality is going to take 2 builders per platform per patch. We only have 4 slaves per platform on Try and so you can imagine that this will slow down Try significantly. A Linux unittest run only takes ~ 1hr though so that's good.

Anyway, enjoy the unittests. Watch for more soon.