Want to help? Encouraging community contributions to Release Engineering

In a timely confluence with Mozilla's new Steward initiative, I'm preparing to get some community contributors engaged with some of the projects we work on in Release Engineering.  A fair amount of our production infrastructure has to be locked behind VPN and sekrit passwords (we have 400+ million users to protect) but there are more and more RelEng side projects. We provide tools to the larger developer community and solve interesting scalability challenges with our unique (and massive) automation systems that can be worked on by any interested person in their own local test environment and then integrated into our /build repos. My personal goal is to try and get 2 or 3 regular community contributors to come work with us on tackling these.

In order to solicit contributions I have been working with David Boswell. We added Release Engineering to the mozilla.org/contribute 'areas of interest' page and I have created the beginnings of a RelEng-specific contribution page. The first two areas that I think would be a great introduction to working with RelEng code & tools are the TryChooser and our upcoming Autoland system.  For the latter, our intern Marc Jessome is sticking around this fall as a contributor to carry on the amazing work he put into this system over the summer.  He'll be continuing to debug the code and improve the portability of it so that we can get it into a beta testing stage by the end of October.  As that work is being done we also need someone to help us write the API functionality that will allow sheriffs and developers to write tools that utilize this new hands-off landing queue.  We'd also be happy to have people work on the issues that come up when we take Autoland to the next level - auto-landing on a production branch.  To do this we'll want some automated backing out, bisection, and the ability to wait on getting patches reviewed before continuing.

Another great area for someone interested in helping out Firefox developers is working on the TryChooser syntax and features.  There is a whole tracking bug dedicated to try_enhancements and most of those bugs are ones that can be worked on in a local staging environment.  It's a chance to get your feet wet with buildbot and our custom scheduling setup. Some of these smaller bugs would be short on time commitment and high on developer appreciation if you fix them. That can be a winning combination for a new contributor, I speak from experience on that :)

So, if you're reading this post and you or someone you know is interested in dipping their toes into becoming a Mozilla contributor and these projects make you curious then come find me and we'll get you set up with a staging environment so that you can start fixing real world tools and automation bugs in no time.

Tree Closing Downtime Notice - 4am - 8am PDT Thursday June 16, 2011

Trees will be closed for downtime so that we can land the following:

1. https://bugzilla.mozilla.org/show_bug.cgi?id=662396 -- Fix time on dm-wwwbuild01

2. https://bugzilla.mozilla.org/show_bug.cgi?id=600980 -- Set journal_mode = WAL for dirty places profiles -- This mean new performance numbers will start on Thursday morning after the downtime

3. https://bugzilla.mozilla.org/show_bug.cgi?id=649123 -- Run ANALYZE on dirty places.sqlite files --  This mean new performance numbers will start on Thursday morning after the downtime

4. https://bugzilla.mozilla.org/show_bug.cgi?id=663568 -- reboot the DNS and DHCP servers in scl1 -- Rebooting these servers has been shown to burn builds in the past, requires a short (~5min) outage to reboot these servers to allow updates to take effect.

5. https://bugzilla.mozilla.org/show_bug.cgi?id=663963 -- change LDAP to see if that speeds up mercurial -- This change should be entirely transparent.  Hg processes that are running at the time that the change was made will have already loaded the NSS LDAP module and will continue to use it until they exit.  The only issue to be aware of is that changes to hg access (group membership, or the creation of a new account) will not automatically propagate to the hg servers the way they do now.  If any hg access changes need to be pushed urgently, we can do that manually.

If anyone has a reason not to proceed with this downtime, please let me know.

Use Try? Read this.

Two updates to Try are about to go into effect which enforce asking for what you want using the try syntax and configuring how much email you want to get with your results.  Read more below.

Bug 661409 - Now that this has landed, a push to try only generates email about a particular try builder's results if it does not succeed.  You can adjust this to be more verbose by adding a -e/--all-emails to your try syntax if you miss getting over all those emails, or you can just shut off the emails completely with a -n/--no-emails in your commit syntax. Note that you must be using the "try: " syntax for these email flags to be picked up which leads quite handily to...

Bug 649402 - Try syntax use is about to be mandatory as soon as this bug is fixed and the hg hook is enabled on the try repo. We're doing this to encourage developers who use try to take an extra moment and request only the resources they absolutely need on their push.  This should reduce the test/talos load that has been increasing wait times across all branches during busy periods.  One additional psychological change is that the "try: -a" syntax has been removed and in order to ask for a mozilla-central matching run you must be more explicit: "try: -b do -p all -u all -t all". I've updated the docs to reflect this change as well as the TryChooser syntax helper webpage. We're really not trying to make your life harder with this change, approximately 50-60% of pushes to try currently use the try syntax and if you push to try without it you will get a helpful message pointing you to docs and syntax builder.  Check with #developers for tips and tricks from the folks who've been using this since the beginning, I know they have many including using the newly-minted Mozilla-Inbound repo where a push will get the complete set of tests/talos if you'd like to let your patch bake for a bit after doing a selective try run.

Try Server Road Map - Q1 2010

The other day this post by Google with slides detailing their Chrome release cycle speed up was going around and it mentioned how try and CI were key to their success. It got me thinking that it's time for another update about the upcoming improvements our try server automation. Most of my Q1 work will be on the try server, with some time on Fennec Beta releases, and a bit of time also working on making it much easier to spin up disposable project branches.

The road map image above shows how there are three areas of focus and here they are now in a more detailed list with bug number attached:

Improving Current Automation

• Bug 617321 tracks adding two new try buildbot master instances to our other buildbot-masters in the Santa Clara colo.  This gives us flexibility to have rolling downtimes on try (as we already have on the other masters) where we can update things behind the scenes and it also helps by adding redundancy to the try automation in case of a colo outage.
• Bug 580346 is almost done and it adds xserves to try which gives us some faster macosx building power, that along with about 40 ix builders for win/linux builds will crank out more try builds faster.
• Bug 594236 is key to getting TryChooser syntax turned on as a default.  With an interactive hg prompt on push to try, you should be able to select what you want and/or have your syntax validated.  Khuey started something to do this and if anyone is up for taking it to the next level before I can get around to it, please please do.

Add Features

• Bug 430942 is what I am actively working on today and the rest of this week, I'm about to have a second draft ready for review and I expect you can watch for this to land in the next few weeks. With or without try syntax, you will be able to specify the bug number in your push comment and have the results of your try run posted as a comment on that bug.  Hopefully this will help out development in letting people know where something is at if you are away when the results come in. It's also part of our old bug (pre-2010) smack down goal so finishing it will be one more step toward that carried-over goal being met.
• Bug 615705 is tracking a few more tweaks to the try syntax that will give users more flexibility in the syntax and choice about what to build.
• Bug 421895 is another old bug and Chris Atlee is close to getting it up for poking at. It provides a way for users to cancel their own try build requests without having to ping RelEng.
• Bug 621681 addresses having better threading/headers since the current headers only help with threading for some clients. When I first wrote it I was testing with Thunderbird, where it works as intended but apparently Gmail and other clients need some help.

Future

Looking to the future right now we have bug 625464 which talks about setting up something to scan bugzilla for a flag on attachments that will trigger an automatic try run with that attachment and either tip of trunk or perhaps a user repo, which would require the other main future bug, 625463. With the ability to poll and run try on hg.m.o user repos we can have project branches (temporary branches that are loaned out to devs or teams to work on a particular project) toggle a setting that would have their pushes to the repo get run through try instead of the main mozilla-central automation.  This could be handy when you want to limit the machines you are building/testing on with the TryChooser syntax.

I hope anyone reading this will find the upcoming try work as exciting as I do.  Reading the Google slides, I couldn't help but sit up straighter at the mention of try being one of the reasons they were able to speed up their release cycle. I'm hoping we can get there too and that our try server will be more robust and ready to handle our soon-to-be-speedier release process too.

Please use TryChooser

Recently there were some improvements to the trychooser and the landing of those changes led to a couple of bugs[1][2] being discovered and quickly fixed.  It is thanks to those who are regularly using the trychooser that we are able to find bugs quickly and also continue to improve the tryserver.

Right now there are over 350 backed up test/talos requests for the tryserver and when I checked our report for trychooser usage it shows that the average number of users pushing with a try syntax has fallen below 50% where it used to be closer to 60%.

I encourage you to please use the trychooser syntax as much as possible. If you do not need every single try result for your patch, do not just push to try and use up all the resources needlessly. Take a moment to insert some try syntax into your commit message.

See https://wiki.mozilla.org/Build:TryChooser for details and http://people.mozilla.org/~lsblakk/trychooser/ for a simple try syntax builder.

Thanks in advance.

Disposable Project Branches - aka Twigs

Hey Firefox Developers - Want a place that's kind of like Tryserver but all yours for a short period of time?  Release Engineering now has 3 'disposable' branches set up in our build automation that you can use for testing a mozilla-central clone or a clone of your own m-c-based repo.  I call them Twigs because they are for temporary, small projects.

The Twigs (maple, cedar, and birch) each have their own tinderbox pages, run the full suite of unittests, and give you talos results by default.  Like Tryserver, they also allow you to override the mozconfig with a custom mozconfig so that you could kill off platforms you are not interested in results for.  Doing so will give you a cleaner tree and will allow our build and test resources to be used where they are most needed.

To use a branch simply grab a twig from the bookings page, file an IT bug to have your repo cloned over to your selected twig repo and then start pushing to the twig's repo to see your builds start.  At the moment one small glitch is that the first push doesn't get you any builds (bug 562026) - so you need to push twice on a fresh repo clone and the second set will get picked up by the hg poller.

I look forward to hearing if this setup helps you out.

Upcoming improvements to Talos documentation and test suite creation

This quarter I'm going to be joining Alice in trying to improve the system for adding new suites to Talos.  The current system involves a lot of hackery on our side and slows down the ability for us to get Talos suites up and running as quickly as might be desired.

So with John's help to create a prioritized list of suite requests, we will be doing a lot of communicating with developers in the coming months to get them up and to improve the process and documentation at the same time.  Currently there are 10 new suite requests waiting that are known and there may be others.  

Part of the issue with adding new suites is that there is a lack of documentation and tools for developers.  Our new system will look more like this:

* A request is made for a new suite and a developer is attached to the request who will be the lead person for working with us to get the suite into production

* The dev will be able to use tools we provide (standalone talos, corral of staging-talos slaves) to do proof of concept on the suite so that it works and is ready to go up in staging when it's handed over to RelEng

* RelEng will enable the test suite in staging and verify that changes in staging work fine with the other existing jobs being run on the same machines. Once all is well, then rollout to production would happen

As we progress through the suite requests, this process should get easier for all parties and more streamlined.  We hope that by the time we reach suite #10 it will be much easier and faster for developers and RelEng to get the proposed new Talos suites into production.

I mentioned the developers will have tools provided by us. We need to do a bit of work to make these tools usable by developers and the first place to start is with our documentation of what Talos is and how it works.  Following this we will have discussed having boilerplate code for creating each of the two styles of tests startup or pageload.  Also, it might be beneficial to have a coral of Talos machines that can be loaned out to a dev for a limited time in order to test a suite during creation and debugging.  This coral could then be re-imaged and passed along to the next suite developer.

Here is the current documentation page.  Doesn't give you much to go on, right?

Well this is about to change.  Given my complete lack of Talos knowledge, I will be writing up what I learn about Talos as it's happening so that hopefully a more complete set of docs will exist for the Talos neophyte and folks who want to work with us to add new suites will benefit from this as well.

Here's the current list of the docs to be created based on what we think you might want to know:

* How Talos works and an overview of the development from past to present

* What preferences Talos runs with

* A description of each test suite, what each runs

* What the numbers mean

These are the things I don't know - is there anything you don't see listed here that you want to know more about?  Feel free to make suggestions in the comments.

New Branch Timeline: Places

A brief rundown of what was involved setting up the Places project branch. This is based on the time since the branch request was given the go-ahead, not when the bug was filed since that happened quite a bit earlier.

2009-04-29 17:17:43 PDT

• Bug requesting the branch [459269] was re-opened
  

2009-05-18 16:31:46 PDT

• Created a tinderbox page for the branch to report to [Tinderbox Page]
  

2009-05-19 09:16 PDT

• A separate bug was filed requesting a repo [493745]
  
  

2009-05-26 16:32:18 PDT

• Repo is created and [493745] is closed as FIXED
  

2009-06-24 10:22 PDT

• Patches submitted to update config files for Staging-Master and Production-Master
  

2009-06-30 12:51 PDT

• After testing and patch updates, the config files are checked in and P-M is reconfigured
  
  

2009-07-01 08:49 PDT

• Add Nagios monitoring support by filing a bug with IT [501710]
  

2009-07-08 08:04:40 PDT

• [493740] is fixed to deal with the scheduler not picking up the new Places poller after a reconfig, only after a stop/start
  

2009-07-09 14:36 PDT - 2009-07-10 12:50 PDT

• Patches submitted to turn on talos and graph server support for the new builds. The first set were not patches to Talos-Pool so a second set was required.
  
  
• New row in graph server added for branch (bug 459269. IT (justdave) ran the INSERT statement against the production database)
  

2009-07-13 11:18 PDT

• Patches submitted to turn on leak testing debug builds. This was checked-in and P-M was reconfigured the same day.
  

2009-07-13 14:57 PDT

• Bug closed - project branch is up and running on P-M