Tryserver 2.0 - Fine tuning and learning the hard way

I really thought the try server as a branch was ready to roll out when I did it.  Seriously.  It took a couple of months to get it to the point where I felt it was ready for the public.  So I pushed it live last week - detail here.

Immediately a few issues came up that ruined my vision of a smooth transition:

* Emails were going to the changeset author and not the person pushing the change. Turns out I had changed this default behaviour and didn't know that there was a reason why we used to grab the email address from buildbot sendchange info instead of from hg author.

* Packaged unittests were being merged.  This was quite a big deal for the first full day of being live because people were getting all kinds of strange emails about results that weren't theirs.  The reason this wasn't caught in testing was that there is no 1:1 mapping between a build and its packaged unit test results (except when you go look at tbpl). We should get one set up for MozillaTest where our staging results go in order to make sure something like this doesn't happen again.

* The URL included in an email regarding test results didn't contain a changeset - only the build and leak test builders were setting the information needed to grab this for the emails.  Thanks to dbaron for catching that quickly and bringing it to my attention.

* The try-mac slaves got delayed on their way to the new tryserver's slave pool because of some glitches with puppet and so a huge backlog formed for OS X builds and people thought they didn't even exist.  Because the backlog got so large (80+ full-length builds) I opted to restart the master, wiping those out of the queue in order to get the tryserver back to decent turnaround for all platforms.

I'm still ironing out some issues, tweaking the email results, and I've temporarily disabled the web interface as I work on getting it to use hg push so that all the inputs to tryserver arrive in the same way.  I'm also trying to get more slaves to add to the builder pool since the packaged unittest builders are builder hogs. 

Let me know if I've missed anything.  It's my goal this quarter to make tryserver as helpful as possible in keeping mozilla-central green. 

Week [1] - Learning to set up Buildbot

Okay, it's time for another update as to my activities in MV.

By the time Robcee left last Friday to go back to the picturesque province of New Brunswick, I had come pretty close to having Buildbot installed and ready to be deployed on my CentOS VM.

Today in between wrestling with trying to build on Vista with VC9 (I must remember to "run as administrator" on the mozilla-build shell) I have been learning to deploy my own local buildbot master and a linux slave.

In getting to this:


there were a couple of roadblocks.

1. The current master.cfg and mozbuild.py are written for Builbot 0.7.5 and so I had to make a couple of small changes as per the documentation to account for import changes (no more step, using steps.shell or just steps instead) and also html.Waterfall is deprecated we should use html.WebStatus now.

2. It took a little while to figure out that the "force build" option that you can get when you click on a slave's name link in the waterfall is actually an option passed in to html.Waterfall as in html.Waterfall(http_port=2005, allowForce="true") and this gives you a nice little html form where you can force a build as you need one.

3. The last little glitch was just making sure that all the names matched up. The major lesson learned here was: Never follow directions. Just because someone says to do

buildbot create-slave slave localhost:9989 linux mozilla 

doesn't mean that will work. You have to look beyond the literal instructions and realize that the slave name must match what is in auth.py and same with the password.

Next goal - learn how to kick unittest machines when they misbehave.

Source Server Tweaks - Now with refactoring and a clean_root function

Things to remember:
1. When you post your patch you select review ? and not review + (I had been confused about why I couldn't put my reviewer's email address in next to the +)
2. When your previously working code stops working suddenly and print statements galore are not helping, and you know what part isn't working but not why...Stop poking at the code and go to MXR -- Thank you MXR for helping me catch (through a line by line comparison) that I had accidentally deleted a key line thinking it was my own addition. I think this calls for an editor that does coloring on text not for syntax but for diff'd text.

Tomorrow should be a big day. I'll be in a little box office at the Royal Cinema all day and night working for Cinéfranco - if anyone wants to see a French movie, I can hook you up - and I'll be anxiously awaiting a review result because this is it people, these are the tweaks that should net me a test version of a nightly build.

Fingers crossed that one particular Build guy will be working on Saturday...