Crashing Into Open Source (without a paddle): May 2010

Wednesday, May 26, 2010

Tryserver 2.0 - Fine tuning and learning the hard way

I really thought the try server as a branch was ready to roll out when I did it. Seriously. It took a couple of months to get it to the point where I felt it was ready for the public. So I pushed it live last week - detail here.

Immediately a few issues came up that ruined my vision of a smooth transition:

* Emails were going to the changeset author and not the person pushing the change. Turns out I had changed this default behaviour and didn't know that there was a reason why we used to grab the email address from buildbot sendchange info instead of from hg author.

* Packaged unittests were being merged. This was quite a big deal for the first full day of being live because people were getting all kinds of strange emails about results that weren't theirs. The reason this wasn't caught in testing was that there is no 1:1 mapping between a build and its packaged unit test results (except when you go look at tbpl). We should get one set up for MozillaTest where our staging results go in order to make sure something like this doesn't happen again.

* The URL included in an email regarding test results didn't contain a changeset - only the build and leak test builders were setting the information needed to grab this for the emails. Thanks to dbaron for catching that quickly and bringing it to my attention.

* The try-mac slaves got delayed on their way to the new tryserver's slave pool because of some glitches with puppet and so a huge backlog formed for OS X builds and people thought they didn't even exist. Because the backlog got so large (80+ full-length builds) I opted to restart the master, wiping those out of the queue in order to get the tryserver back to decent turnaround for all platforms.

I'm still ironing out some issues, tweaking the email results, and I've temporarily disabled the web interface as I work on getting it to use hg push so that all the inputs to tryserver arrive in the same way. I'm also trying to get more slaves to add to the builder pool since the packaged unittest builders are builder hogs.

Let me know if I've missed anything. It's my goal this quarter to make tryserver as helpful as possible in keeping mozilla-central green.

Thursday, May 20, 2010

Tryserver 2.0 is Live

This morning the quietly running in the background try-as-branch became the new try server and the try you've known for a few years now is about to be turned off.

You will now be getting opt and debug builds, as well as packaged unit tests for all current try platforms (64 bit are on the way). If you would like to read more about how the new try server works you can look to https://wiki.mozilla.org/Build:TryServerAsBranch where how to add a custom mozconfig and where to file bugs on tryserver issues is covered, as well as other info.

Now that this new tryserver is up and running, the remaining try slaves from the current tryserver are quickly being moved over and we will be back to full speed on try builds in the next few days.

Upcoming work on try will see 64 bit builds, sending the unit tests over to talos user desktop platforms for testing (just like mozilla-central has now) and also allowing you to select with more granularity which try builds/platforms you would like your patch to be done on. Also there will be a try branch for 1.9.2-based builds coming soon - tracking bug for that is 563822

If you have questions, come find me (lsblakk) in #build or #developers or you can file bugs on tryserver.

Speaking of bugs, this new try-as-branch will allow RelEng to close the following pretty much right away:

I really appreciate everyone's patience during this transition, I hope that the new tryserver gets us better results and helps keep trunk building green.