Final Demo of Source Server for Mozilla

Click to Play

This is the demonstration I showed today for the final wrap-up of my DPS911 class - "Open Source Project".

The Source Server will be ready for public consumption

Okay.

As I download VS Express so that I can test it in that environment, I've uploaded two patches in the continuing saga that is Source Server.

Here's what we've learned since my last post:

* cvs.exe that comes with mozilla-build has issues so it's necessary to point your path to a standalone version
* the tinderbox cvs_root uses private key access so we have to alter symbolstore.py to check for an environment variable of SRCSRV_ROOT which will be set in the tinder-config.pl file to the public :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot.
* the reason that %fnchdir% wasn't working in my VS was because the srcsrv.dll it was using was an earlier version. once I copied the version from WinDBG over to the VS devenv folder, it worked. Still don't know where %fnchdir% came from (googling it only turns up myself) but I am glad it's working now and hopefully VS2008 express comes with this newer version - otherwise the documentation will be instructing people to do all sorts of extra work
* the last piece of the puzzle - Why Doesn't the Code Show Up After Downloading? - turns out there's a little checkbox in the options for debugging - something about requiring the exact file match...uncheck that and VOILA! Source Server worked in Visual Studio!!!

This is awesome. I am so thrilled that when the two bugs are resolved, downloading a nightly and debugging it should be really really easy!

Time to update some screenshots :)

Tweaking locally - ftw

So in order to get the current version of nightly builds to work for me, I had to add a srcsrv.ini file in the same place as the srcsrv.dll and devenv.exe - on my computer this is C:\Program Files\Microsoft Visual Studio 8\Common7\IDE

In that srcsrv.ini the only lines needed are:

[variables]

MYSERVER=:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot


The reason for this is that anything in your local srcsrv.ini will override what's in the pdb file. In the current situation, the pdb files for the nightly debug builds have MYSERVER=:ext:ffxbld@cvs.mozilla.org:/cvsroot since that is what the computer they were built on used. That cvspath does not work without a key though, so it is no good to the average user of source indexed builds.

The second tweak is still making sure that the code is being located properly once it's downloaded. There is a way to change your registry to alter where Visual Studio looks for its cache of source code. I tried this and it worked but I'm not sure it was necessary. It's all still a bit of a grey area.

However, the silver lining? The patch does work - with the tweaks which are pretty simple - so source indexed builds exists and hopefully some people will start using them and familiarizing themselves with source server. In the long run, the more people using it, the more people who will be able to hack on it.

Thanks especially to this article which helped tonight as I tried to find the right combination of settings.

Now it's time to go write some things into the documentation.

Testing the indexed nightly

So the fix worked and now the Mac/Linux |make buildsymbols| functionality is working again. As well, the nightly debug build from last night had source server indexing.

I downloaded the windows nightly to test it - in both WinDBG and Visual Studio 2005.

All looks good on VStudio - the symbols download and then when I try to break debugging I get this:
"This is great!" I think to myself and I happily hit "Run" several times to the various prompts for cvs commands.

Then...NO Source code can be found anywhere. Why? Well, it didn't download with that cvs command. See how it's got ":ext:ffxbld@cvs.mozilla.org:/cvsroot"? Well that doesn't get me any code, what it does get it timeouts and errors like this.

So now what? I was able to run the command with :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot and pull the code files to their default location - which by the way is not anywhere you would find it by accident. Here's what I found out about the location that SourceServer puts your code by default:

You may be wondering where Visual Studio puts the source server download cache. The default location is C:\Documents and Settings\Username\Local Settings\Application Data\SourceServer. This is great because with proper security in place only logged-in users can see that directory.

http://msdn2.microsoft.com/en-us/magazine/cc163563.aspx

So I ran the command in that directory and when VStudio is done asking me over and over again about running cvs commands - I do get the source file I checked out appearing in my VStudio solution.

What now? I don't know if this is something that needs to be changed in the mozilla code or if this is now about doing local changes with a srcsrv.ini file. Hopefully I will get some good advice about this soon so that I can demo it tomorrow without too much hacking and with a little bit of authority even :)

Linux headache...

So my patch was backed out because it broke make buildsymbols on the Mac and Linux platforms. From looking closer at the error message and the patch, I deduce that it's GetVCSFilename containing the offending line - a return file where a tuple (two return values in python) should be returning, so I change it to return (file, None) and after testing the patch on a clean Mac build I am presented with the successful making of buildsymbols.

Now enter evil Linux (cue evil music)- the one platform I have yet to build on in any serious fashion. I should test this patch on Linux to make sure that it works there too. Here's the bullet form of what happened (as best I can remember at this point in head bashing on keyboard):

1. I ssh to 'Liberia', one of the Quad's running Fedora in the ORI lab
2. Check out, apply patch and build a debug build on it - fast! It takes ~3 mins to checkout and ~13 to build
3. Run make buildsymbols and I get "make: nothing to do for 'buildsymbols'" (thanks.)
4. Clobber build, tweak .mozconfig adding --enable-application=browser
5. Build - run make buildsymbols - still nothing to do for buildsymbols
6. After attempting to bike home, getting a flat tire from a massive pothole on Keele St and having to bus it I ssh into Liberia again and start from scratch. At this point both Armen and Dave start builds too, all of us using --enable-debug and other assorted .mozconfig settings
7. No one can make buildsymbols. Without the patch there are errors and with the patch - same errors.
8. In objdir/config/autoconf.mk I check and see that MOZ_CRASHREPORTER = 1 so make should be recognizing this.
   
9. I try a clean checkout and can't even build because of errors with dump_symbols.cc

Swimming in a sea of red herrings is not fun. Tomorrow's another day and another stab at getting this going. At this point I'm convinced that there's something wrong with locally building and not in fact with the patch since it does not touches dump_symbols.cc and since today's Linux symbols are up and fine (ted checked) which means that make buildsymbols does in fact work somewhere...just not for me (or Dave).

Build & Release - Part Two - more meetings...

Continuing along with the education of future Build and Release engineers, Armen and I were introduced to automation with Robert Helmer. Rob's leaving Mozilla in a couple of weeks and it's too bad we won't get to work with him this summer. He's really got this all figured out and most of his presentation was way over my head. I said to Ben Hearsum later that it feels like once I'm actually touching the build system it will probably make more sense, but just talking about it? That's a bit confusing. In any case, Rob presented for about an hour on the automation process from past to present and then there was a lively discussion about what's coming next.

The next day Ben Hearsum talked about Try server and showed us the current set up. I'm really excited to learn more about Try server, and also to having permissions to use it myself. Considering that my recent patch had to be backed out for breaking buildsymbols on the Mac and Linux platforms the Try server could really help me out. I'll be talking with Ben about how I can get in on it. I have an LDAP account now...

Following Ben, Nick Thomas showed us the flow of releases from tinderbox to virus scanning to their final resting place on the ftp servers. In his diagram you can see pre Sept 2007, how it is currently and then on the right - what we're aiming for.


The last presentation of the day was Rob Campbell with his presentation on unit tests. Again we were treated to a thorough explanation of how things were done, how they're being done now and where things are heading.

The most exciting part of the week was realising that there is still a lot of room in Mozilla to have an impact on how things are done. On Wednesday morning we went for breakfast and Armen and I were lucky enough to be sitting next to Mike Schroepfer - VP of Engineering for Mozilla. He shared a lot of information with us and also pointed out how every single person at the table (the Build, Release and Automation team) has contributed significantly. That Mozilla has been able to grow as quickly as it has without falling down is a testament to all their hard work (and that of many others) and I am inspired to be part of that.

Thanks to Dave and to John for helping organize this opportunity. I think that my transition to interning this summer will be much smoother as I have met many of the people with whom I'll be working, plus the overviews of each area of Build were invaluable.

Build & Release - Learning about Talos

Armen and I are in California attending Build and Release team meetings this week. Over the next two days we'll be introduced to the many facets of the Build and Release workflow.

Todays first session was about Talos with Alice.

Here is the diagram of Talos (copied from the diagram Alice drew - yes Talos is a robot):


Alice walked us through how Talos gets its information from browser builds by having buildbot read for new builds from quickparse which is a text file. Buildbot has a script that knows what to look for in order to find new builds and there is a 5 minute delay before Talos is deployed because the information can get into quickparse before the build is finished and so therefore does not technically exist.

Currently there are 30 Production and 20 Stage Talos machines running, this past December there was only 1 Production machine and the stage machines.

This huge increase of Talos machines has led to an insane amount of data being gathered and a database which is in serious need of some help.

After Alice's presentation we all tried a standalone Talos so we could see the tests at work.

Anyone can try them, just follow these directions. If you are using a recent nightly you might need to add security.fileuri.strict_origin_policy : false to your sample.config file preferences because of new security features. Also, you can comment out the tjss tests because those are kind of boring - the fun test is the svg since you'll see a lot of graphics tests running on your browser. This standalone runs on a new profile so it's okay if you already have Firefox running when you run this script.

So the information that's generated is good for recognizing regression like in this bug, where if you look at the graph you can see how the build was chugging along, something got checked in that affected performance and then it was backed out and the performance went back to normal.

Pic here, see bug #425941 for more info:


More information on Talos Machines.