0.3 Release of the Source Server for Mozilla Firefox

Well, the big ball 'O duct tape that is my hacks on symbolstore.py are complete for now - here is the patch. This project will be carried on in the next term at which point I hope to address some of the current issues, namely - manipulating the directory structure, cleaning up the python code and assembling a unified documentation page.

For now - you may test out the fruits of my labour by downloading this installer of Firefox 3.0b2pre which will install Minefield in your Program Files/Minefield directory (you have no choice for installation dir.) and then copying these symbols into the folder where your Visual Studio (or Windbg) is looking for debug symbols.

The last thing you have to do in order for this to work is to put a path to cvs.exe in your PATH - I personally pointed it to the mozilla-build/mysys/bin/cvs.exe but if you like you can download cvs.exe and put it wherever you want.

Now that you have all you need, open up your debugger (VS or Windbg), attach it to the Minefield process and go!

It will ask you about the cvs calls that gather up the source code, say "OK" or "Run" or whatever is positive. The file should download into the proper /mozilla/whatever structure - on my computer it goes into Program Files/Microsoft Visual Studio 8/Common7/IDE/mozilla/...

That will be a focus in the next round, is manipulating the cvs command so that the source files go to a c:/temp or something. The debugger will automatically want to look for the path that the source file originally came from (in this case, my c:/ff_clean/mozilla/...) - you will have to point your debugger to the correct file the first time and then after that it knows to look in the mozilla folder you selected.

It works for now like what it is; a ball of duct tape that allows you to get the right source code for where you are in the browser without having a build on your computer.

I look forward to feedback about how this works for you, if you use it and if you have any problems arise.

To the wall...

After many nights of tweaking this space, that path, this logic - the symbolstore.py script is now playing nice with pdbstr and the pdb files are getting indexed. So begins the second battle - how to get Visual Studio to pull the code out of CVS and locate that code on the local drive. Right now, when you have the symbols loaded up and you attach to firefox as a process, a window comes up confirming the call that will be made.

If I copy and paste this command into my command line (I have CVS in my path from mozilla-build\mysys\bin) then I can pull the file and it goes into the directory structure as it is in the command starting from my current working directory.

When I click on "Run" in Visual Studio however, the code may or may not be getting pulled from CVS - I have no feeback and if I then try to look at something on the call stack, I get a dialog asking me to find the source code file. BUT WHERE IS IT?

Visual Studio is trying to find it in the hierarchy that my build was based on...that is not going to exist on the remote user's computer.

I've made a few tweaks to symbolstore.py to try and create a %CVS_WORKINGDIR% in the srcsrv data block where the code would then be pulled to. This does not appear to be working, as the directory I am instructing it to create is not getting created.

This is the current symbolstore.py. More tweaking on the horizon.

Many huge thanks to Ted for figuring out the minute details that fixed the call to pdbstr.

The most up to date set of symbols can be found here: http://avnerd.tv/symbols/crashreporter-symbols-firefox-3.0b2pre-WINNT-2007120422.zip

The installer for this indexed version is here: http://avnerd.tv/sharedFiles/ff_SourceServer.exe.zip

Symbols and Installers are giving me a headache.

I'm not sure where things are going wrong right now so I am hoping that you - faithful readers - will be able to help me out. I have run the "make buildsymbols" and checked with srctool.exe to confirm that the pdb files are source indexed. So following that I made an installer to see if I could attach it in Visual Studio and pull some code. However I am not getting the symbols to load.

Here's what I have:

1. A modified symbolstore.py: http://pastebin.mozilla.org/255918


2. Symbols and source indexed .pdb files here: http://avnerd.tv/symbols/test/crashreporter-symbols-firefox-3.0b2pre-WINNT-2007120422


3. An installer for the build I was working on when I did this: http://avnerd.tv/sharedFiles/ff_SourceServer.exe.zip

When I point Visual Studio to the symbols I can't get anything to load up. So this means that I am unable to test if the source indexed code's cvs commands actually work.

There weren't any gotchas the last time I uploaded symbols - so I'm not sure where to begin in figuring out why Visual Studio isn't grabbing them properly.

I have to work all day tomorrow - and I have a big assignment still to work on for my BTS system design class, so anyone who can take a minute to poke at this and give me any feedback would be a hero!

Thanks!

3 main barriers to a complete 0.3 release of the source server

Here is the code I have so far in symbolstore.py
http://pastebin.mozilla.org/254868

My portion is specifically lines 36 - 71 where I am now gathering up a string of source files along with the revision number, and generating a (almost) properly formatted stream file in the pdb's directory.

Here are the barriers:

1. I would prefer to have a template where I have a variable %SOURCEFILES% that can be replaced and output to a new file for each pdb, instead of writing out each line of the stream file in symbolstore.py. If I can't get that happening for this release - oh well, Dave said it can be a bit ugly at the 0.3 stage. At least what I have right now, is creating the stream file properly.

2. On lines 38 and 39 - what I am making here is a string that looks like this: c:\mozilla/xpcom/glue/nsTraceRefcnt.h and in fact, this has to be c:\USERS_SOURCEFILE_DIR\mozilla\xpcom\glue\nsTraceRecnt.h -- so two things need to happen here, first, I need the / to be \ (except for the one after c:, cause it's right already) and second, for now I can probably name the dir "ff" and the user can make sure that VStudio is pointing to that path (c:\ff) when looking for source files...but in the long run? I don't know yet.

3. This is along the same lines as the template issue. The reason I haven't successfully implemented the template is because I don't know where to put, or how to create a path to, the right folders when symbolstore.py is being called. Unfortunately, it is not as easy as I had hoped in that putting the template in the same folder as symbolstore.py would allow me to open it with its name only. So as well as not being able to open the template, I don't know how I can call upon pdbstr from inside of symbolstore.py

That's going to be the last part, the first two are a little more pressing.

Source Server home stretch...on the road to mere python coding

Huge changes since last night's post.

I can insert a .stream file (see previous post for what a .stream file data block contains) into an unindexed .pdb file via pdbstr -w as long as it has certain variables.

I still need to test what the minimum info needed is, right now I just copied and pasted from one that works. I will slowly delete variables from it until it doesn't work anymore.

Using cvs to pull a specific revision of a file looks like this: cvs export -r 1.23 filename

So - in my test.pdb.stream file I changed the CVS_EXTRACT_CMD to :

CVS_EXTRACT_CMD=%fnchdir%(%CVS_WORKINGDIR%)cvs.exe -d %fnvar%(%var2%) export -r %var4% -d %cvsdatetarg% %var3%

(p.s. %var#% are based on * delimited text in the source files section)

In the source files, i add a source file that contains cvs in the symbol file
like so:

SRCSRV: source files ---------------------------------------
c:\ff\mozilla\accessible\public\msaa\AccessibleMarshal.def*MYSERVER*mozilla/accessible/public/msaa/AccessibleMarshal.def*1.3

Then, when I write that to the accessiblemarshal.pdb file followed by a call to srctool -f on that .pdb file, I am greeted with a source indexed file that contains the cvs command that the source server will use to extract the source code when the time comes.

The result looks like this:

[c:\ff\mozilla\accessible\public\msaa\AccessibleMarshal.def] cmd: cvs.exe -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot export -r 1.3 -d 12-01-07 mozilla/accessible/public/msaa/AccessibleMarshal.def

c:\ff\mozilla\objdir_debugInfo\dist\crashreporter-symbols\2007120119\accessiblemarshal.pdb\8EEE4D7316254E22AE3D99EB4082744C3\accessiblemarshal.pdb: 1 source files are indexed - 199 are not

So now I need to learn how best to add this into the symbolstore.py file.

My first instinct is to add something in the ProcessFile function that keeps track of all the files that start with cvs, they look like this:

FILE 2 cvs:cvs-mirror.mozilla.org/cvsroot:mozilla/accessible/public/msaa/AccessibleMarshal.def:1.3
FILE 3 c:/program files/microsoft visual studio 8/vc/platformsdk/include/RpcNsi.h

I already did a little playing around with the python script and was able to insert some code here where I wrote if filename.startswith("cvs") then some debug statement to see that I could in fact catch the right lines (which I did).

So let's break it down:

1. I need to track all lines that start with "cvs" and then turn these into proper file paths



2. I need a (minimal) template .pdb.stream file that I can write the source file paths to in source files section



3. Then I will call pdbstr -w and write that .pdb.stream (each named for the pdb it will write to) to the appropriate pdb file

The final hurdle - learn to write good python code to make this all happen.

Moving pdbstr.exe into the mozilla environment

Working on 0.3 release of source server is all about getting the pdb files gathered up when symbolstore.py is made and then feeding them to pdbstr.exe. I've copied the binary into the mozilla/toolkit/crashreporter/tools folder where symbolstore.py is called and I've spent the last 4 hours or so playing around with python.

So far I have learned this:

1. THIS IS WRONG, PLEASE IGNORE That I can call pdbstr.exe on an unindexed pdb file and it will in fact read and generate a data block that has all the information needed to set up the cvs cmd calls - see this output from calling pdbstr -r -p:thepdbfile -s:srcsrv > thepdbfile.stream ** for corrections see next post **

2. The next part would be to write this block back to the pdb file i think - so that it's available to the source server

This is what calling pdbstr -r gets me:

SRCSRV: ini ------------------------------------------------
VERSION=2
INDEXVERSION=2
VERCTRL=http
DATETIME=Tue Nov 27 03:56:53 2007
SRCSRV: variables ------------------------------------------
CVSDATE=11/25/07
CVSDATETARG=11-25-07
CVS_WORKINGDIR=%targ%\%var2%\%fnbksl%(%var3%)
SRCSRVVERCTRL=http
MYSERVER=:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot
HTTP_ALIAS=http://avnerd.tv/source
HTTP_ALIAS=http://avnerd.tv/source
HTTP_EXTRACT_TARGET=%HTTP_ALIAS%/%var2%/%var3%/%var4%/%fnfile%(%var1%)
SRCSRVTRG=%http_extract_target%
SRCSRVCMD=
SRCSRV: source files ---------------------------------------
c:\ffDebug\mozilla\accessible\public\msaa\AccessibleMarshal.def*MYSERVER*mozilla/accessible/public/msaa/AccessibleMarshal.def*1.3
SRCSRV: end ------------------------------------------------


So the next goal is to successfully keep a list in symbolstore.py of all the pdb so that pdbstr can be called on them.

Eclipse and keycode mapping

Okay, this is take 2 because my first post was unceremoniously 'disappeared'.

My BTS team is getting geared up for writing Unit Tests in JUnit. We're using Eclipse with subclipse for SVN. Two teammates are Windows, one Linux and me on the Mac. Mac's version of Java is Apple's not Sun's so my JRE profile is not compatible with theirs. This means I will be using my Linux VM for the project - getting to know Ubuntu a little better.

Today's the first day of significant coding and I am already frustrated that I have to reach back to use the ctrl key instead of my trusty apple key for shortcuts. Also the tilde key actually produces < and there's no delete functionality (what Mac calls delete is really backspace).

So I have found a solution and here it is:

First, create a file (mine is called wickedCoolKeycodeFinder - you know, so I can always find it again)
and in that file put this:
xev | grep -A2 --line-buffered '^KeyRelease' | sed -n '/keycode /s/^.*keycode \([0-9]*\).* (.*, \(.*\)).*$/\1 \2/p'

chmod +x this file and then run it to produce a neat little window that will tell you what keycode is returned for whatever key you press.

Then, once you know the keycodes of the keys you want to map, create a file called .xmodmaprc and in that file I put:

!this sorts out the apple key + backspace to do delete
keycode 113 = BackSpace Terminate_Server Delete
! map tilde and grave
keycode 49 = grave asciitilde
!map apple key to ctrl
remove Control = Control_L
keycode 115 = Control_L
keycode 116 = Control_L
add Control = Control_L


Now I just need to figure out how I can get my two-finger scrolling on the touchpad to work and I'll be a Linux user forever!

Circling, like a tiger ready to pounce

So for my 0.3 release of sourceServer, I am tasked with altering the code in symbolstore.py so that it will track the source files and then call pdbstr.exe on them.

To do this, I need to understand what PDBSTR.EXE does. This is what I have to go on:

Once the code has looked at all the source files in the PDB file, it's time to call PDBSTR.EXE to write the index stream, called SRCSRV, to the PDB file. When debugging, the debugger looks for this stream. If it is found, the debugger knows there's a source server involved and calls into SRCSRV.DLL to execute the version control system to ensure the right file is accessed.

I've been studying the pdb files pre and post indexing to gain insight into what happens when I use the Debugging for Windows srcsrv tools. Circling the symbolstore.py code getting ready to make changes to see what I can do.

Any minute now the big picture will come into focus...back to my pdb files.

In case anyone was looking for power point slides about the Source Server...

Just a quick post to share my SourceServer Presentation if anyone's interested in an overview of what my project this term in DPS909 has entailed.

XPcom Lab II

Well there was an initial kerfuffle trying to get the extension "firstxpcomchrome" working in my Minefield build. Turns out that when I was attempting to connect it up via a text file in the profiles/myprofile/extensions folder, I had put the path as a c:/.... path which, in Unix (ie Mac), is not correct. This was affecting my ability to install the extension through drag n drop because the file for firstxpcomchrome@senecac.on.ca was not being overwritten so even though the extension appeared to install correctly in the add-ons manager window, upon restarting it was nowhere to be seen. Deleting the erroneous flat file and doing a drag and drop installation cleared this up and I was able to continue with the lab.

Aside from crashing the browser every time I called on "Reload All Chrome" from the Extension Developer add-on, the lab went smoothly.

Results:
(please enjoy the File menu behaviour that is happening in the background)
Call


Response

Source Server 0.3 goals

Well, it turns out that my 0.2 release was a little buggy but Ted (luser) was able to hack his srcsrv.ini file to overwrite the URL I had indexed into the pdb files so that he was able to pull code from my server. Instead of hacking the 0.2 and fixing that though, I am moving forward with 0.3 which is to put into symbolstore.py the ability to do the cvs pulls.

This is the most complicated part yet, for me, because I am not too familiar with python and because I still see all this code as a nebulous cloud and finding the path from A to B is difficult.

So my steps toward my 0.3 goal so far have been:

• Reading up on Python and how it works with some: Python Tutorials


• Studying the logic of symbolstore.py

And coming soon, some hacks of symbolstore.py to see what kind of havoc I can wreak.

Xpcom success

Thanks to Armen's post and all the sub helpers that led to his post, I did successfully install the Extension Developer add-on and I managed to get my firstxpcom extension working in the JavaScript shell:

XPcom - not right now.

Well, I greatly enjoyed Benjamin Smedberg's talk - power outage and all - and much was learned about Xpcom. Not enough however, to enable me to complete the Xpcom lab that was assigned to us last week.

Last week I pounced on the lab, raring to go with my existing build of firefox and ran into a wall - something is just not making right. Some poking around, consulting with those who have done the lab generates no leads since the trouble I'm having is of course, unique, and so they do not know where I am going wrong.

Fine. I can handle this. I will do it all from scratch, checkout code, rebuild, start the lab and guess what? Still can't do it. Currently I am stopped by this:

make[1]: *** No rule to make target `install.rdf', needed by `libs'.

I don't understand Xpcom enough to really troubleshoot this so I think this lab is going in the "to-do later when you know more" pile for now.

Newsflash, I never created the install.rdf file - so now it's working to the point where I can see it in my Add-ons manager. However! Ted's Extension Developer add-on does not work with my build because it does not support secure updates...must be a new ff3 feature...so now I have to wait until tomorrow to ask luser if there's a work around.

Source Server 0.2 release - HOWTO and Contribution Opportunity

Here are the goals for my 0.2 release

• release build of ff with debugger-info enabled to create pdbs without being a full debug build


• using lib xul - make an installer for people to download


• take the symbols and index them and distribute the source


• set up the symbol server but indexed so that someone can pull the source code


• which involves running the cv2http.cmd on the indexed code

And the results:

How to set up the Source and Symbol Server

1. add a path to my symbolServer http://avnerd.tv/symbols/2007110200 as per MDC docs in Tools->Debugging->Symbols
   
   
   


2. check the option for turning on the source server in Tools->Debugging->General
   
   
   


3. Download the installer from here Firefox Installer **(FYI you have to use my installer because it is the one that I indexed code and uploaded symbols for - if you do not use this installer you will not get the proper results)



4. install the firefox you just downloaded (it will install as Minefield in Program Files\Minefield)



5. make sure you run it with --no-remote and -profilemanager so you don't mess up your normal Firefox experience



6. once it's running, go to Tools->attach to process... and attach VStudio to the firefox instance



7. after it's attached, you can go to Debug->break all and then poke around at the firefox functions code (You will get a warning about downloading code from an unknown source, click on "run" because you trust me)

This is what it might look like:

Contribution Opportunity

You need to be in the Windows platform and have a copy of Visual Studio to run the above instructions. If this applies to you, please take a minute to download the installer and follow the instructions as I've outlined them.

Please leave a comment answering the following questions:

1. Did it work for you?


2. Were the instructions clear?


3. Any constructive feedback?

Contrib Opportunity! Help with resolving a path with spaces and a perl script

I have a perl script called cv2http.pl that is called upon by cv2http.cmd

Both are in c:\Program Files\Debugging Tools for Windows\sdk\srcsrv directory, which is where I am making this command line call from.

I am trying to do this:

walk.cmd c:\symbols\*.pdb cv2http.cmd HTTP_ALIAS http://avnerd.tv/source

What this will do is swap out the cvs information and replace it with a path to my server where I have the source code.

When I run it I get this:

"Can't open perl script "C:\Program": No such file or directory

for every .pdb that is found.

It looks like it is breaking on the space between Program Files in the path.

So - does anyone know how to either: make this script be forgiving of blank spaces in a path or change the directory this points to so that I can move it and try from there.

**** UPDATE *****
Turns out it was as easy as adding "" around the ~%d0\cv2http.pl call in the cs2http.cmd file - thanks to all who helped on #seneca

Test Day X 2

Well today I got to run Full Functional Tests on Firefox 3 beta 1 and then this evening as I procrastinated from dealing with my Hamiltonian circuit assignment, I ran tests on Miro's 1.0rc release.

The two testing systems are very different even though they both use Litmus-based testing. Miro's tests are fewer, some are not well written and in some ways I find them to be very superficial. The Firefox tests have been around a lot longer so I'm sure that has allowed for quite a bit of fine tuning.

I'm not giving up on Miro but so far it's been challenging to get involved with that project to the degree with which I have been able to immerse myself in Mozilla projects. The lab the other day threw me into Thunderbird in a quick 'n dirty way and that was great - the documentation is all there. With Miro - well...when I went to look for docs on how to build with Windows...this and this. The gist of it is: "We can't really tell you how to build on windows because it's hard".

Well, now that I'm working on the Mac Book and I have VMWare Fusion with an Ubuntu vm, perhaps I can try building it in a Unix/Linux environment? There's a doc for Mac OS X and for Linux. We'll soon see.

Now, back to Mozilla stuff. My 0.2. release is due soon, I have already done a non-debug build of FF with the --enable-debugger-info-modules in the mozconfig and I've made the buildsymbols for it. Now I just have to index the pdb files and get the symbols and the source up on my server. Immediately after that is done (in the next day or two) I will look into a) how to make an installer and b) how to work with the CVStoHTTP script that comes with the srctools.

When I have an installer up, I'll be putting out the contrib call for anyone who can to grab the installer and try to hook it up with my symbol and source servers.

Stay tuned.

Thunderbird Bug Fixing Lab - an up close and personal lesson on patch review

Filing a bug

First I had to create a bug over on landfill - explaining the issue

The gist is that anything that has characters@...Someothercharacters turns into a mailto: link in Thunderbird and this was annoying Dave so the whole class jumped to fix it.

Fixing the problem

Dowloaded and built a copy of Thunderbird's trunk on my MacBook and then navigated over to mozTXTToHTMLConv.cpp in order to alter some code which checks for a '.' after the @ symbol but does not also check that there is not a ".." in that string.

This is a one-line fix but I was having some trouble getting my code changes to show up. Funny story, this happened to me in class too and both times it was because I forgot a ) in the code.

Creating a patch

This is easily done by calling cvs diff -u8p . > mailtoPatch.txt in my mozilla directory.

Requesting Review and the Results

We were going down the line as reviewers - I asked Mmullin and he had already done a review so he handed me off to Armenzg who passed my patch with the comments to remove my printf statement and to make the two if statements into one evaluation. It turns out that he should not have passed my patch, instead he should have made the comments and let me fix it before approving the patch.

Even though Armenzg passed my first patch, I did in fact re-do the patch minus printf and simplifying the if statement. I am waiting for a proper approval as I write this but I am sure it will be approved.

After it was all over, I was then the reviewer for Peter's patch and I was impressed with his one line change and passed it immediately.

The whole process was a great learning curve. Now that I've seen how the patch/review process works I imagine that I could fix a bug one of these days.

New MacBook with no Boot Camp Utility?

Some of the newest Santa Rosa chipset MacBooks have been deployed without Boot Camp Assistant in the Utilities folder. I'm sure this will be fixed soon. Here's the quick fix if anyone else needs it:

Put the Install Disc 1 in, then in Finder, to to Go -> Go To Folder... and type:

/Volumes/Mac OS X Install Disc 1/System/Installation/Packages

in the dialogue box. There you should see BootCamp.pkg. Running that with all the defaults puts the Boot Camp Assistant in /Applications/Utilities/ (source)

So glad I wasn't going crazy...spent a lot of time poking around my new MacBook muttering "but it's supposed to have boot camp!".

The to-do list goes into high gear now as I have a couple of programs to write for BTP500, some database stuff due for BTS530, and of course all the DPS909 labs and project updates.

All this and setting up the new mac with a windows vm so that I can continue to work on the source server with Visual Studio.

Something to show...

For my 5 hours between classes on Friday. Hopped into Litmus and joined in on Test Day. Managed to do 58 tests and filed my first bugs: 402219, 402228, and 402238.

I feel very accomplished. It's fun to find a bug, and I had an especially great crash that I was able to reproduce 3 times, and after Vlad's talk yesterday I am also looking forward to working on a patch and fixing a bug.

The past few days I have also been testing the waters over at Miro. What a difference size makes. Miro has only 11 staff - how many does Mozilla have? I would guess at least 20 or 30 times more. So far I haven't done a build of Miro yet, I'm going to work on that over the weekend. Can't wait to look at the source code and see what makes a video player. The funny (and somewhat frustrating) thing is that I have had to create new bugzilla and litmus accounts for Miro as my Mozilla ones don't carry forward. Someone hurry up and get that OpenID stuff working on everything!

Reverse Engineering

Since FSOSS my brain has been churning with many thoughts about my place in open source. According to Dave in this week's class we are all more elite than we may think and we all have potential to do great things in open source. It's hard to think that I can be in any way elite when I just failed a test in my data structures class...this is the first time I have failed anything school related and it doesn't inspire a sense of "I can take on the world". However, I know that it's just a glitch in my otherwise strong school record and so I will get beyond this.

What I've been thinking about lately is how I came to be in this program, in this open source class and how I seem to be experiencing a bit of a feeling that I am reverse engineering all the concepts that make up the open source community. My background *is* community whether it's theater, experimental films, activism, journalism, or just cultural - I have been and continue to be very active in a lot of communities. To me, a lot of the discussion about open source seems to be directed at technically sound cowboy coders who don't know how to work with others, trying to extol the virtues of a larger community and their immersion in it. I'm in the opposite lane - I know that community works, how it works and why I should be involved with it but I am constantly playing catch-up with the technical knowledge that others seem to eat and breathe.

Learning can be so erratic, it's challenging to stay focused and to not get discouraged at the fact that I often don't understand what it going on around me. In order to keep my head up I try to do something every day that is connected to school and open source learning. That can look different depending on the day. A couple of nights ago it was doing a build on a MacBook to practice for when I have my own. Yesterday it was signing up for Miro's tester mailing list and lurking on their irc channel (though that netted nothing...they are quiet folk).

One foot in front of the other, I will walk backwards and try to keep working on being a super user instead of just a user, try to be a contributor instead of a watcher, and keep telling myself that someday I will be an expert about something. That's the dream, that someday people will say "Oh you need _______, Lukas is the best person for that".

Long Day at FSOSS 2007

Well, it's the end of a long day. Starting out on my bike at 7:15 am and ending now, at 12:45 am (but I probably won't actually sleep until 1). I went to several talks, took lots of pictures and just wrapped up the rough draft of my FSOSS Report.

Had a great chat with Ted (:luser) about what goals to set for 0.2 and those have been posted to the project page.

That's all folks. See you tomorrow.

0.1 Release

Here's what we're looking at.

I built a version of ff that was checked out from mozilla's cvs. From this version I called "make buildsymbols" and created the folder 2007092823 which contained all the pdb and symbol files for that build.

For information on how to use the symbol server - check out Benjamin Smedbergs blog. Instead of pointing VStudio to the mozilla symbol server, I wanted to point to a local server of symbols.

In order to test whether these symbols worked, I set up IIS on my laptop and served up the pdbs locally to make sure that Visual Studio was able to pull them properly. After a quick detour to set up the MIME types on my server, this worked just fine.

Then I used the indexing tools that come with the microsoft.com/whdc/devtools/debugging/default.mspx Debugging for Windows package.

First - using cvsindex.cmd and pointing out the symbol path and the source path, the pdb files are then indexed with the location of the source code for that build.

Second - using srctools.exe I pulled a sample list of the source files that are contained in a pdb file

All of this is available in a zip file - PDB files and sample source list and I look forward to feedback from anyone who wants to look at the sample source list and discuss how the next step of connecting the cvs to the local pdb files might work. Also you can go to my wiki page to sign up as a contributor or to read more.

At the moment this will not work for someone who is debugging unless they have my build of FF. That's what 0.2 is all about.

Extensibility with Mark Finkle

Today's class was a presentation with Mark Finkle from Mozilla Corp. These are my notes.

Started off in proprietary software
Has only been with Mozilla for a year
The joy of extensions
- Primary way to get involved is through add-ons (extensions)
- Plug-ins killed the web (according to some) and are kind of discouraged
- Any app that uses the mozilla platform can be enabled to use extensions
- There's built in support in the UI, cross platform, platform APIs

Sometime people put their content in a .jar and then in the .xpi necessary for installation. Mark doesn't not advocate for this method because it can make things more confusing to new developers than need be.

He suggests keeping things as simple as you possibly can. Don't use jar files, multiple levels, keep manifests simple. Don't mimic other extensions that are often written by more experienced programmers.

Ted (:luser) has created a xul editor that can generate the skeleton for you, and if you use this then you have everything you need - just add content.

Use the pages on the MDC about the boilerplate aspects of creating extensions.

Stressed the importance of using the id of the rid firefox element to add your customized elements. Otherwise they may show up as document.getElementByID but they will not in fact appear. They will be in the ether until you merge them properly with the id. Examples are vbox id="appcontent" or statusbar id="status-bar". Not merging properly is 90% of what people get stuck on when working on the UI elements of their application and it's often that they are not using the id properly, don't have the proper id or a syntax error.

Best Slide: "Mozilla is like an Onion: It makes you cry, it smells, it has layers"

Mozilla's Tech Layers (1. XUL/JS/CSS, 2. XBL, 3. XPCOM)

Talking about layers he tells us that the top layer (the UI) is the easiest to get involved with and make changes to, but it's also the least documented and has no API. Whereas the XPCOM layer is clean and documented and has APIs so in some ways it's great to work with because it's designed to be extensible.

A demo follows about trying to dig into the code via DOM inspector.

Google searching hint - put MDC in the search criteria to help narrow the results.

Highlighting of the Code Snippets section in MDC which can really help point you in the right direction.

Thanks for coming and talking with us Mark. I look forward to working on extensions in the future.

Learning to Litmus

Yesterday's Club Moz meeting was quite productive as we tackled doing QA tests with Litmus.

I encourage more folks to come to Club Moz next meeting: Tuesday October 30th at 4:30 in SEQ2119 (next to ACS). Every Tuesday is a test day for Mozilla@Seneca and this is a great way of getting your feet wet in Mozilla participation with ZERO prior experience.

If you know how to browse the web you're all set.

Here's a quick how-to for those who might want to get a head start at home:

1. Set up an account with Litmus (yes, i know another mozilla account...one day this will all be better, I'm sure)

2. Pick the build version you will run tests for (it's recommended that you use a clean, new profile):


3. Enter your Build ID and your system info. An easy way to find out your build id is to type "about:" in the browser address bar:



4. This takes you to a page where you can select what kind of tests you are interested in running might be:



5. Make your selection and here you go - a test to run:




Many of the tests are simple things that you might do anyway.

For the competitive out there, Club Moz president Anthony Hughes is currently the 4th highest tester with 3477 tests under his belt...does anyone want to give him a run for his money?

Wearing grey socks makes all the difference, right?

What did I do?

Today, the indexing worked.

See the command here:

cvsindex -server=cvs -source=c:\ff\mozilla -symbols=c:\symbolServer\2007092823 -debug > output.txt

And the output.txt.

See how now it is indexing the files?! Cool.

So what did I do differently than last time?

Last Time:

ssindex.cmd [STATUS] : Server ini file: C:\Program Files\Debugging Tools for Windows\sdk\srcsrv\srcsrv.ini
ssindex.cmd [STATUS] : Source root : c:\ff\mozilla
ssindex.cmd [STATUS] : Symbols root : c:\ff\mozilla\objdir\dist\crashreporter-symbols\2007092823
ssindex.cmd [STATUS] : Control system : CVS
ssindex.cmd [STATUS] : CVS Root : :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot
ssindex.cmd [STATUS] : CVS program name: cvs.exe
ssindex.cmd [STATUS] : CVS Label :
ssindex.cmd [STATUS] : CVS Date : 10/09/07
ssindex.cmd [STATUS] : Old path root :
ssindex.cmd [STATUS] : New path root :

This Time:

ssindex.cmd [WARN ] : Command line option "-server=cvs" is unrecognized.
--------------------------------------------------------------------------------
ssindex.cmd [STATUS] : Server ini file: C:\Program Files\Debugging Tools for Windows\sdk\srcsrv\srcsrv.ini
ssindex.cmd [STATUS] : Source root : c:\ff\mozilla
ssindex.cmd [STATUS] : Symbols root : c:\symbolServer\2007092823
ssindex.cmd [STATUS] : Control system : CVS
ssindex.cmd [STATUS] : CVS Root : :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot
ssindex.cmd [STATUS] : CVS program name: cvs.exe
ssindex.cmd [STATUS] : CVS Label :
ssindex.cmd [STATUS] : CVS Date : 10/17/2007
ssindex.cmd [STATUS] : Old path root :
ssindex.cmd [STATUS] : New path root :

So really, the only differences between this time and last time is that I pointed to the symbols on my IIS symbolServer instead of the ones that were in the crashreport-symbols. Same symbols, different location. I think that must have made more of a difference than using -debug which did not seem to do anything (and that's probably because I just re-read the documentation and it's /debug).

Now I am going to test it and run some tools on the pdb files to see what's in them.

Going in circles

I don't know how many times I can read the 3 main sources of information about this source server business. Everytime I read them it's like being in a house of mirrors. Tonight I found a new site that refers to a source server. Once again I hear how amazing this source server is, yet no real information about how to make it happen. His final sentence - "After all, "It just works!" Why would I need to know anything more? :-)" is just poking at me as I am so in need of knowing more and can't find a single human being who knows more.

The main msdn help files state that:

Generally, binaries are source indexed during the build process after the application has been built. The information needed by source server is stored in the PDB files.

So does this mean that I need to add something to the firefox build files that will do this indexing around the same time that the buildsymbols are pulled? Around the time that the build wraps up? And if yes, then what do I put there? Why is there not ONE example of someone successfully writing a source server for their project? If this source server is so amazing (and it sure sounds like it will be) then why no examples? No discussion in the msdn community forums, no blogs, nothing.

I emailed a person who sounded like they had written one for their versioning system, he wrote back and said that it actually wasn't him, it was another team member.

Is the source server really an urban legend? Has *anyone* seen it in action?

First extension

Is done.

First attempt lead to error messages about the malformed or non-existent install.rdf file. Well, I knew it existed because I brought it into existence. So I looked around the other mozilla documents and made a replacement install.rdf - no error.

After this point though, I hit a wall because I could install the extension and yet the tabs weren't being added to the side. So I checked the Error Console, no errors related to the addtabbeside extension. After entering some dump() messages in the addtabbeside.js file I try again, starting ff with -console. Nothing to go on.

Anyway, a while later I found mullin on IRC and we troubleshooted it. He knew that the xml spacing is all messed up with you copy and paste from the wiki - I started from scratch paying closer attention to the indentation and voila!

So what I have learned from this is to be careful of indentation when I copy and paste. I look forward to my next extension, when I can really get deeper into modifying the browser, like menu items and whatnot.

Oh Bugzilla

Well I'm a little late in reporting on my experience of watching someone on Bugzilla. This is partly because I didn't read the "to-dos" closely enough and then it was because it took a few hours to find out how to watch someone. Well, asking on IRC, I was informed that you go to your user settings. What?! That is so strange. I wish that Bugzilla had a better interface where you could search by user, email, etc. Something to connect you to the people more. The current interface assumes you already know how to use everything.

Anyway, I put Ted (:luser) on my watch list because he's my main contact for the project I'm working on.

Mullin already said it, but I will say it again - the flood of emails made no sense, I had no way of knowing who commented from the subject line of the email, and basically it took less than one day before I pretty much stopped paying attention to anything from bugzilla-daemon@mozilla.org.

I am impressed with anyone who uses this system and gets something out of it.

Armen says there's no way bugzilla can change because it's too big - is that really true?

Ontario Linux Fest 2007

My Experience at Ontario Linux Fest 2007 - At the Toronto Congress Center.

Woke up at 6 am to get the dog walked and eat some breakfast before heading out to get Cesar. Thanks Cesar for being awake and ready to go!

We had to go up to Seneca first and grab the display material from Mary, who was also awake and on time. Perfect so far. Then we headed up to the airport vicinity where the google map had said the center was and I promptly got us a little lost. After checking at the gas station for direction, we were set straight and arrived on time to the conference center where we signed in, got a bag 'o shwag and found our table.



Once we were set up, I went to the "Women in Open Source" talk that Angela Byron was giving at 9:30. This was an excellent presentation and I was really excited to meet her in person and to talk briefly about her Summer of Code experience. I have been stalking that site for 2 years now and was never sure if it was something I could do or if only elite hackers applied. Meeting a real person who has done it and can speak to how it worked for her was inspiring. She works with Lullabot and thus, Drupal. After her talk I went to a "Hands on Demo" of Joomla and got to see how the other half lives. Joomla is exciting to me and I look forward to having a little bit of time to test out setting up a simple site with it.

Cesar had dutifully staffed the booth for a couple of hours so I went and took over so he could wander around and go to OpenMoko after lunch. At one point though, we plastered all the tables in both session halls with FSOSS flyers so as to get maximum awareness.

Talking to people about FSOSS was great - a lot of folks already knew about it and were coming. I spent a little time talking with Dru Lavigne and learning about FreeBSD. I even got a few install disks so maybe I'll be testing it out on a virtual machine soon. A few good networking opportunities presented themselves. I met the guy who does all the Linux on mainframes stuff for IBM and he gave me his card to contact him about IBM online learning.

In the afternoon I listened to the story of a LUG partnering up with the United Way up around Owen Sound and fixing up computers for families in need. I hear that we're doing some similar stuff at Seneca, so I'm going to be looking into how I could help with that project. What I like about this kind of project is that it can be anonymous - so that people in need don't have to feel like they are getting handouts. Also, this is a great opportunity to work on making Linux useable to people with ZERO technical skills. That's something I think really needs to keep being improved if there is going to be wider adoption. It also interests me because I need to become more comfortable with installing and configuring Linux so I can be an evangelist.

All in all, a great time was had and I came away with lots of new knowledge and curiousity about Linux and the community surrounding it plus, some new friendships in the making. I could get used to this conference lifestyle.

As a parting note:

did that penguin just move?

I will do anything

To get the information that I need to make this Source Server work.

So in the srcsrv.doc file that comes with the debugging package it says:

Anyone interested in using Source Server with CVS should send email to windbgfb@microsoft.com.

So I did.

Here's the response I just got:

Good things come to those that wait – and you will not have to wait long. The next debugger release will have cvs source indexing scripts in it. You will probably need to tweak them since cvs installations seem to differ greatly and I was unable to come up with something that was truly general-purpose.

So watch our web site for the release. It will be out soon.

.pat styles

Well that's interesting because in my sdk I have a CVS.pm and a cvsindex.cmd which is what you need to make this stuff work with CVS. So does Pat know that? Is there a better version on the way?

What I don't understand right this second is two things:
1. How does the actual indexing of PDB files happen? According to my main source of information (becoming as dog-eared as a web page can get):

When you run SSINDEX.CMD, after it validates parameters it asks the version control system for the list of all source files (and their version numbers) in the target project and below. PDB files are then searched for recursively. For each PDB file found, the Perl code calls SRCTOOL.EXE to extract the list of source files. The code looks through the source files in the PDB file and sees if they match any of the files stored earlier from the version control search: if a match is found, the code saves that file info.

Does this mean that I should be calling SSINDEX.CMD and not CVSINDEX.CMD? Does this look through all folders in the named path? When I ran it yesterday I did several variations but to no apparent success.


and...I've forgotten the second thing. It will come to me.

My first patch, a true story

Well. I am jazzed up while I sit in BTB right now because I MADE A PATCH!

Yessir. The lab for today's class asked us to try and modify our browsers so that opening a new tab makes it appear to the right of the current tab, instead of the way it currently opens the new tab at the far right end of all tabs.

We all jumped on to MXR and ... it froze. So for a bit, I stared at the screen and wondered - "How hard is this going to be?".

Well, not too bad actually. I searched for "New Tab" and found /browser/base/content/tabbrowser.xml which has a comment:

// We're adding a new tab here.

Looks like I'm in the right place.

Turns out tabbrowser.xml is the best place to go because that's where my changes will be made and it also happens to be where all the methods regarding tabs are located.

Now I'm staring at some javascript. Immediate guilt sets in for not being more competent in this language. When did we take javascript? Back in third term. It's now buried beneath a tiny bit of perl, a pinch of php and attempts at ruby.

loadOneTab() is called when a new tab is added, loadOneTab() calls on addTab(). These are all in the same file. I'm focusing on addTab(). This is my target for adding some lines.

addTab() creates a tab object with the variable name t and then appendChild(t) to mTabContainer. Let's look at mTabContainer...

What do you know - it has a method called moveTabTo(). Cool, that's what I want to do. I want to moveTabTo() the index next to the one that's calling the addTab().

See tabPatch to get the rest of the story I've got to go do a victory lap.

Source Indexing - setting up for CVS

Included with WinDBG's debugging tools is an sdk that has some scripts to use for making a Source Server.

From what I have gathered so far, I needed to configure the srcsrv.ini file to point to my CVS repository - check, i pointed it to MYSERVER=:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot

Then I ran ssindex.cmd –server=cvs –source=c:\source –symbols=c:\outputdir with the appropriate locations.

First error:
ssindex.cmd [ERROR ] : A source control system must be specified using either the
"-SYSTEM=" option or by defining SRCSRV_SYSTEM in your environment.

Okay, so I realized that I should have been calling cvsindex.cmd NOT ssindex.cmd. Re-do the call...

Second error:
ssindex.cmd [ERROR ] CVS: CVS_LABEL or CVS_DATE not defined. Can't continue.

Gotta love that "can't continue", how dramatic. I remembered reading about the Label and Date setting when looking at the documentation for cvsindex.cmd -?? and thanks to Dave for noticing that it says "OR" not "AND" - so I set the CVS_DATE to today's date and go for third.

Third results:
--------------------------------------------------------------------------------
ssindex.cmd [STATUS] : Server ini file: C:\Program Files\Debugging Tools for Windows\sdk\srcsrv\srcsrv.ini
ssindex.cmd [STATUS] : Source root : c:\ff\mozilla
ssindex.cmd [STATUS] : Symbols root : c:\ff\mozilla\objdir\dist\crashreporter-symbols\2007092823
ssindex.cmd [STATUS] : Control system : CVS
ssindex.cmd [STATUS] : CVS Root : :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot
ssindex.cmd [STATUS] : CVS program name: cvs.exe
ssindex.cmd [STATUS] : CVS Label :
ssindex.cmd [STATUS] : CVS Date : 10/09/07
ssindex.cmd [STATUS] : Old path root :
ssindex.cmd [STATUS] : New path root :
--------------------------------------------------------------------------------
ssindex.cmd [STATUS] : Running... this will take some time...

And it did take time...but after all was said and done - nothing seems to be indexed. Minor success anyway, I learned how to run the indexing command.

Back to the drawing board.

Thanks to everyone on #seneca who reassured me that I wouldn't break the CVS repository by messing around with this.

When doubt creeps in...

Oh boy, what have I taken on?

Going deeper and deeper into any documentation I can find about how one goes about source indexing. I have no idea what I've signed up for.

In my C:\Program Files\Debugging Tools for Windows\sdk\srcsrv directory there is an excellent source server document that walks through the steps and for the moment all I know is that I'll be needing Perl.

So I'm downloading ActivePerl as I write this and hopefully once it's installed I can chip away at setting up some indexing scripts.

IIS is serving up PDB files locally

Well, thanks to Peter McIntyre (Resident Windows Genius) at Seneca College, I have now managed to set up my local symbol server.

Peter actually sent me a link to information about configuring IIS 6.0 but it turned out I had 5.1 - no problem he had given me enough so that a quick Google was all it took to find this script to which I merely added ".pdb", "application/octet-stream" and voila - my MIME map is updated. Now my localhost symbol server can serve up .pdb files without a hitch.

Started up Minefield, attached to VStudio and the symbols loaded up as smooth as can be.

Small success leads to much dancing around with the hound dog.

In other news, finished part 1 of the IBM "Master the Mainframe" contest (if anyone wants to do it, register using Peter as your Faculty person) and am slowly working through part 2...with breaks for reading up on bugs. I still find the whole bugzilla environment quite daunting, it's like that feeling that everyone else knows what they're doing except me. This is me, peering around the corner and listening in...

Setting up a Symbol Server Sandbox

After today's IRC chat with luser, I now have a list of things to do in order to acheive 0.1:

• set up a localhost server


• make buildsymbols from my own build


• load those symbols onto the local server


• connect them up to my debugger to make sure it all works


• get the microsoft scripts to work adding source code to my local pdb files


• so that the debugger can access the source code from being pointed to my server

Starting off, I set up IIS on my computer so that I can have a localhost webserver. This was a bit tricky with some unexpected authentication issues but I think that I have it working now. If I point my browser to http://localhost/symbolServer/ I have a directory of the pdb files that I created by calling make buildsymbols in my objdir.

The next move was to point my debugger (Visual Studio 2005) to the localhost symbol server. However, this is where I got stuck. I could load up the microsoft symbols but it would skip right over my firefox ones.

So I'm going back to square 1. I'm rebuilding with debug disabled because this may be a part of the issue. If this doesn't work I need to look into either a) symbol server directory structure because maybe I'm missing something about the hierarchy or b) perhaps my IIS set-up authentication issues are preventing VStudio from accessing the symbols.

Of course, in the back of my head I know it could also be c) something else entirely.

Back to the building.

Learning to debug - Part 1

So, I'm trying to hook up the Mozilla Symbol Server in order to better see how my attempts at knitting a source server to it will work.

First - I need to know what version of Firefox I should be doing this with. Currently I have 3 options: ff2, nightly build and my own build from yesterday's class. Talking with Dave on IRC, it seems like the nightly is the one to go with.

Now I need a sure-fire browser crashing bug. A few attempts at finding simple-seeming bugs in mxr is fruitless. I try a Gmail bug, the printer bug - no crash. Then bsmedberg provides me with this.

So, with my visual studio 2005 attached to the process that is Minefield, I install the toxic extension and restart Minefield - "Crash Bang Boom!" (i love this, btw). So now I can see some stuff in Visual Studio. What I'm looking at, I'm not exactly sure yet.

Then I did something and cleared out the results of the crash and now I can't get them back because Minefield crashes too fast for me to attach to process in VStudio. Bsmedberg gave me the following commands to try and get it back:
set env NO_EM_RESTART=1
devenv -debugexe firefox.exe -P test

I'm fiddling with this as we speak.

Stay tuned for part 2 where I hopefully will start to see the use of symbols.

Heading to 0.1

Today has been a blow-out day of looking at other people's code. First in class we explored mxr and tried to follow simple browser functions back to their origins. In looking up where the code for the "Clear Private Data" option was, I discovered that I can choose between being asked to confirm before clearing or not. I personally will choose not because I hate the additional click of confirming.

Following the one hour bike ride home and the dog walk to and from the library I sat down to read Chapter 11 of Diomidis Spinellis, Code Reading: The Open Source Perspective
. I liked the simplicity of his example. While I followed his trajectory through adapting a large source code base I picked up on some great ways to avoid trying to master the entire source and instead poke away at what you want to accomplish.

So I need to apply this to what a 0.1 release of my project will look like.

I looked briefly at the source code for the symbol server. There are several references to code on my project page.

Two things that are missing for me:
1. Seeing a debug in action so that I can get a visual reference for how the symbols work with the debugger.
2. Learning how to add to the existing code so that I can test whether my changes will work.

I'm going to continue looking at resources that cover how to debug in windows and see if I can make some headway. If anyone is interested in giving me a demo of how they debug in windows using the symbol server, I would appreciate that.

Running 2 versions of Firefox

In IRC chat the other day I asked how often Ted (aka luser) did builds and he said not that often - pointing me instead to the nightly builds saying that he used these instead since they were updated automatically every day.

So I downloaded one and installed it.

Now, my favourite extension is the foxmarks synchronizer app because I find myself travelling between my desktop Mac at home and my PC laptop, as well I will often work at jobs where there's a need for a third foxmarks setup.

This extension doesn't work for 3.0 yet - so I've currently been opening the command window and opening each version with the -Profilemanager so that I can use my default profile for 2.0 (with foxmarks) and my "nightlyBuilds" profile for exploring 3.0

Now I read in a post that I can just manually change the version. Sounds great, but I don't know if I want to go through this everytime there's a new nightly version number.

Anyone else know a better way?

Ted's visit

Learning from today's class with Ted Mielczarek:

- lots of successful builds, so things are getting easier
- the error that we are getting about directory names with spaces is an advancement over the past when it just failed for an unknown reason
- there's a bug right now with building on Vista, there's a work-around right now
- VM is a great way to test in different environments (I have VMware, just need to get it going)
- ac options are for autoconf whereas mk options are for make
- a reminder to use an object directory with builds
- why client.mk? you can build mozilla without it - but it's easier with client.mk - it handles the MOZ_CO_PROJECT stuff
- looking at tinderbox - not everyone is using tinderbox...lots of people use buildbot so it would be great to get more of the mozilla stuff into buildbot
- there's work being done to get rid of autoconf - switch to python script instead
- talking about replacing make but not sure to what
- google-breakpad, a crash reporting system, integrated with mozilla
- demo of crashing, reporting the crash - bug reports (this is all still a little overwhelming)

And then a break.

- make -C
- mozconfig options for debugging:
ac_add_options --enable-debug (in optimizations)
- demo of searching code at lxr.mozilla.org


Final note - in order to do the Source server I apparently don't have to be stellar at C++, phew!

Starting the Project - Mozilla Source and Symbol Server

Today we get to hear Ted Mielczarek speak on the Mozilla Build system. According to Dave, I will be communicating with Ted about the project I signed up for, the Mozilla Source and Symbol Server.

Let me say first of all that I chose this project because I know nothing about it. Let me emphasize this: NOTHING.

So, to get started - looking over the resources provided, reading up on what a source server is and what a symbol server is. I think that I need to see the symbol server in action so that I can know why a source server would be beneficial. This means I need to learn how people debug Mozilla.

Apparently you can debug both in VStudio and also in something called Windbg (which makes me think "windbag"). I have VStudio 2005 already installed and last year a fellow classmate showed me how to do some basic debugging of my C++ programs. At the time that was a huge breakthrough - to be able to see exactly how far into a function I could get, to see what a variable was set to was a huge help in figuring out where something was going terribly wrong.

I don't know if debugging Mozilla is like this. I haven't got it set up yet. My next goal is to find someone who can show me their debugging process. I learn best visually. So for now, I have just downloaded windbag...

Makefile experiment

Okay, as per this week's "To Do List" I followed the instructions and created a basic makefile for the nled editor. I haven't used nled since first year, what a flashback.

Anyway, it was pretty simple. Not sure if I "cheated" by commenting out a tiny statement in the qkdisp.c file in the qb_beep function as follows:
/*#elif WIN95
MessageBeep(MB_OK);*/
#elif __OS2__

because I was getting an error about it.

In any case, the makefile looks like this:

and the end result? No .obj files left over when I ran it with the command --- make && make clean ---
as well...ta da!!