A day that shall go down in infamy...

Today at 6pm I have finished the necessary changes to my Source Server patch so that it is ready to be checked in. This will be my first check in to the Mozilla code and I'm pretty psyched that after about 6 months of working on this, this particular chapter is about to close.

The last few changes today were quite interesting and good learning for me. Basically, I had to take my | if self.srcsrv | stuff and put it into a separate method. Ted pointed out that this was because it's win32 specific code, so there's no need for it to be in the general Dumper class.

Hello object-oriented programming? This is exactly what I need to learn this in an applied manner and I will remember this lesson better than having just read it in a book.

So now there's a declared SourceServerIndexing method in the Dumper, and then a detailed method of the same name in the Win32 Dumper - which is where all the action is. I wonder if there's a chapter somewhere down the line where I will try to take this SourceIndexing into the Linux and Darwin dumpers.

When I first pulled everything out into the method, I had to wrap my head around the way that Python calls methods. Basically, if I called it from within Dumper, I need to do self.MethodName so that it looks for the method inside the Dumper class. Without the self. in front, I got an error message saying that a global name "MethodName" was not recognized. Also, when I was running | make buildsymbols | I wasn't source indexing the pdbs. So I put in some print statements to see if all the right steps were happening. They were. Confusing.

Eventually I noticed that there's a |shutil.copyfile(file, full_path)| which is what puts the pdb files in the place that I want them to be for indexing and I was calling my method before that. Quick switch and the lights turned on, the music played and my pdb files were indexed.

This has been a great project for teaching me how to set up a work flow as efficiently as possible. Now I'm at the point where I can get right into working on the code and testing frequently with minimal effort. The closer I got towards getting it all working, the easier the work flow got. Right now I can test this patch with just MingW32, a command shell and my Crimson Editor. I can tell within the first few seconds of |make buildsymbols| running if it has worked and then a ctrl + C to interrupt it and get back to the tweaking of code.

It feels great to have put this up for approval. Can't wait to see what happens next.

Temporary Geek Home

Went to the Toronto Mozilla office today. Spent the afternoon reviewing add-ons and I also got to put some faces to names. Since last Friday when we got training from Alex Polvi to be editors for AMO, I've reviewed 14 extensions and I'm learning a lot about the process.

Ted said he'd be reviewing my patch tomorrow and we're hoping to land it in the next few days. As far as I can tell right now, the only issue should be that my error handling is currently just printed to stderr and doesn't actually change the srcsrv flag.

So fingers crossed - there might be source server on the debug builds by the end of the week. This means the rest of term can be devoted to pdbstr.exe. I have yet to communicate with Timeless who is the recommended contact to discuss things of this ilk.

Over reading week I should be able to do some more digging into those hex dumps as well as start testing the source server once it's on Tinderbox.

Keeping busy

Well as reading week approaches and I wait (again) for feedback on my patch, I am trying to keep busy in Mozilla activities.

One of my recent activities is helping out with Live Chat user support. It's amazing! A very small team of dedicated people are helping individuals in real time with their wide variety of issues. I've been learning tons about the issues that users face - right now firewalls and security updates seem to be a frequent problem. I'm also jumping headlong into another thread of the vast Mozilla community, this one mostly volunteers, and getting to know some folks. The EST hours for Live Chat are from 4pm - 9pm and from 10pm - 12am. I mostly have been able to pitch in on the later shift because that's about when I am tired of doing my homework :)

Getting involved with Live Chat was super easy - I read the documentation, created and account and was shadowing experienced helpers less than an hour later. One week later and I'm in on a phone conference to discuss what the priorities are for the next 3 months. Again, I say amazing.

I encourage all Mozilla or open source involved Seneca students to jump into Live Chat and try helping out people in real time. It's intense, and can be challenging (I think my success rate is only about 50%) but it's an eye-opening experience for how the other half lives. The half what don't have #seneca for all their questions.

Better late than never

Although my goal was to have the symbolstore.py patch approved for committal by week's end, I am very close. Today I spent about 5 hours tweaking the patch as per the most recent comments.

What was done:

* Makefile now checks for the environment variable of PDBSTR_PATH to be set before assigning the flag to source index
* Several minor logic changes to the symbolstore.py like removing redundant file path fixing
* I finally understood what Ted was saying about using a temp filename before checking for VCSinfo. This small change made it possible to remove 3 lines of work-arounds that I had been doing :)
* The hard-coding of the path to the pdbstr.exe executable is gone and has been replaced by whatever is in the environment variable

A few glitches:

* Making changes to the Makefile that resulted in errors left me with a makefile that couldn't update itself and Ted pointed me to doing a make -f client.mk configure which regenerated the Makefiles and cleared that up
* I had to look into how to get the environment variable in python - thanks to web search I found the os.environ.get() function

Other than that, it's all working and I'm very happy to be awaiting another review.

Smart Install Instructions

Today as I was setting up a mysql GUI I saw this:


It made me very happy to see that part about Ejecting and ridding yourself of the installer because a lot of folks still might not know that. I've definitely met some people who drag firefox to their dock from the installer instead of copying it to their Applications folder. Every time they open the application from the dock, the installer has to mount and someday down the line they might delete it and then not know why the application no longer launches from the dock. OS X just gives the ever so helpful "?" and the user is left to wonder what happened. That far down the line - it's not so apparent that they should have dragged to the Application folder and ejected the installer.

Most Mac installers these days do make it quite clear that you should drag to your install folder and I would love to see them add this tip too - just to help folks finish what they start.

Upcoming Source Server Demos

This is mostly a reminder for me, but if anyone is interested in seeing how the Source Server is working/coming along, my demos are scheduled for:

• Thursday March 13th at 1:30pm in ORI


• Thursday April 3rd at 1:30pm in ORI

The first demo should involve the symbolstore.py patch having been committed, there should be a debug build that can be downloaded and attached to VStudio and/or WinDBG where the source code is then pulled via the Source Server.

The second demo will hopefully be a look at a reverse engineered pdbstr.exe - something that can read and write to pdb files the source indexing data block.

Now on Windows

Using StraceNT I was able to get this output.

Here's a snippet:

IntellectualHeaven (R) System Call Tracer for NT, 2K, XP, 2K3.
Copyright (C) Pankaj Garg. All rights reserved.

Tracing command: ["pdbstr" -r -p:accessiblemarshal.pdb -i:am3.stream -s:srcsrv]
[T3600] TlsGetValue(1, 0, 2bfef8, 182020, ...) = 2c7778
[T3600] EnterCriticalSection(77c61b30, 2c7778, 2bfed0, 77c3a03b, ...) = 0
[T3600] LeaveCriticalSection(77c61b30, 2bfed0, 77c3a0fa, d, ...) = 0
[T3600] EnterCriticalSection(77c61b18, 2c7778, 2bfed0, 77c3a06c, ...) = 0
[T3600] LeaveCriticalSection(77c61b18, 2bfed0, 77c3a108, c, ...) = 0
[T3600] HeapFree(2c0000, 0, 2c7778, 0, ...) = 1
[T3600] TlsSetValue(1, 0, 0, 2bfef8, ...) = 1
[T2556] LeaveCriticalSection(2c1fdc, 6f1c0, 77c2d154, 4, ...) = 0
[T2556] LeaveCriticalSection(2c7718, 6f1d8, 77c3b967, 13, ...) = 0
[T2556] HeapFree(2c0000, 0, 2caa48, 1058d24, ...) = 1

The results are certainly a bit clearer looking than the Linux/Wine results. I am still clueless however to the deeper meaning. I've been told I need to talk to timeless on IRC, that he is the one with major knowledge on reverse engineering.

This is turning into quite the rabbit hole.

Exploring what pdbstr.exe actually does

As I wait to find out what else I can do to get the symbolstore.py patch commit-worthy, I thought I would start to look into reading and writing to pdbs as described here.

I downloaded Paws.exe which is a hex editor program and also installed Borland's C++ compiler so I could try to dump the contents of a pdb with Borland's tdump utility.

This netted me a whole lot of hex code and so I stared at that for a while, consulting Jeremy Gordon's information, and didn't really figure much out. I can see the data block that pdbstr write to the file, it's just not clear after comparing the dump information of a couple of pdbs how or where the data block's write position is determined. This is a hex dump.

For kicks my friend John Ford and I looked at pdbstr.exe through strace and Wine on his Ubuntu box. The results are here.

There is a StraceNT for Windows so I will try to run pdbstr with that and hopefully end up with results that are a bit clearer since they won't include the Wine calls.

Anyone with any ability to read into either of these dumps - feel free to comment and share your insights.

The Gap.

This morning I tried to explain to my Deaf-Blind student the usefulness of Bookmarks. What he and my friend's mom who I was helping with a website the other day have in common is that they both save web pages to their desktops. For the mom, this results in an mhtml file on her desktop that, once opened, doesn't help her know what website she is visiting...and she mistakenly thinks she is on the web when she looks at it because the "file:///" URI signifier means nothing to her.

When my student saves the page, I believe he is selecting a different option which creates a folder on his desktop containing as much of the page as Firefox will grab. Again, when he goes back into this folder later many things happen:

1. He doesn't know why the folder is on his desktop


2. The contents of the folder are all files that are unfamiliar (.css, .js and .html)


3. There is no signifier of what the original web page was

So today I thought - okay, let's learn about bookmarks.

I showed him how to drag and drop to the Bookmarks Toolbar and how to Bookmark using the pulldown menu. I'd like to think he "gets it" but I know that it will take many more lessons for this to get across. What did get a little recognition of concept was that he would have access to the most recent content if he used bookmarks instead of saving a local copy. That was appealing.

All this comes to me now as I am reading these studies done on bookmarking habits, and reading into the logic behind Places and how the changes to the Home button are going to be initiated in Firefox 3 and I feel concerned.

I love the idea of bookmarking with tags and never having to scroll down a list again. I love the awesome bar's quick access to recent pages and I love that with minimal typing in the location bar I can see my history and get to previously visited pages quickly. Here's the thing - I'm able to take in the whole screen at once, I see little details quickly and I know what I'm looking for.

The mom, the student, and I'm willing to bet a lot of unstudied people out there are not doing this and are way behind on the idea of tags let alone how to use them.

I would love to do studies of web usage and get a really huge pool of participants because from what I've read so far, the largest group was 320 people at a tech conference and in my opinion that's a lazy study that will only confirm what the researchers and pushers of web 2.0 want to hear.

Moving forward is amazing and fun for me but I'm loathe to leave all the people I know and love behind to wander around lost and confused. My mom said the other day that she now understands less than 50% of what I'm talking about when I describe the projects I'm working on. 50%?! That sucks! My mom is actually a very astute person who even took some computer programming back in the 80's and she is very competent with power tools and techno-gadgetry. I want to be at least 80% compatible with my mom when talking about my projects. It would be great if it was possible for her to participate in the discussion instead of just listening politely.

Today I'm lamenting that only a select few are steering the discussion about the future of bookmarking and the student and the mom are left on the other side of a widening gap. Their bookmarks and habits are just as, if not more, important.

Eric Raymond peeved me this morning...

I was on my morning commute to school, reading my ethics textbook, happily processing information about intellectual property and copyright law (my favourite things!) when I read this:

"Anybody who has studied software engineering knows that programmers do not actually spend most of their time originating software. They spend most of their time on service updates and maintenance. Nobody thinks about the implications of this: that the software industry is actually a service industry operating under the delusion that it is a manufacturing industry. Software producers are operating under a manufacturing and cost model, under which the way you make money is building a product and getting it out the door. Because they have this model of themselves as a manufacturing industry, all the bright people go to production and the dumb people go to the support desk. That's why when you call a vendor support line you have to fight your way through three layers of idiots to get down to anyone who knows anything. " - Eric Raymond in an interview.

Well, this got me a little angry.

Now, I won't say that I have never had to do any "idiot filtering" when calling a large company's support services. I blame this on the McJob model of customer service where companies hire people at low wages and then "train" them by giving them scripts to diagnose the simplest problems. Perhaps the company thinks it is also "idiot filtering" for the person who hasn't plugged in their modem and we are all losing with this approach.

This is not exactly what I feel Eric Raymond is saying though. What I read into his statement is that there is a natural filtering of "bright" people and "dumb" people - like natural selection or something and that is just plain wrong.

I'm in the BSD program at Seneca College largely because I would like to be able to better support the hundreds of people I know who need technical support on a semi-regular basis and at the same time be able to help the hundreds of thousands of people I don't know who also need support. I'm interested in technology and have always gravitated towards it while at the same time noticing that others do not. This puts me in a place where I could be a great "bridge" between techies and non-techies.

Why can't programmers and other technically-stimulated people understand that they are a minority? Most people need help to operated systems, use gadgets, understand software and troubleshoot their life-enhancing tech gear. To be someone who supports them does not mean that you are too "dumb" to be on the creative end (which he already acknowledges is a small portion of the industry).

I'm sick of people treating documentation like a hassle, like cops to paperwork, as though it doesn't matter...this leads to poor documentation done for documentation's sake and not for the end user. In my opinion, you have to be "bright" to create tech support documents that help a user continue to patronize your software and not just quit. From what I've learned as a customer service representative in many forms, people will share their bad experiences to approximately 10 times more people than they will share their good ones.

I take this to mean two things for how to go about life:

1. Share your good experiences often to try and make up for how often people share bad ones
2. Be someone's good experience so they have something positive to share

It's a challenge to me and to other programmers to push ourselves to keep things clear and easy to grasp for the lowest common denominator as much as possible. This doesn't mean the software is dumb or low level, nor does it mean that the user is. It means we will agree to speak a certain language together so the most people can benefit. People will clamour for software that does what they need with the least amount of work.

The article is good, I recommend reading it because Eric Raymond obviously knows his stuff and has some great insights to share. I'm inspired by his vision of alternative business models driven by open source. The quote just sparked something that was obviously bugging me already.

Back to reading up on ethics...

Getting closer to committing?

So I feel like I am close to having closed up most of the original issues with the patch. After the conversation with Ted I was feeling good about how to wrap the source indexing so that it's only applied to a Win32 platform.

The newest patch is here and basically the outstanding issue is just the path to the pdbstr binary. What I'm stuck on is that right now I'm passing a path to pdbstr in my build directory but when this patch is committed it will be somewhere else - which may allow for the setting of the path using global variables in the Makefile that I can't use on my local debug build.

Some sleuthing should clear that up soon. Perhaps 0.6 will be a commit? I'm setting my sights high for that, and then I could spend some time writing a pdbstr replacement in the last few releases.

What's coming for 0.6

Recently I had a conversation with Ted on IRC about how to take steps to make the patch work properly for Win32 platforms.

Here's the gist of what I'll be working on for 0.6:

ted: http://lxr.mozilla.org/mozilla/source/Makefile.in#174
ted: that's where symbolstore.py actually gets run
ted: you see how it uses $(MAKE_SYM_STORE_ARGS) ?
ted: up above that line we define those per-platform
ted: like windows gets -c to copy .pdb files
ted: in the main() function in symbolstore.py, we process the cmdline args
ted: so ideally we'd have another flag there
ted: --generate-source-server, or something

That's my task for now. Off to play with the Makefile.

Moz Dev Demo - 0.5 Release of Source Server for Windows Builds

Click to Play

This is a video demo of the status of my current bugfixing on the bug I filed regarding the adding of Source Server to windows debugging of mozilla code.