Want to help? Encouraging community contributions to Release Engineering

In a timely confluence with Mozilla's new Steward initiative, I'm preparing to get some community contributors engaged with some of the projects we work on in Release Engineering.  A fair amount of our production infrastructure has to be locked behind VPN and sekrit passwords (we have 400+ million users to protect) but there are more and more RelEng side projects. We provide tools to the larger developer community and solve interesting scalability challenges with our unique (and massive) automation systems that can be worked on by any interested person in their own local test environment and then integrated into our /build repos. My personal goal is to try and get 2 or 3 regular community contributors to come work with us on tackling these.

In order to solicit contributions I have been working with David Boswell. We added Release Engineering to the mozilla.org/contribute 'areas of interest' page and I have created the beginnings of a RelEng-specific contribution page. The first two areas that I think would be a great introduction to working with RelEng code & tools are the TryChooser and our upcoming Autoland system.  For the latter, our intern Marc Jessome is sticking around this fall as a contributor to carry on the amazing work he put into this system over the summer.  He'll be continuing to debug the code and improve the portability of it so that we can get it into a beta testing stage by the end of October.  As that work is being done we also need someone to help us write the API functionality that will allow sheriffs and developers to write tools that utilize this new hands-off landing queue.  We'd also be happy to have people work on the issues that come up when we take Autoland to the next level - auto-landing on a production branch.  To do this we'll want some automated backing out, bisection, and the ability to wait on getting patches reviewed before continuing.

Another great area for someone interested in helping out Firefox developers is working on the TryChooser syntax and features.  There is a whole tracking bug dedicated to try_enhancements and most of those bugs are ones that can be worked on in a local staging environment.  It's a chance to get your feet wet with buildbot and our custom scheduling setup. Some of these smaller bugs would be short on time commitment and high on developer appreciation if you fix them. That can be a winning combination for a new contributor, I speak from experience on that :)

So, if you're reading this post and you or someone you know is interested in dipping their toes into becoming a Mozilla contributor and these projects make you curious then come find me and we'll get you set up with a staging environment so that you can start fixing real world tools and automation bugs in no time.

Mozilla Seeks Program Manager for Open Web Innovation Incubator

Ok, I'm a little biased - full disclosure: I work for Mozilla.  But even if I didn't I suspect I'd be impressed with the amount of amazing innovation and hustle that Mozilla's community puts out towards making the open web more accessible to everyone.  Recent projects like Popcorn and Butter are changing the way we work with video on the web. Hackasaurus is reaching out to kids, getting them to move beyond consuming the web (read-only mode) to being able to build and design their own web experience (read/write).  Addons SDK, Open Web Apps, Browser ID, the list goes on and on and I'd better stop now or I'll lose you before getting to the good stuff.

Mozilla may be a huge brand but we're actually a relatively small group of people doing this work all around the world.  Which is why we now have WebFWD, a program to help innovators of the open web get a chance to hook in to the resources of Mozilla (space, mentors, public reach, food and housing, and more) to help bring their products to the world.  With this rolling program of projects Mozilla can be a driving force in getting more open web to the people who need them.

Right now we need a visionary and hard working Program Manager to become the leader of this movement.  I'm attaching the posting below, get in touch with p at mozilla dot com with your resume for consideration.

/// WebFWD - Program Manager

Mozilla, the organization behind the Firefox Web browser, is looking for 
an all-star to join our new accelerator/incubator program WebFWD 
(http://webfwd.org) which aims to do for the open Web what organizations 
such as Y Combinator, TechStars and Seedcamp have done for startups.

As the Program Manager of WebFWD you are charged with leading the 
overall program - from designing and managing the curriculum, supporting 
the selected teams locally as well as globally, working with our 
ever-growing list of mentors and partners to organizing events.

If you're passionate about the Web, want to help people build amazing 
products and are willing to roll-up your sleeves, then this position is 
for you.

Primary responsibilities:
* Design and manage the curriculum for both the Fellow program as well 
as the Bootcamp
* Work with and support teams in the program (both locally and remote)
* Work with our mentors and partners
* Coordinate community and press outreach on a worldwide basis
* Create and run events (locally and remote)

Requirements:
* Excellent written and verbal communication skills
* Experience with organizing and running events
* Experience working with startups, entrepreneurs, venture capital and 
incubators / accelerators a huge plus
* Proven ability to work independently and in cross-functional teams
* Familiarity with Web technologies
* Passionate about helping people and solving problems
* Enjoys learning and teaching others
* Works effectively in a fast-paced, start-up environment
* 3-5 years of relevant job experience
* BA/BS, or equivalent experience

Try results to the bug(s) of your choice upon completion

The TrySyntax helper and TryChooser wiki docs have both been updated to reflect the new option when pushing to try where you can now ask to have your complete summary of results (and a link to the tbpl page for your revision) posted as a comment to the bug on completion.  Here's a live example to check out:

Sample comment in a bug when using --post-to-bugzilla in your syntax.

Now you have more control over how you get your try results and how noisy a try push is.

Please send comments and issues to the bug tracking this work.  Thanks for trying it out!

A quick morning rant about "gender" and data collection

This morning I read that Google+ is going to make your name and "gender" required to be public if you want to participate.  This bothers me for several reasons:

Web sites and forms notoriously say "gender" when they mean "sex" and only put M/F or Male/Female as options. When this type of choice is required but called "gender" it erases many people who do not feel that those options cover their gender since that is actually something way more mutable than your assigned sex at birth.  Solutions: Call it "sex" which is really what those two categories are or don't make something that is not in fact binary into a required choice of two options.

Google are so proud of being all scientific and data driven and I'm frustrated that they would not take the opportunity on their new potentially game-changing social platform to re-vamp data collection. Don't they have the processing power to allow people to put in whatever they like as "gender" and let the power of the search sort things out in the end?  If a small number of people want to put "jedi" or "dog" let those people find each other!  Who cares if there are some people who don't feel like Male/Female defines them?  Why Google? Why do you want to act like two boxes can cover the breadth of human experience as it relates to gender in this world?  Why can't you innovate on the small things as well as the big things that affect human interactions?

I'd really like to see a shift in how we collect data where there is more trust that the user knows who and what they are and that they want to share this information at their comfort level and that those on the other side, let's call them advertisers (cause isn't that what it all comes down to?), be the ones to deal with the outliers and uniqueness of human experience instead of trying to bash everyone into a two-party system.

Sidenote: When I have collected data recently for PyStar and allowed the gender field to be a text box I have found that the expected percentage (98%) of people entered "typical" information like woman, girl, female and that those who needed to express a different response appreciated the ability to do so by entering something else.  Leaving this field to user input choice did not result in a messy, chaotic list of random words or unidentifiable descriptors.  I fear not that most people will suddenly start to be something else when given more autonomy on forms.

Tree Closing Downtime Notice - 4am - 8am PDT Thursday June 16, 2011

Trees will be closed for downtime so that we can land the following:

1. https://bugzilla.mozilla.org/show_bug.cgi?id=662396 -- Fix time on dm-wwwbuild01

2. https://bugzilla.mozilla.org/show_bug.cgi?id=600980 -- Set journal_mode = WAL for dirty places profiles -- This mean new performance numbers will start on Thursday morning after the downtime

3. https://bugzilla.mozilla.org/show_bug.cgi?id=649123 -- Run ANALYZE on dirty places.sqlite files --  This mean new performance numbers will start on Thursday morning after the downtime

4. https://bugzilla.mozilla.org/show_bug.cgi?id=663568 -- reboot the DNS and DHCP servers in scl1 -- Rebooting these servers has been shown to burn builds in the past, requires a short (~5min) outage to reboot these servers to allow updates to take effect.

5. https://bugzilla.mozilla.org/show_bug.cgi?id=663963 -- change LDAP to see if that speeds up mercurial -- This change should be entirely transparent.  Hg processes that are running at the time that the change was made will have already loaded the NSS LDAP module and will continue to use it until they exit.  The only issue to be aware of is that changes to hg access (group membership, or the creation of a new account) will not automatically propagate to the hg servers the way they do now.  If any hg access changes need to be pushed urgently, we can do that manually.

If anyone has a reason not to proceed with this downtime, please let me know.

Thoughts on cultivating an "Everyone is Remote" attitude

As I write this I am working from Paris and our team timezone spread looks like this:

•  Rangoria, New Zealand: UTC (+12)
•  Bucharest, Romania: UTC (+3)
•  Istanbul, Turkey: UTC (+3)
•  Paris, France: UTC (+2) <--- ME!
•  Ottawa, ON: UTC (-4)
•  Toronto, ON: UTC (-4)
•  Philadelphia, PN: UTC (-4)
•  Clifton Park, NY: UTC (-4)
•  Chicago, IL: UTC (-5)
•  San Francisco, CA: UTC (-7)
•  Mountain View, CA: UTC (-7)

I'm going to go out on a limb here and say this: Release Engineering does a good job of working remotely with each other. We are 15-16 people (with a few more contractors/fte on the way) and it doesn't matter where you live for you to work with us. Here we are in our meeting yesterday:

Releng Weekly Meeting - June 2011

Quite the impressive Brady Bunch layout, right?

Here's what we do that I think works well for working remotely:

* We meet once per week as a whole group on Mondays. This starts the week off with a status update on our major projects and also a chance for individuals to speak up about anything they're working on that they'd like people to be aware of.

* We are always having conversations in IRC amongst ourselves and with others in several channels. We use #mozbuild as a backchannel for our inter-team discussion, #build for access to a larger group of fellow Mozillians (like philor, Kairo, and ted for example, who often need to liaise with us), #developers is also a place we frequent and then there are some IT/mobile/QA/release-specific channels we hang out in as needed. I think this helps us have a presence in many areas of engineering/dev/IT and even with some of the non-technical teams at Mozilla where inter-team communication needs to happen. It keeps us in the loop on what various teams are up to and also provides the IRC equivalent of being able to overhear water-cooler chat and participate as well.

* We keep wiki pages for most everything. From "how-to" pages for our own release process, automation details, and project planning all the way to pages for outside-releng folks like the Try Syntax. While I find wikis frustrating the minute the information is out of date, the fact that I can update them and find them in my awesomebar quickly when I need them is very valuable to me.

* We email our group with important notices and changes to how things are done. There are not often times when someone will say "Oh I didn't know about that" and the response is "It came up in the hallway when I was talking with so-and-so". More often than not, the person driving a particular upgrade or change to current practices will send out an email to the group with details of : a) what the change is b) what it means going forward c) how the message has been disseminated to a wider audience (if needed) and finally d) where the wiki pages (and bugs, if needed for reference) can be found. This allows any of us to find the information N time units later when the change actually comes up in your daily work and you're wondering "What was I supposed to do when trying to use the new X again?"

* We all meet up face to face approximately once per quarter. Twice a year for Releng work weeks and twice a year at Mozilla all-hands/summit gatherings. We take these as opportunities to discuss larger topics with lots of brainstorming, whiteboard scribbling, and animated opinion-sharing. Notes from meetings like this turn into wiki pages (often during the meeting itself) and those can become specs for projects/bugs to carry the work that needs doing to the next level.

I think that gives a good idea of our team practices. Now here are some thoughts I've been having about lately with regards to working remotely in Mozilla as a whole. It helps that I'm currently working in Paris right now and am pretty much completely opposite of the PDT work day but some of this was on my mind even when I was in SF.

I think Mozilla has an amazing opportunity to set trends in how to work with distributed teams. We already have people in every time zone! Even with the incredible advancements we've made with our use of video/audio/irc tools (airmozilla/vidyo), there are some ways in which MV is still the eye of Mordor for the company.

I would like to see us shake that up so I think we should try:

* Not having meetings in large groups in MV (except at all-hands). Instead, put small groups of people in various rooms around the building so that "we are all remote" is a reality for everyone so that the clarity of the communication channels are taken seriously. This means we all become just as invested in the quality of audio/video feeds, using tools like Etherpad for public collaboration, and advocating best practices for the speakers/presenters as those who are not in MV. I bet we'd see an increase in contributions to new tools & meeting practices if we were all experiencing meetings remotely on a regular basis.

* Rotate the hosting of the Monday meeting so that over a series of Mondays it would be run from various remote Mozilla offices and this would mean that it moves in time (which could be scheduled in advance) but it also means that all offices get a chance to feel special and be the center of attention. We'll have an opportunity to get to know our co-workers from other offices better as they present the meeting and I even imagine some friendly competition could develop for who can run the most energetic and engaging meeting.

I'm really interested in trying that second one. The most MV-centric thing we do is have our 11am PDT meeting on Mondays be a locked-down time. What if it rotated around each week and just happened somewhere in the 9-5pm spectrum of your timezone? We could create a schedule for it so folks could have lots of notice for scheduling their other Monday things around it. Also, maybe sometimes you might miss one Monday meeting because it's just not at a good time for you but that's something some of our remote workers might say is just par for the course.

I know the idea needs more work, but there's the nugget of it. Curious to know what others think. I'll be continuing to talk this up - maybe we can have a larger discussion at the all-hands in September. Eventually I'd like to see us get to a point where we all think of ourselves as remote since if you look at Mozilla as a whole there does not really need to be a "hub" where one would be "local" compared to everyone else - there's just planning for timezones/meetings and then all the people we work with doing their amazing stuff.

Use Try? Read this.

Two updates to Try are about to go into effect which enforce asking for what you want using the try syntax and configuring how much email you want to get with your results.  Read more below.

Bug 661409 - Now that this has landed, a push to try only generates email about a particular try builder's results if it does not succeed.  You can adjust this to be more verbose by adding a -e/--all-emails to your try syntax if you miss getting over all those emails, or you can just shut off the emails completely with a -n/--no-emails in your commit syntax. Note that you must be using the "try: " syntax for these email flags to be picked up which leads quite handily to...

Bug 649402 - Try syntax use is about to be mandatory as soon as this bug is fixed and the hg hook is enabled on the try repo. We're doing this to encourage developers who use try to take an extra moment and request only the resources they absolutely need on their push.  This should reduce the test/talos load that has been increasing wait times across all branches during busy periods.  One additional psychological change is that the "try: -a" syntax has been removed and in order to ask for a mozilla-central matching run you must be more explicit: "try: -b do -p all -u all -t all". I've updated the docs to reflect this change as well as the TryChooser syntax helper webpage. We're really not trying to make your life harder with this change, approximately 50-60% of pushes to try currently use the try syntax and if you push to try without it you will get a helpful message pointing you to docs and syntax builder.  Check with #developers for tips and tricks from the folks who've been using this since the beginning, I know they have many including using the newly-minted Mozilla-Inbound repo where a push will get the complete set of tests/talos if you'd like to let your patch bake for a bit after doing a selective try run.