Wednesday, December 14, 2011

Want more women in Open Source? Donate to Ada Initiative today!

Short version: If you love women, or even like them just a bit, go right now and donate to the Ada Initiative to show the women in your life that you value their contributions past, present, and future to the wonderful world of Open Source. I'm going to make a donation in my grandmother's name this year and I know she'll be happy to have supported such a valuable project.

Long Version

I love Open Source.

When it first came to my attention, in the first year of my degree in software development at Seneca College, I knew we'd be a good fit.  There's something about the spirit of Open Source that instantly clicked with my existing guerilla activist sensibilities.  The way that you just take what you want and make it happen.  That you create and give away. That you work with other passionate people to make cracks in the surfaces of monopolies that only want you to be able to do things through their (usually financially) gated communities.  It reminded me of how I had approached being a filmmaker - taking $50 of Super 8 film and developing it myself in 16L bucket in a dark bathroom then submitting the results to a prestigious film festival and being accepted.  Having my work shown alongside films with budgets bigger than the cost of a house was an amazing experience and taught me that not everything has to be polished to be valued.

Open Source is like that to me, the diamond in the rough.

While I was working on my degree I of course noticed (and was not surprised by) the lack of women in my classes.  I was surprised when I started to get involved in Open Source to discover that there were less women in FOSS than in proprietary software companies. That seriously BLEW MY MIND.  I mean, if Unlocking the Clubhouse is to believed (and it is very thorough research) then technical women want to do work that is meaningful and helps people.  Why that sounds a lot like Open Source doesn't it?  So why aren't there more women in Open Source?  I'll let you Google that question to your hearts content, there's a lot written on the subject and so much more could be.  The point though is that the Ada Initiative is a new project that is here to take on that very question through ACTION.  They will DO things to get more women in Open Source.  Women don't have to be dragged into FOSS kicking and screaming.  Trust me, after seeing the overflowing wait list for the Grace Hopper Celebration of Women in Computing's FOSS day, there are a ton of talented and smart women interested and able to do work in Open Source.  We (all of us who have already drank the Kool-Aid) need to help them get integrated and feel comfortable staying in FOSS. 

When I first met my future team at Mozilla in April of 2008 there was a woman on my team (!) and she self-identified herself to me as a feminist within the first 5 minutes we were together.  As someone who was coming in as a student with zero experience in professional tech workplaces I was so thrilled to have an immediate feeling of relief, trusting that if she was respected there I would be too.  She also introduced me to wonderful internet properties such as GeekFeminism and Sociological Images both of which helped me start connecting with other feminists in tech fields.  Almost three years later I am starting to feel like I've been successful in building the community in FOSS around me that I want to be a part of.  It's a wonderful mix of the talented people I work with at Mozilla, the folks I'm working on planning the next Dare 2B Digital with, the programmers I organize PyStar workshops with, the Women Who Code, the Women 2.0, and of course - The Ada Initiative.

I'll leave you with their own words about why you should go straight to the donation page:
We’re proud of what we’ve accomplished already.  Since our founding in
early 2011, we helped over 30 conferences and organizations adopt an
anti-harassment policy, organized the first AdaCamp unconference,
provided free consulting on high-profile sexist incidents, wrote and
taught two workshops on supporting women in open tech/culture, and ran
two surveys, among other things.

http://adainitiative.org/what-we-do/

We need your help to achieve our upcoming goals.  The Ada Initiative
is funded entirely by donations.  Without your financial support, the
Ada Initiative will have to shut down in early 2012.

http://supportada.org/donate

Your donations will fund upcoming projects like: Ada’s Advice, a
comprehensive guide to resources for helping women in open
tech/culture, Ada’s Careers, a career development community, and First
Patch Week, where we help women create and submit their first open
source patch.  You can learn more about how the Ada Initiative is
organized and operated on our web site and blog:

http://adainitiative.org

Whether or not you can donate yourself, you can help us by spreading
the word about our fundraising drive.  Please tell your friends about
our important work.  Email, blog, add our donation button to your web
site, and tweet.  Here’s how:

http://adainitiative.org/support-us/spread-the-word/

You don’t have to stand on the sidelines any longer.  You can help
women in open technology and culture, starting today.


Sunday, November 20, 2011

My First Startup Weekend: Women 2.0 Startup Weekend

On November 18th, 2011 I jumped into the deep end of the Bay Area startup culture I have been lurking on the periphery of for the past two years of living here. After going to my first Geek Girl Dinner at Microsoft a month ago, and preparing to talk about women in open source at the Grace Hopper Celebration of Women in Computing, it seemed very much up my alley to sign up for the Women 2.0 Startup Weekend held in SF at The Hatchery.  Originally Angie had asked me to be one of the mentors which, while incredibly flattering, seemed beyond my current skill set.  I do always have ideas for new projects/apps though and I've been trying to get even more full on development experience under my belt so it seemed like a deal to get to spend 54 hours working on a startup idea for $50.

[tangent]
I love the immersion-as-classroom experience, btw.  I made my first Super 8 film in 1999 at G.I.F.T.S under similar conditions where I lived with my other new-to-filmmaking cohort in a couple of trailers-turned-bunkhouses over on the beautiful island of Galiano and for one week we did nothing but eat, sleep, and breathe guerrilla filmmaking.  We shot, hand-developed, transferred, and then edited our work, cranking out an entire short film in just one week.  I left that experience filled with confidence that I could make a movie a week for the rest of my life!*
What I was hoping for out of my first Startup Weekend was an up-to-your-armpits code-a-thon and what I got was...very much NOT that. Here's what I got out of Women 2.0 Startup Weekend instead:

Pitching 101
Some people had come prepared.  They had read an email I missed, knew what was supposed to be in a pitch, perhaps even had some code or a site name or some idea of what they would need to take the next step into their imagined company.  I had none of these things.  I had 2 ideas, one of which had occurred to me the week before on a bike ride.  I jotted down my ideas quickly and 'pitched' them to a couple of women I knew from other local events (like my CodeChix pal, Vicki).  Both of my ideas seemed to get people interested and with the help of a few very kind listeners, I chose which one I would officially pitch and worked on naming the imaginary app as well as figuring out what salient points I wanted to get across.  It seemed wise to me to get into the early round of pitchers, little did I know that there would be about 67 pitches.  I was #6 and so it took a long time to get to the point of being able to move about the room chatting people up, which I am sometimes not so good at

What I did: I pitched it, was too shy to really reach out to strangers and try to woo them to my idea, I hid my sign for a while only taking it out when people asked me about it, I got 7 sticky-note 'votes' for it (which was amazing to me), but I already knew that I would not be working on this project over the weekend and I was shopping around for a team that I would be excited to spend my next 54 hours with.
What I will do next time: Work more on my pitch ahead of time, have a clear idea of WHO I would like to join me, go around the room and find people who match those roles, have more research about my 'market' ready to help with the business side of things.
What I wish Startup Weekend organizers could improve: Help people match up by roles - so have all the designers go to one corner, all the marketing folks, all the developers, etc.  Give us a visual of who's there looking to do what so that we can more easily go around and network.  It seems less efficient to me that I would have to go chat up 10 people and perhaps discover that none of them are a match for what I'd be seeking. Even putting this info on people nametags would help - especially for folks who have multiple skills they want to highlight.



A Team is Formed

The eliminations were happening and I already knew I was going to put my idea aside for another time, so I had to figure out where I would lend my energy for the 54 hours to come.  I'd been interested in a project called Safe Steps whose goal was to help independent women set a timer on their travel to ensure safe arrival at their expected destination. I spoke briefly with the woman who pitched it, and I had already learned from a conversation with a volunteer that the pitcher was a seasoned pro at marketing.  I felt like I would learn a lot in that team but I was still checking around for other ideas.  In the chaos of the eliminations I ended up behind a pillar with 4 people (one is a coworker at Mozilla) and two of them I had met briefly when they accosted me, they were looking for people who could program in C (and though I did it in school 4 years ago, I was not about to claim any proficiency). I asked if they had found what they were looking for and inquired about what they were planning.  Judy explained her pitch about doing an educational project with the Kinect to teach language to children. I have experience teaching technology to both youth and adults, so working on anything that helps make educational materials accessible to all types of learners, as well as the possibility of doing hands-on Kinect-hacking for the first time, was all it took to sweep me up into this team that was bouncing off the walls and repeating those magic words: "Kinect" and "education".

Team Roles
We had 54 hours to come up with a demo of our 'company' for a panel of judges to evaluate based on marketability, creativity, and feasibility so when we got our workspace assigned to us at 9:30pm that Friday night we went straight to work. Introductions all around, describing our experience and what drew us to the project, came first and then we divided up into the technical team: James and I, and the Business/Strategy/Research team: Judy, Elsa, and Jen.

Our technical idea seemed simple at first - Grab the Kinect motion data and send it to Processing.js so users could interact with a language learning flashcard game that was one of many 'decks' our 'platform' would support. The initial deck would be a simple game with a bear where the bear calls out a verb, enacts it, then waits for the user to imitate both the motion and the word.  I really did go into this thinking that was simple.  Am I crazy?  Turns out none of that was within our reach in a 54 hour period. The challenges are too many to list completely but here's a few: both James and I were completely new to Kinect-hacking. While open source Kinect hacks exist there were lots of library conflicts, documentation gaps, and finicky installations that led to failure on several frameworks we were trying and build on. I could get the Depth.js example to work in Chrome for a second (but never again for unknown reasons), but couldn't compile the native google plugin from the depth.js project so couldn't write new code for the extension. I couldn't build the OpenNI Sample-NiUserTracker after altering it to add a network tunnel so that it would report data to a node.js server (though I'm happy to have now touched Node.js even just a bit!). By Saturday late evening we had nothing to show except an intimate knowledge of library linking errors and compile failure messages.  There still isn't a ton of material online about how to work with the Kinect data in a usable way.  This actually gets me excited for future projects but at this point in Startup Weekend, we had to get ourselves a demo for Sunday's judging.

We decided to move on to the Kinect SDK that Microsoft provides, we installed Visual Studio 2008 Express and an open source gesture recognition library which allowed us to capture a movement and assign it to a saved gesture namespace.   In the end, our demo was created in a few hours by James using those tools (and a bit of C#) while I came up with some very quick animations objects and put together our landing page.

Needless to say, the weekend was nowhere near being a code-a-thon.  It was surprising to me how long it took to try and get a development environment setup and what I take away from this experience is that when the time comes to work on my own idea, if I bring it to a Startup Weekend, I should have the beginnings of an implementation already and have settled on a framework to build on that I am familiar with so that I can spend more time being creative about the idea and less time fiddling with configurations and installations of unfamiliar code.

Oh ya, but we won!
I should mention that the whole time we were having our ups and downs with the technical side Judy, Elsa, and Jen worked hard at analyzing all the angles of language learning by doing.  I listened in at one point on a very helpful discussion with Cindy Alvarez who asked great questions about "what next?". Sure, verbs and kids are easy and lots of language-learning stops there - how would we push the envelope and take people to other levels?  We had lots of mentors come by, and all of them poked and prodded at the research and story-shaping that the business end of the team was doing. At the end of the weekend our team won first place with a demo that had very little custom code in it, but I think we did well because we told a great story and had an extremely well thought out marketing strategy. When our demo was complete the judges were silent at first.  Finally one of them asked the question we had prepared for "so, after learning verb with bears- what next?" to which I answered that we could build a platform for AI interactions in WebGL 3D space with the Kinect.  Yes, I like to promise technology that doesn't really exist yet.  It sure is exciting to imagine it though.

Some final thoughts
Startup Weekend, to me, felt a lot more like a school project than 'real life'.  This is most likely due to the fact that I have a really great full time job right now and am looking at startup ideas mostly as learning and hobby and not necessarily something I would do for money for a few more years (at least). All the reading I have done about startups gets me thinking that I would likely go the way of bootstrapping and working on my scalable project in my spare time instead of trying to get a big VC investment and leave a steady job for the unknown. In terms of working during the weekend there are lots of ways to fall down rabbit holes and lose focus when you are working on something that is completely new. I love getting to learn about new technologies but there was this time pressure that kept us coming back to a general goal of having something to show at the Sunday evening presentations. The Startup Weekend environment isn't one for coding/development efficiency. It's distracting to have other people and their ideas/surveys/questions coming around a lot and to be working out in the open with your entire team instead of under noise-cancelling headphones as I normally do.  It's not bad, it's just not a focused environment and it's good to know that for next time.  I think it would help me set my expectations differently.  It was important for me to learn that the goal of Startup Weekend is not necessarily to have a working application at the end but to have a really well thought out idea and story about your company's goal.
Speaking of story, come on out to TEDx Women next week where Elsa from Words With Bears will be presenting ours! Most importantly,  I want to say that Words With Bears was a great team to work with. I heard stories of teams falling apart or losing team members, none of that touched us at all. We started strong and we ended strong. We're continuing to stay in touch and aim to develop this idea into something bigger. 

* This is not what ended up happening, but I will always carry with me the knowledge that with little else than enthusiasm, a couple of rolls of film, and willing friends, a tremendous amount of creative output is possible in a short time with no budget.

Wednesday, October 26, 2011

New to IRC? Never tried it?

Do you shy away from IRC because it seems daunting, complicated, or completely foreign to you? A pretty large chunk of the Mozilla online community lives and breathes in IRC so I encourage you to come give it a(nother) try. I recently updated https://wiki.mozilla.org/IRC#Getting_Started to help get you started. Hit me up with any questions here or find me as lsblakk in IRC :)

I'll re-visit the instructions soon to write about screen & irssi to keep a perpetual connection going in an attempt to make that option more accessible to folks who might feel it's too technical.

Where Are My $project-branch Nightly Builds?

Did you know that we don't build a fresh nightly on a branch unless there's fresh code?  Well, now you do!  In the interest of saving even _more_ resources and network bandwidth so that we can accommodate even  _more_ project branches we have added this little bit of logic to our nightly build scheduler.  It makes sense, right?  I mean, why have a nightly that is just like last night's nightly?

In other news, Ben just added 4 more twigs (aka Disposable Project Branches) for side project work and one of them is still available for booking.

Monday, September 26, 2011

Want to help? Encouraging community contributions to Release Engineering

In a timely confluence with Mozilla's new Steward initiative, I'm preparing to get some community contributors engaged with some of the projects we work on in Release Engineering.  A fair amount of our production infrastructure has to be locked behind VPN and sekrit passwords (we have 400+ million users to protect) but there are more and more RelEng side projects. We provide tools to the larger developer community and solve interesting scalability challenges with our unique (and massive) automation systems that can be worked on by any interested person in their own local test environment and then integrated into our /build repos. My personal goal is to try and get 2 or 3 regular community contributors to come work with us on tackling these.

In order to solicit contributions I have been working with David Boswell. We added Release Engineering to the mozilla.org/contribute 'areas of interest' page and I have created the beginnings of a RelEng-specific contribution page. The first two areas that I think would be a great introduction to working with RelEng code & tools are the TryChooser and our upcoming Autoland system.  For the latter, our intern Marc Jessome is sticking around this fall as a contributor to carry on the amazing work he put into this system over the summer.  He'll be continuing to debug the code and improve the portability of it so that we can get it into a beta testing stage by the end of October.  As that work is being done we also need someone to help us write the API functionality that will allow sheriffs and developers to write tools that utilize this new hands-off landing queue.  We'd also be happy to have people work on the issues that come up when we take Autoland to the next level - auto-landing on a production branch.  To do this we'll want some automated backing out, bisection, and the ability to wait on getting patches reviewed before continuing.

Another great area for someone interested in helping out Firefox developers is working on the TryChooser syntax and features.  There is a whole tracking bug dedicated to try_enhancements and most of those bugs are ones that can be worked on in a local staging environment.  It's a chance to get your feet wet with buildbot and our custom scheduling setup. Some of these smaller bugs would be short on time commitment and high on developer appreciation if you fix them. That can be a winning combination for a new contributor, I speak from experience on that :)

So, if you're reading this post and you or someone you know is interested in dipping their toes into becoming a Mozilla contributor and these projects make you curious then come find me and we'll get you set up with a staging environment so that you can start fixing real world tools and automation bugs in no time.




Monday, July 25, 2011

Mozilla Seeks Program Manager for Open Web Innovation Incubator

Ok, I'm a little biased - full disclosure: I work for Mozilla.  But even if I didn't I suspect I'd be impressed with the amount of amazing innovation and hustle that Mozilla's community puts out towards making the open web more accessible to everyone.  Recent projects like Popcorn and Butter are changing the way we work with video on the web. Hackasaurus is reaching out to kids, getting them to move beyond consuming the web (read-only mode) to being able to build and design their own web experience (read/write).  Addons SDK, Open Web Apps, Browser ID, the list goes on and on and I'd better stop now or I'll lose you before getting to the good stuff.

Mozilla may be a huge brand but we're actually a relatively small group of people doing this work all around the world.  Which is why we now have WebFWD, a program to help innovators of the open web get a chance to hook in to the resources of Mozilla (space, mentors, public reach, food and housing, and more) to help bring their products to the world.  With this rolling program of projects Mozilla can be a driving force in getting more open web to the people who need them.

Right now we need a visionary and hard working Program Manager to become the leader of this movement.  I'm attaching the posting below, get in touch with p at mozilla dot com with your resume for consideration.

/// WebFWD - Program Manager

Mozilla, the organization behind the Firefox Web browser, is looking for 
an all-star to join our new accelerator/incubator program WebFWD 
(http://webfwd.org) which aims to do for the open Web what organizations 
such as Y Combinator, TechStars and Seedcamp have done for startups.

As the Program Manager of WebFWD you are charged with leading the 
overall program - from designing and managing the curriculum, supporting 
the selected teams locally as well as globally, working with our 
ever-growing list of mentors and partners to organizing events.

If you're passionate about the Web, want to help people build amazing 
products and are willing to roll-up your sleeves, then this position is 
for you.

Primary responsibilities:
* Design and manage the curriculum for both the Fellow program as well 
as the Bootcamp
* Work with and support teams in the program (both locally and remote)
* Work with our mentors and partners
* Coordinate community and press outreach on a worldwide basis
* Create and run events (locally and remote)

Requirements:
* Excellent written and verbal communication skills
* Experience with organizing and running events
* Experience working with startups, entrepreneurs, venture capital and 
incubators / accelerators a huge plus
* Proven ability to work independently and in cross-functional teams
* Familiarity with Web technologies
* Passionate about helping people and solving problems
* Enjoys learning and teaching others
* Works effectively in a fast-paced, start-up environment
* 3-5 years of relevant job experience
* BA/BS, or equivalent experience

Monday, July 18, 2011

Try results to the bug(s) of your choice upon completion

The TrySyntax helper and TryChooser wiki docs have both been updated to reflect the new option when pushing to try where you can now ask to have your complete summary of results (and a link to the tbpl page for your revision) posted as a comment to the bug on completion.  Here's a live example to check out:
Sample comment in a bug when using --post-to-bugzilla in your syntax.

Now you have more control over how you get your try results and how noisy a try push is.

Please send comments and issues to the bug tracking this work.  Thanks for trying it out!

Wednesday, July 6, 2011

A quick morning rant about "gender" and data collection

This morning I read that Google+ is going to make your name and "gender" required to be public if you want to participate.  This bothers me for several reasons:

Web sites and forms notoriously say "gender" when they mean "sex" and only put M/F or Male/Female as options. When this type of choice is required but called "gender" it erases many people who do not feel that those options cover their gender since that is actually something way more mutable than your assigned sex at birth.  Solutions: Call it "sex" which is really what those two categories are or don't make something that is not in fact binary into a required choice of two options.

Google are so proud of being all scientific and data driven and I'm frustrated that they would not take the opportunity on their new potentially game-changing social platform to re-vamp data collection. Don't they have the processing power to allow people to put in whatever they like as "gender" and let the power of the search sort things out in the end?  If a small number of people want to put "jedi" or "dog" let those people find each other!  Who cares if there are some people who don't feel like Male/Female defines them?  Why Google? Why do you want to act like two boxes can cover the breadth of human experience as it relates to gender in this world?  Why can't you innovate on the small things as well as the big things that affect human interactions?

I'd really like to see a shift in how we collect data where there is more trust that the user knows who and what they are and that they want to share this information at their comfort level and that those on the other side, let's call them advertisers (cause isn't that what it all comes down to?), be the ones to deal with the outliers and uniqueness of human experience instead of trying to bash everyone into a two-party system.

Sidenote: When I have collected data recently for PyStar and allowed the gender field to be a text box I have found that the expected percentage (98%) of people entered "typical" information like woman, girl, female and that those who needed to express a different response appreciated the ability to do so by entering something else.  Leaving this field to user input choice did not result in a messy, chaotic list of random words or unidentifiable descriptors.  I fear not that most people will suddenly start to be something else when given more autonomy on forms.

Tuesday, June 14, 2011

Tree Closing Downtime Notice - 4am - 8am PDT Thursday June 16, 2011

Trees will be closed for downtime so that we can land the following:

1. https://bugzilla.mozilla.org/show_bug.cgi?id=662396 -- Fix time on dm-wwwbuild01

2. https://bugzilla.mozilla.org/show_bug.cgi?id=600980 -- Set journal_mode = WAL for dirty places profiles -- This mean new performance numbers will start on Thursday morning after the downtime

3. https://bugzilla.mozilla.org/show_bug.cgi?id=649123 -- Run ANALYZE on dirty places.sqlite files --  This mean new performance numbers will start on Thursday morning after the downtime

4. https://bugzilla.mozilla.org/show_bug.cgi?id=663568 -- reboot the DNS and DHCP servers in scl1 -- Rebooting these servers has been shown to burn builds in the past, requires a short (~5min) outage to reboot these servers to allow updates to take effect.

5. https://bugzilla.mozilla.org/show_bug.cgi?id=663963 -- change LDAP to see if that speeds up mercurial -- This change should be entirely transparent.  Hg processes that are running at the time that the change was made will have already loaded the NSS LDAP module and will continue to use it until they exit.  The only issue to be aware of is that changes to hg access (group membership, or the creation of a new account) will not automatically propagate to the hg servers the way they do now.  If any hg access changes need to be pushed urgently, we can do that manually.

If anyone has a reason not to proceed with this downtime, please let me know.

Thoughts on cultivating an "Everyone is Remote" attitude

As I write this I am working from Paris and our team timezone spread looks like this:
  •  Rangoria, New Zealand: UTC (+12)
  •  Bucharest, Romania: UTC (+3)
  •  Istanbul, Turkey: UTC (+3)
  •  Paris, France: UTC (+2) <--- ME!
  •  Ottawa, ON: UTC (-4)
  •  Toronto, ON: UTC (-4)
  •  Philadelphia, PN: UTC (-4)
  •  Clifton Park, NY: UTC (-4)
  •  Chicago, IL: UTC (-5)
  •  San Francisco, CA: UTC (-7)
  •  Mountain View, CA: UTC (-7)

I'm going to go out on a limb here and say this: Release Engineering does a good job of working remotely with each other. We are 15-16 people (with a few more contractors/fte on the way) and it doesn't matter where you live for you to work with us. Here we are in our meeting yesterday:


Releng Weekly Meeting - June 2011

Quite the impressive Brady Bunch layout, right?

Here's what we do that I think works well for working remotely:

* We meet once per week as a whole group on Mondays. This starts the week off with a status update on our major projects and also a chance for individuals to speak up about anything they're working on that they'd like people to be aware of.

* We are always having conversations in IRC amongst ourselves and with others in several channels. We use #mozbuild as a backchannel for our inter-team discussion, #build for access to a larger group of fellow Mozillians (like philor, Kairo, and ted for example, who often need to liaise with us), #developers is also a place we frequent and then there are some IT/mobile/QA/release-specific channels we hang out in as needed. I think this helps us have a presence in many areas of engineering/dev/IT and even with some of the non-technical teams at Mozilla where inter-team communication needs to happen. It keeps us in the loop on what various teams are up to and also provides the IRC equivalent of being able to overhear water-cooler chat and participate as well.

* We keep wiki pages for most everything. From "how-to" pages for our own release process, automation details, and project planning all the way to pages for outside-releng folks like the Try Syntax. While I find wikis frustrating the minute the information is out of date, the fact that I can update them and find them in my awesomebar quickly when I need them is very valuable to me.

* We email our group with important notices and changes to how things are done. There are not often times when someone will say "Oh I didn't know about that" and the response is "It came up in the hallway when I was talking with so-and-so". More often than not, the person driving a particular upgrade or change to current practices will send out an email to the group with details of : a) what the change is b) what it means going forward c) how the message has been disseminated to a wider audience (if needed) and finally d) where the wiki pages (and bugs, if needed for reference) can be found. This allows any of us to find the information N time units later when the change actually comes up in your daily work and you're wondering "What was I supposed to do when trying to use the new X again?"

* We all meet up face to face approximately once per quarter. Twice a year for Releng work weeks and twice a year at Mozilla all-hands/summit gatherings. We take these as opportunities to discuss larger topics with lots of brainstorming, whiteboard scribbling, and animated opinion-sharing. Notes from meetings like this turn into wiki pages (often during the meeting itself) and those can become specs for projects/bugs to carry the work that needs doing to the next level.

I think that gives a good idea of our team practices. Now here are some thoughts I've been having about lately with regards to working remotely in Mozilla as a whole. It helps that I'm currently working in Paris right now and am pretty much completely opposite of the PDT work day but some of this was on my mind even when I was in SF.

I think Mozilla has an amazing opportunity to set trends in how to work with distributed teams. We already have people in every time zone! Even with the incredible advancements we've made with our use of video/audio/irc tools (airmozilla/vidyo), there are some ways in which MV is still the eye of Mordor for the company.

I would like to see us shake that up so I think we should try:

* Not having meetings in large groups in MV (except at all-hands). Instead, put small groups of people in various rooms around the building so that "we are all remote" is a reality for everyone so that the clarity of the communication channels are taken seriously. This means we all become just as invested in the quality of audio/video feeds, using tools like Etherpad for public collaboration, and advocating best practices for the speakers/presenters as those who are not in MV. I bet we'd see an increase in contributions to new tools & meeting practices if we were all experiencing meetings remotely on a regular basis.

* Rotate the hosting of the Monday meeting so that over a series of Mondays it would be run from various remote Mozilla offices and this would mean that it moves in time (which could be scheduled in advance) but it also means that all offices get a chance to feel special and be the center of attention. We'll have an opportunity to get to know our co-workers from other offices better as they present the meeting and I even imagine some friendly competition could develop for who can run the most energetic and engaging meeting.

I'm really interested in trying that second one. The most MV-centric thing we do is have our 11am PDT meeting on Mondays be a locked-down time. What if it rotated around each week and just happened somewhere in the 9-5pm spectrum of your timezone? We could create a schedule for it so folks could have lots of notice for scheduling their other Monday things around it. Also, maybe sometimes you might miss one Monday meeting because it's just not at a good time for you but that's something some of our remote workers might say is just par for the course.

I know the idea needs more work, but there's the nugget of it. Curious to know what others think. I'll be continuing to talk this up - maybe we can have a larger discussion at the all-hands in September. Eventually I'd like to see us get to a point where we all think of ourselves as remote since if you look at Mozilla as a whole there does not really need to be a "hub" where one would be "local" compared to everyone else - there's just planning for timezones/meetings and then all the people we work with doing their amazing stuff.

Friday, June 10, 2011

Use Try? Read this.

Two updates to Try are about to go into effect which enforce asking for what you want using the try syntax and configuring how much email you want to get with your results.  Read more below.

Bug 661409 - Now that this has landed, a push to try only generates email about a particular try builder's results if it does not succeed.  You can adjust this to be more verbose by adding a -e/--all-emails to your try syntax if you miss getting over all those emails, or you can just shut off the emails completely with a -n/--no-emails in your commit syntax. Note that you must be using the "try: " syntax for these email flags to be picked up which leads quite handily to...

Bug 649402 - Try syntax use is about to be mandatory as soon as this bug is fixed and the hg hook is enabled on the try repo. We're doing this to encourage developers who use try to take an extra moment and request only the resources they absolutely need on their push.  This should reduce the test/talos load that has been increasing wait times across all branches during busy periods.  One additional psychological change is that the "try: -a" syntax has been removed and in order to ask for a mozilla-central matching run you must be more explicit: "try: -b do -p all -u all -t all". I've updated the docs to reflect this change as well as the TryChooser syntax helper webpage. We're really not trying to make your life harder with this change, approximately 50-60% of pushes to try currently use the try syntax and if you push to try without it you will get a helpful message pointing you to docs and syntax builder.  Check with #developers for tips and tricks from the folks who've been using this since the beginning, I know they have many including using the newly-minted Mozilla-Inbound repo where a push will get the complete set of tests/talos if you'd like to let your patch bake for a bit after doing a selective try run.

Monday, May 23, 2011

Update on the Auto/Assisted Landing System

Almost a week since the post introducing the design attempt for auto/assisted branch landings via Bugzilla and Try and guess what? We re-wrote everything!

The details are in the wiki, bugs have been filed, code is being written.  We are working on making this system use a message queue and also see if we can work with mozillapulse to get information on bug changes from Bugzilla.

I'd love to tell you more about it but you can read the wiki and I'm excited to get back to my SchedulerDBPoller component.

Tuesday, May 17, 2011

Assisted/Automated Landing - Designing the Systems

Ehsan's blog post wishing for assisted landings on mozilla-central started a lot of people talking about this being a very desirable and useful tool for developers, where they could set a flag in Bugzilla and then be free to do other work until the results of their push were posted back to the bug. As part of enhancing the Tryserver I was already working on a way for users to signify in their try-syntax that the results of the push should go to the bug and these two ideas started to fuse into a dreamworld where someone could attach a patch to Bugzilla and have it be tried and pushed to trunk all with some magical bot automation.

After doing a very short survey of developers and their try usage I have observed that there are two very different stakeholders here and both of them need separate-but-related tools:


DevelopersThe Bot (automation)
  • ease of use
  • better reporting (less email anyone?)
  • option to post to bug(s) *after* a try run has indicated success
  • queuing of patches culled from a flag in Bugzilla
  • automatically apply to tip of repo
  • push, and report back with results

After soaking in the survey feedback and a first attempt with a whiteboard yesterday, I woke up this morning with some clearer ideas on how to take a first run at creating this system.  It involves creating several new tools, one new database, and enhancing our existing buildapi.

New tools for Developers:
  • Adding more Try syntax options:
    • include list of the bug(s) that you would like your try results posted to (however many make for a complete run on your push, this can be one linux build or a complete ~186 builder try: -a buildset) 
    • turn off email notifications
  • Adding functionality to the self-serve api view for a revision (eg: https://build.mozilla.org/buildapi/self-serve/try/rev/de8ea75bc48e) that will better show your results for that push and provide a button which will post the patch(es) to a specified bug
  • Auto-landing from a bug in Bugzilla using the [autoland-try] whiteboard tag where any attached patches which are not obsolete, and have nothing set for 'r' are applied to the current tip of mozilla-central, pushed to try and those results are returned to a comment in the bug
New tools to Automate landings (bot or script):
  • Crawl Bugzilla for bugs where [autoland-$branchname] is in the whiteboard and automatically push to tip of named branch, get the results, and return them to a comment in the bug (stripping out the whiteboard tag on completion)
    • bot will grab all non-obsolete, r+ patches (if $branchname != 'try') 
    • interdependent bugs will not be handled in this first swipe at a working system
    • pushes will have autoland-$bugnumber as the reason for the build in schedulerdb so that the results can be watched for, aggregated, and reposted to the bug on completion
  • Watch results coming back for one or two oranges (we can set a threshold) and re-triggers those, watching for the second set of results - to attempt catching intermittent oranges
  • Backout patches where even with a rebuild on an orange, there still remain orange results
  • LDAP authentication checking for bugzilla patch author -> hg commit permissions and being able to ensure that only people with the right credentials can trigger automatic landings. This may mean checking the reviewer too before allowing a patch to be applied & pushed.
The next step is to get this design organized into bugs so that we can parcel out the work involved and start testing/completing segments and features as we work towards the whole. We have a RelEng intern this summer, Marc Jessome (Another Canadian in RelEng!), who will be doing a lot of the work between now and the end of August. Stop by anytime to say "Hi" to Marc and to chat with either of us about the project - feedback is always appreciated.  I'm happy to say that 52 people filled out the Try Usage survey just from posting it on Yammer. That was super helpful, thank you.

Wednesday, May 4, 2011

A PyStar Supernova in the Sky

The first Bay Area PyStar event has come and gone. I'm finally getting a moment to regroup and ponder all the trial and error of being the organizer of this event as well as having time to look at some of the statistics we gathered. Just from an organizing perspective here are a few items I'd like to share about the process.

Things to do differently next time:
* When creating the Eventbrite event, add questions like "What level do you want to learn at?" "Meat or Vegetarian?" "Operating System?" to the registration so there's no need to send out blanket emails to attendees to try and get that information after the sign up.
* Only do one day workshop instead of Friday night installation and Saturday workshop. I think that for many people the setup could have been done in the first hour and the rest of the day been spent learning instead of having a night session that only is needed by a handful of people.
* Have the teachers/assistants already assigned to a particular level of instruction - prepare topics, tutorial materials, and class size ahead of time so that on the day of the workshop there might only be a handful of late arrivals to place and the other attendees will already be set up in the right learning level as requested in the sign up. 

Things that really worked this time:
* Eventbrite! They have amazing tools, stats, emailing options, charts, and also a way to see where your sign ups come from which showed us that we got a TON of views from Tweets which apparently was an impressive number (I am told by one of our attendees who is an Eventbrite employee)
* Mozilla! By sponsoring the event - providing the space and food - being able to let people/groups spread out and work in our various conference rooms as well as having lunch on site was very much appreciated by attendees (and of course by me!)
* CodeChix!  This peninsula-based group of women coders accounted for 30% of our attendance and also netted some teacher/assistants for the workshop. CodeChix co-sponsored the event and helped get word out as well

Attendance
There was something odd happening with the Eventbrite signups. In a couple of short bursts, a ton of tickets were being snatched up by names that seemed slightly suspicious. Now the event has passed and I've checked in all the attendees as well as accounted for the no-shows (almost all of whom took a moment to send in their regrets so the tickets could be freed up for another person - very sweet!). It looks to me like about 40% of our attendees were fake accounts. Julie (who works at Eventbrite) and I took a look at the numbers and she's kindly offered to look into it further to see if there is indeed something fishy happening. All that aside, we had 47 people! That feels like great attendance to a first workshop, on a Saturday, in Mountain View.
Speaking of Mountain View - we had attendees come from all over Northern California. I love this view of how spread out geographically we all were:

This graph is useful for seeing how my own promotion attempts were successful.  The original spike of page views is obviously when I first announce the event link. CodeChix, Baypiggies, and Devchix were the mailing lists I sent emails to with the link. While that got the ball rolling, it was the tweets and emails sent out almost 3 weeks later - a week before the event where the event got lots of attention.  It probably helped that PyStar Minneapolis was happening then too so #PyStar got lots of tweets (sorry to the person who's twitter nick is @pystar).
Can I just say that I am so thrilled with the amount of people who volunteered to teach/assist?  Seriously. Amazing. I love that there are people out there who really enjoy getting newbies involved, who can share their skills, and who will give their time to events that grow community.
Finally, here's a breakdown of where we got ticket "sales" from via Eventbrite. This is another reason they rock - they help you promote your event!  As you can see here the Twitter share link definitely got us the most eyes even though direct invitation resulted in more actual signups. For next time I would send the link to a few more mailing lists like SF Python Meetup, Systers, and also next time we'll be able to invite the folks who came to the first one as well as those who couldn't make it.

In follow-up posts I will post and analyze some of the survey results of both the PyStar Bay Area and the PyStar Minneapolis. I need to go learn how to create charts from Google doc spreadsheets. Also we need to figure out how to set up our site and materials to be easily updated and adjusted by a distributed team without having to break off into separate sites.  Finally, the curriculum needs an overhaul. We kept an etherpad during the event to track issues so that I can go through post-workshop and take advantage of all the feedback to improve our offerings.

What's Next?
The next PyStar I plan to organize will be in late July or early August and I'd like to do that one in SF.  Following that I'm going to plan one in Toronto for mid-to-late October.  What we did this past Saturday is only the beginning. I'll be working with all the folks in the pystar group to get this program shaped up into a much more modular system for learning Python and Django in stages (badges) and also will be setting up sub-groups for things like hack nights, code-masters (think toastmasters but writing code in front of people), and I have this idea of taking the PyStar lessons into women's prisons as a way to get marketable skills into the hands of people who need them badly.

Anyway, first we'll get more material prepared and digest/incorporate all the excellent feedback. Then we can take over the world :)

I hope I'll see you at future events. Thanks to everyone who helped make this a great day!

Friday, April 22, 2011

Captain Destructo Breaks Everything

Alternate title ideas: "It's not all s/Tryserver/Try"  or "What I should have done, and didn't"

I bet you get the point by now. Today I caused a fairly lengthy, unnecessary downtime on Try.  Now that I'm writing this, things are under control again and there's a few small niggly bits left but nothing that will keep me up at night.

It all started with a bug about graphserver posts from tests not getting through because they were looking for MozillaTry (the tinderbox name for Try) but instead the graphserver only knew about Tryserver (the branch name for Try) and nothing was using Try (except the repo for Try) which is what it ought to have been doing in the first place.

Now that I've been adding a lot of project branches in a short amount of time, certain things have become more streamlined and so I felt that the best option was to go through and rename Tryserver/MozillaTry to Try everywhere so that from the repo going forward, everything was the same. This has been working extremely well for our project branches and helps make setup a snap.

Here's where it gets all broken. I approached this bug with a quick swipe at this problem was superficial and ended up causing some preventable burning.  I shall now list for you (and future me) what I did and what I should have done:

Did:
  • hg rename on configs for desktop
  • branch configs for s/tryserver/try
  • updated graphserver branch name to Try
  • a quick downtime window from 10am - 12pm in order to prevent builds from getting split into two different upload dirs
Should have done:
  • hg rename on configs for mobile
  • grep of buildbotcustom for "tryserver" as we have special casing for it in several files
  • log uploader and post_upload scripts to make sure everything about the try build was going to the right place
  • updated the dir permissions on ftp for the new upload location and ensure that the archive is on nfs mount
  • edited cronjobs on staging to catch the new try builds
  • updated graphserver machines table for each try platform's builder name
  • more notice for downtime, with a 4 hour window that would have allowed a test push to make sure everything was wired up correctly  
  • updated the treeclosure hook to include the new tinderbox page
Some of the things I should have done didn't have an impact on the burning/try closure but it's fair to say that if I had done a staging round of all my plans first I would have caught more of the obvious things that I missed. I would have then planned the downtime better and been prepared to ensure the disturbance would have been minimal since this was, after all, a really low priority bug.

Aki told me that he had a manager who said "you don't learn til you break something".  Well I broke everything try-related today and here's hoping that I have learned something because the stress of this whole day is not something I want to experience often. It's that feeling you get when you realize you've started something that you can't back out of and there's no way to go but forward, even though everything in front of you now appears hopeless and messy.


So here's some lessons to take away:

  1. Staging is not to be underestimated even for just renaming things that are already working
  2. Taking the time to search with grep/mxr and find the terms you are replacing before starting the upgrade in production will help find wiring you might have overlooked in your preparations
  3. Prepare more thoroughly and have a clear idea of the env. you started in and what it will take to have that env. back when you're done. Leaving dangly bits is not ideal.


Happy Friday.
(and many thanks to Aki)

Wednesday, March 23, 2011

Hey BBC would you like to know how releasing software works?

Dear BBC,

Today on the front page of your technology section you said that downloads for Firefox 4 have been lower than they were for Firefox 3 and that:
The lower figure may be explained by the widespread availability of pre-release versions of Firefox 4 in the months ahead of its launch.

First of all, you forgot that we've had 3.5 and 3.6 between those two and so we now have users spread out a bit across versions. Second, here's an overview of how we're organizing the release of Firefox 4:

  • We put out the RC and picked up users from outside of our usual beta testing pool in order to give our final candidate some solid tire kicking
  • Firefox 4 went live but our users on 3.5 and 3.6 are not offered the update automatically yet, they must "Check for updates" in order to be asked if they want to upgrade to Firefox 4 
  • Once we have more coverage of the new release for a couple of weeks and are even more confident that we've got an amazing browser out there we will turn on the Major Update notification which will offer our 400+ million users the chance to come on up and experience the next level of the web
According to W3C school's stats(which are measured by visits to their site) the browser distribution of their visitors looks like this:


2011 Internet Explorer Firefox Chrome Safari Opera
February 26.5 % 42.4% 24.1% 4.1% 2.5%
January 26.6 % 42.8% 23.8% 4.0% 2.5%


2011 Total FF 4.0 FF 3.6 FF 3.5 FF 3.0 Other
February 42.4 % 1.9 % 35.8 % 2.9 % 1.5 % 0.3 %
January 42.8 % 1.5 % 36.1 % 3.1 % 1.7 % 0.4 %

What this says to me is that our more than 8 million downloads since yesterday morning PDT only shows us how many people are paying attention to the fact that Firefox 4 has launched and is available for download. It's not representative of our 400+ million active daily user base (the people who just use the browser but perhaps don't read your blog or mine).  These people will soon learn about Firefox 4 through their browser's update notification window. We'll be seeing a spike in downloads in a couple of weeks and I hope you'll report on that.

Wednesday, February 23, 2011

Bay Area Video Coalition - Teaching Open Video Part 1

Last night was the first meeting of The Factory and Mozilla. The partnership if a result of work between Mozilla Drumbeat and Web Made Movies. Ben Moskowitz and Brett Gaylor invited myself and Atul Varma to what is to be the first of three sessions helping teens learn about the budding open web technologies that can be integrated with video.

About 13 kids streamed into a computer lab at 4:30pm and we began with some introductions detailing who we were and why this stuff is interesting. The group were very engaged and eager to dive right in with whatever we had for them. So we started with Atul's Web X-Ray Goggles so the students could see what exactly the web was made of. The idea was to just grab parts of whatever websites you liked and change them right on the page so you could see how easy it is to "hack" the web. Some of the teens went even further and started building their own pages with Etherpad by grabbing snippets of code from sites. About 4 kids said they had done a View Source on a page prior to this class and 30 minutes into this workshop they were all doing it like pros.

Once they had a chance to remix a web page we moved on to the next exercise which was to select 4 popular sites of their choosing (Etherpad democracy!) and those sites were printed out on paper, the teens were split into 3 teams, and each team did paper prototyping of a new site using elements from the originals. I was very impressed with how the students took the idea and ran with it. Each team worked fast (they had 7 minutes) and no one was hanging back keeping their opinions to themselves. The teams produced 3 new site mock-ups that each had a very simple look, with a video as the prominent element on the page but they also took time to add site navigation and social media integration by putting Facebook and Twitter in the sidebar or footer.

In the last 30 minutes of our time Brett and Ben demonstrated Butter also explaining how very "caveman" the technology is right now. With only a glance at the interface and a basic explanation of how it's wired up the teens jumped right in with suggestions and ideas about what they would like to be able to do:
* Hide popcorn elements when nothing is showing in them
* Be able to zoom in or out on Google maps while the video is playing
* Click on something in the video (example: coffee mug) and have it trigger an event

Ben made a really great point about how it's also important to look at something like Butter and think "How can you go beyond the interface?". How do you make your story more interesting from the beginning knowing you can use this tool throughout instead of just tacking on events and additional information to a completed video that is done in a standard format?

We'll be working with them again tonight, with chunks of a film they made last summer called "The List". More updates and more potential bugs and feature requests coming soon.

Also, if anyone is planning to teach a class like this you might want a few things in a "kit":
* Portable printer (and paper)
* Scissors
* Tape or glue
* Handout with links to the tools/sites

Just to save some time :)

Monday, February 7, 2011

Volunteers needed for upcoming HTML5/Open Video tutorial

I'm hoping if you're reading this that you might be interested in volunteering this coming Saturday to help 12-16 year old girls at the upcoming Dare 2B Digital conference learn about HTML5 and open video.  There's more information and background on what's happening on this wiki page.

Two kinds of volunteers needed:

1.  Someone who is in the Bay Area and available this coming Saturday from 9-3:30pm to be on-site with us in Mountain View at the Computer History Museum and will work hands-on with the girls to demo Miro Converter, Universal Subtitles, and a little bit of Popcorn.js.

2. Anyone, anywhere, who can do translations to any language and who is available on Saturday anywhere in the 10:15am-3pm PST window to do some 'live' subtitling and show the workshop participants how amazing the universal subtitles project is.

Please get in touch if you are interested/available. Or sign up on the wiki.

Thanks in advance!  I will be posting any demos, workshop materials, and an update post-event on how it went.

Thursday, February 3, 2011

Automated Try Results Posted to Bugzilla - A request for input on what the comment should contain

Lately I've been working on a a script which can check your try syntax for a bug number and a setting asking for --post-to-bugzilla.  If you've provided both, your try server results can be posted directly to the bug.  This is just part of a larger project to have patches submitted on a bug get automatically tried out, results posted, and at some point down the road they could even be pushed to trunk after a successful try run so "look Ma! No hands".

Today I have a script running in staging, polling for completed try runs and doing dry runs of posting to bugzilla in log format only so I can keep an eye out for unusual output.  Already this has shown me a couple of bugs to sort out, and I anticipate having them ironed out very soon.  However, before this lands I would like to get some feedback/ideas/suggestions on what the output to the bug should look like and there could be a couple of options even, with a setting in your try syntax.

In my early testing I posted to our landfill bugzilla and here's what a couple of results looked like:
Lots of success looked like too much info

So I took it out, and only printed the warnings and failures

Which works on small runs

From what you see here, I'm sure you can imagine what it would look like if 145 builds all had warnings/failure combos.

So - what do you want to know in the bug? Let's keep it simple, ok? We can add more later and it's important not to create a bug-spammer here that folks will clamor to turn off soon after it goes live.

Off the top of my head, and after talking with Catlee today about it, I think it should look like this:
Try run for $revision with the following comment:
$try_syntax line
S:# W:# F:# (results total) builds complete from N total requests
S:# W:# F:# (results total) tests complete from N total requests
For more information please see http://tbpl.mozilla.org/?tree=MozillaTry&rev=$revision
This gets you a quick glance at total builds/total requests so you can see that everything is accounted for, and where things might have gone wrong in builds vs. tests but doesn't list the failed/warning builder names so you have to follow the link to get them.  Maybe there would be interest in printing what your try syntax request triggered but I'm not sure that's useful in the bug reporting even though it's requested for when a developer is pushing to try. What do you think?

It would be possible to break down the results further by platform instead of or as well as build/test.  Any ideas on how to get that much information across without making a bug comment too verbose? All input appreciated and considered, I'll be trying to land this in the next couple of weeks so comment here, ping me in IRC (lsblakk), or comment in the bug.

Thursday, January 13, 2011

Try Server Road Map - Q1 2010




The other day this post by Google with slides detailing their Chrome release cycle speed up was going around and it mentioned how try and CI were key to their success. It got me thinking that it's time for another update about the upcoming improvements our try server automation. Most of my Q1 work will be on the try server, with some time on Fennec Beta releases, and a bit of time also working on making it much easier to spin up disposable project branches.

The road map image above shows how there are three areas of focus and here they are now in a more detailed list with bug number attached:

Improving Current Automation
  • Bug 617321 tracks adding two new try buildbot master instances to our other buildbot-masters in the Santa Clara colo.  This gives us flexibility to have rolling downtimes on try (as we already have on the other masters) where we can update things behind the scenes and it also helps by adding redundancy to the try automation in case of a colo outage.
  • Bug 580346 is almost done and it adds xserves to try which gives us some faster macosx building power, that along with about 40 ix builders for win/linux builds will crank out more try builds faster.
  • Bug 594236 is key to getting TryChooser syntax turned on as a default.  With an interactive hg prompt on push to try, you should be able to select what you want and/or have your syntax validated.  Khuey started something to do this and if anyone is up for taking it to the next level before I can get around to it, please please do.
Add Features
  • Bug 430942 is what I am actively working on today and the rest of this week, I'm about to have a second draft ready for review and I expect you can watch for this to land in the next few weeks. With or without try syntax, you will be able to specify the bug number in your push comment and have the results of your try run posted as a comment on that bug.  Hopefully this will help out development in letting people know where something is at if you are away when the results come in. It's also part of our old bug (pre-2010) smack down goal so finishing it will be one more step toward that carried-over goal being met.
  • Bug 615705 is tracking a few more tweaks to the try syntax that will give users more flexibility in the syntax and choice about what to build.
  • Bug 421895 is another old bug and Chris Atlee is close to getting it up for poking at. It provides a way for users to cancel their own try build requests without having to ping RelEng.
  • Bug 621681 addresses having better threading/headers since the current headers only help with threading for some clients. When I first wrote it I was testing with Thunderbird, where it works as intended but apparently Gmail and other clients need some help.
Future

Looking to the future right now we have bug 625464 which talks about setting up something to scan bugzilla for a flag on attachments that will trigger an automatic try run with that attachment and either tip of trunk or perhaps a user repo, which would require the other main future bug, 625463. With the ability to poll and run try on hg.m.o user repos we can have project branches (temporary branches that are loaned out to devs or teams to work on a particular project) toggle a setting that would have their pushes to the repo get run through try instead of the main mozilla-central automation.  This could be handy when you want to limit the machines you are building/testing on with the TryChooser syntax.

I hope anyone reading this will find the upcoming try work as exciting as I do.  Reading the Google slides, I couldn't help but sit up straighter at the mention of try being one of the reasons they were able to speed up their release cycle. I'm hoping we can get there too and that our try server will be more robust and ready to handle our soon-to-be-speedier release process too.