Today I have a script running in staging, polling for completed try runs and doing dry runs of posting to bugzilla in log format only so I can keep an eye out for unusual output. Already this has shown me a couple of bugs to sort out, and I anticipate having them ironed out very soon. However, before this lands I would like to get some feedback/ideas/suggestions on what the output to the bug should look like and there could be a couple of options even, with a setting in your try syntax.
In my early testing I posted to our landfill bugzilla and here's what a couple of results looked like:
|Lots of success looked like too much info|
|So I took it out, and only printed the warnings and failures|
|Which works on small runs|
From what you see here, I'm sure you can imagine what it would look like if 145 builds all had warnings/failure combos.
So - what do you want to know in the bug? Let's keep it simple, ok? We can add more later and it's important not to create a bug-spammer here that folks will clamor to turn off soon after it goes live.
Off the top of my head, and after talking with Catlee today about it, I think it should look like this:
Try run for $revision with the following comment:This gets you a quick glance at total builds/total requests so you can see that everything is accounted for, and where things might have gone wrong in builds vs. tests but doesn't list the failed/warning builder names so you have to follow the link to get them. Maybe there would be interest in printing what your try syntax request triggered but I'm not sure that's useful in the bug reporting even though it's requested for when a developer is pushing to try. What do you think?
S:# W:# F:# (results total) builds complete from N total requests
S:# W:# F:# (results total) tests complete from N total requests
For more information please see http://tbpl.mozilla.org/?tree=MozillaTry&rev=$revision
It would be possible to break down the results further by platform instead of or as well as build/test. Any ideas on how to get that much information across without making a bug comment too verbose? All input appreciated and considered, I'll be trying to land this in the next couple of weeks so comment here, ping me in IRC (lsblakk), or comment in the bug.