Today I have a script running in staging, polling for completed try runs and doing dry runs of posting to bugzilla in log format only so I can keep an eye out for unusual output. Already this has shown me a couple of bugs to sort out, and I anticipate having them ironed out very soon. However, before this lands I would like to get some feedback/ideas/suggestions on what the output to the bug should look like and there could be a couple of options even, with a setting in your try syntax.
In my early testing I posted to our landfill bugzilla and here's what a couple of results looked like:
 |
| Lots of success looked like too much info |
 |
| So I took it out, and only printed the warnings and failures |
 | |
| Which works on small runs |
From what you see here, I'm sure you can imagine what it would look like if 145 builds all had warnings/failure combos.
So - what do you want to know in the bug? Let's keep it simple, ok? We can add more later and it's important not to create a bug-spammer here that folks will clamor to turn off soon after it goes live.
Off the top of my head, and after talking with Catlee today about it, I think it should look like this:
Try run for $revision with the following comment:This gets you a quick glance at total builds/total requests so you can see that everything is accounted for, and where things might have gone wrong in builds vs. tests but doesn't list the failed/warning builder names so you have to follow the link to get them. Maybe there would be interest in printing what your try syntax request triggered but I'm not sure that's useful in the bug reporting even though it's requested for when a developer is pushing to try. What do you think?
$try_syntax line
S:# W:# F:# (results total) builds complete from N total requests
S:# W:# F:# (results total) tests complete from N total requests
For more information please see http://tbpl.mozilla.org/?tree=MozillaTry&rev=$revision
It would be possible to break down the results further by platform instead of or as well as build/test. Any ideas on how to get that much information across without making a bug comment too verbose? All input appreciated and considered, I'll be trying to land this in the next couple of weeks so comment here, ping me in IRC (lsblakk), or comment in the bug.
6 comments:
A link to where the try builds can be found would also be handy, I think.
URLs. Of. The. Logs. For every job that wasn't green.
If you don't include this I still have to go look at tinderbox(pushlog), which means the info in the bug didn't save me any trouble. (And if I'm coming to the bug weeks or months later, I can't get to the logs at all.)
I suspect that just the short summary suggested towards the end of the post is a very good start.
Until we have oranges a bit more under control, and have much more reliable parsing of exactly which test failed, it'll be hard to put useful information into bugzilla.
Generally I don't care about which platform failed, but rather which exact test failed. This obviously isn't always the case, but it's far more common than not at least for me.
So as long as there are *any* failures or warnings at all (which is almost always the case due to intermittent orange), the most important information is the tbpl link.
Oh, another thing that would be worth including is a link to the directory where all the built binaries are located.
It would be cool if you could embed the tinderbox push log dashboard for that push as a comment. I imagine that's a lot of work though. ;)
> This is just part of a larger project to have patches submitted on a bug get automatically tried out, results posted, and at some point down the road they could even be pushed to trunk after a successful try run so "look Ma! No hands"
Yes! Please make this happen! :)
Post a Comment