Test Day metrics

Source

Well, first of all, those of you who've noticed the schedule around here may be wondering why I'm not advertising this week's Test Day. That would be because it's on UEFI, a technology that's basically only available to motherboard manufacturer's or companies with access to motherboard manufacturers' advance hardware test programs, so the only people who can contribute to this test day are really folks working for Red Hat or motherboard manufacturers - and all of those already know about it. So, that's that. :) Secondly, I was asked by James Laska if I could come up with some metrics for the video driver triage days we've run for Fedora 11. After an absorbing hour or two whacking at the data set by grepping through the test day wiki pages for bug IDs, feeding them into python-bugzilla and then sorting the result via OpenOffice.org (why yes, I AM a fan of Heath Robinson...oh, and of course, I use nedit for its awesome ability to do rectangular selections), here we go for the Nouveau and Radeon test days. Unfortunately I can't get data for Intel yet because of a bug in python-bugzilla, but once Will's fixed that, I shall. So! To start with Nouveau: 40 bugs were reported as a result of the test day. As of now, 14 (35%) are in ASSIGNED state, 10 (25%) are in NEW state, and 16 (40%) are in CLOSED state. Of the CLOSED bugs, 4 were closed as DUPLICATE, 2 were closed as NOTABUG, and the other 10 were closed as RAWHIDE (i.e. fixed). And, Radeon: 46 bugs were reported as a result of the test day. As of now, 26 (57%) are in ASSIGNED state, 14 (30%) are in NEW state, and 6 (13%) are in CLOSED state. Of the CLOSED bugs, 5 were closed as DUPLICATE, and one was closed as RAWHIDE (i.e. fixed). So the Nouveau test day has already been pretty successful: just two weeks after it happened, it's already resulted in ten fixed bugs. Radeon test day has only resulted in one fixed bug so far, but plenty of valid reports which should hopefully be addressed as work continues on the driver. Also, a quarter or so of the bugs filed haven't yet been triaged (that's the difference between NEW and ASSIGNED), so Matej, François and myself should get those done. Intel data coming as soon as the python-bugzilla bug's fixed!

Comments

jspaleta wrote on 2009-04-09 14:33:
This is great start on metrics. If I may let me give you a personal challenge. Before the test day process for f12 begins can you do a few summary graphs concerning f11 test days. I'm thinking a stacked line graphs showing the evolution of bug states over time, showing exactly the information you wrote here..but in graph form and covering more of the test day items. Are bugs filed as part of test days tagged as such in bugzilla for easy culling? -jef
adamw wrote on 2009-04-09 15:07:
I actually wanted to throw up some pie charts, but I am sadly the king of suck when it comes to such things, and couldn't actually figure out how to do it. Unfortunately bugs filed as part of test days aren't tagged, no. We can't currently do that in a 100% reliable way, because the bugs have to be filed manually by the testers. We could put a link on the test day that filed a bug with a certain keyword and ask all testers to file via that link, but we couldn't guarantee that's actually happening. When we have better tools in future this may be possible. For now, as I said, I just used some quick grep / sort / cut foo on the test day results table source to extract a list of bug IDs. It's not actually very hard to do this, and the command's re-usable, now I have it figured out I can do this for any test day in a couple of minutes.
jspaleta wrote on 2009-04-09 21:24:
I can do plots stupid easy with python-matplotlib..its what I do. If you want pie charts shoot me something I can process into a chart. A well labelled csv spreadsheet is fine. and I'll hand you back a script for you to use again. The really interesting thing I'd like to see is if on average test day bugs get resolved more quickly compared to bugs that just dribble in. Is there an enhancement in efficiency not just in testing..but all the way into resolution? If there is that's going to be a pretty good argument to engage upstream developers about, as they'll have a more self-serving reason to want a test day for their codebase. -jef
adamw wrote on 2009-04-09 22:57:
alright, thanks - once I've done the intel data I'll send you the csv's I used. poke me in a few days if you haven't got 'em by then.
AdamW on Linux and more » Blog Archive » Intel Test Day statistics wrote on 2009-04-15 19:58:
[...] up my earlier post, the python-bugzilla was fixed, so here’s the Intel [...]