Heroes of Fedora is back. First, we’re looking at Fedora 23 Alpha.
Fedora 23 was released recently, and as is now traditional, it’s time we celebrate all the fine folks who contributed to testing with Heroes of Fedora! Heroes of Fedora (HoF) features some exciting statistics analyzing major areas of contributions. Your regular host Roshi is busy at the moment, so I’m standing in.
As usual we’ll be looking at three major areas of contribution: updates testing, release validation, and bug reports, across the three release milestones – Alpha, Beta, and Final. In this post we’ll be covering Alpha. In the last post, we’ll also see numbers from the Fedora 23 Test Days. There are other important areas QA covers, so these posts don’t necessarily include everyone who helped out, but they’re the areas we can easily generate statistics for.
Updates Testing
During the Fedora 23 cycle, Fedora’s update feedback system – Bodhi – got a major upgrade, and the tools we use for sending feedback needed some fixes. So it was a bit harder than usual to send feedback, and the numbers may be a bit down on the previous cycles. Comparing to the Fedora 22 data, we have 8 fewer testers and just about 100 fewer comments. Thanks to all the testers who worked through the issues to provide the much-needed feedback!
Test period: Fedora 23 Alpha (2015-07-14 – 2015-08-11)
Testers: 55
Comments: 135
Validation Tests
Validation testing is when we run a fixed set of tests against a compose to see if it meets Fedora’s release quality standards, and file bugs on the problems we find. Up until now in these Heroes posts, we’ve only included stats for tests of TC and RC (Test Compose and Release Candidate) builds, but for the last few releases, we’ve also been doing validation testing on nightly builds, before Alpha TC1 is built. So I’ve decided to include anothertable with the numbers for the nightly compose validation events, as well as the table for the Alpha TCs and RCs. Here are the nightly stats:
Test period: Fedora 23 nightly validation events
Testers: 7
Reports: 131
Unique referenced bugs: 38
| Name | Reports submitted | Referenced bugs1 |
|---|---|---|
| juliuxpigface | 73 | 1166978 1183300 1204574 1223537 1228834 1229012 1229665 1230396 1235003 1235033 1238803 1240259 1240802 1241228 1241241 1242055 1242080 1242084 1242143 1242242 1242273 1243543 1243743 1245323 1245337 1245348 1245789 1245792 1246252 (29) |
| satellit | 47 | 1229665 1230293 1240354 1244431 1244973 (5) |
| leinfeva | 5 | |
| adamwill | 2 | 1243168 1244023 (2) |
| pschindl | 2 | 1244023 1245191 (2) |
| pwhalen | 1 | 1244558 (1) |
| kparal | 1 | 1235323 (1) |
and here are the Alpha TC/RC numbers:
Test period: Fedora 23 Alpha
Testers: 16
Reports: 321
Unique referenced bugs: 31
| Name | Reports submitted | Referenced bugs1 |
|---|---|---|
| pwhalen | 54 | 1185447 1244558 1247382 1251254 (4) |
| kparal | 49 | 1245423 1252443 1252465 1252756 1252902 (5) |
| pschindl | 43 | 1185447 1247622 1247747 1250409 681250 (5) |
| juliuxpigface | 28 | 1202113 1223537 1249795 1250712 1250738 1250740 1250742 (7) |
| satellit | 28 | 1240354 1244261 1244442 1246915 1247193 1247382 1251298 (7) |
| sgallagh | 26 | 1250724 (1) |
| roshi | 25 | 1247382 (1) |
| adamwill | 17 | 1134882 1245423 1245838 1246901 1250737 (5) |
| lbrabec | 16 | 1247747 1250409 (2) |
| robatino | 15 | |
| kevin | 6 | |
| mattdm | 5 | |
| dustymabe | 5 | |
| alexpl | 2 | |
| cmurf | 1 | |
| kinokoio | 1 |
1 This is a list of bug reports referenced in test results. The bug itself may not be created by the same person.
Comparing the Alpha numbers to Fedora 22, we have 4 fewer testers and just over 100 fewer results. However, this isn’t quite the bad news it looks like. The number of testers reporting 10 or more results is actually one higher (10 vs. 9) – we lost testers from the ‘long tail’ of folks reporting fewer than 10 results, which is a fairly volatile set and might include things as simple as typos. The lower number of test results is most likely due to a couple of factors: there were fewer composes for 23 Alpha (4) than 22 Alpha (5), and we expanded the scope of openQA-based automated validation testing for Fedora 23. These numbers include only human tests, and the more tests we let the robots do, the fewer we need humans to do!
Bug Reports
Just like it sounds – we simply count the number of bugs filed against the release during the Alpha test period. Comparing to Fedora 22 here, we have 26 fewer reporters and 96 fewer bugs. This is clearly a great indication that Fedora is far less buggy than it used to be. Or it means no-one cares about Fedora any more. Interpret as you see fit! 🙂
Test period: Fedora 23 Alpha (2015-07-14 – 2015-08-11)
Reporters: 129
New reports: 317
| Name | Reports submitted | Excess reports | Accepted blockers |
|---|---|---|---|
| Adam Williamson | 22 | 1 (4%) | 2 |
| Tomas Radej | 22 | 0 (0%) | 0 |
| Zbigniew Jędrzejewski-Szmek | 14 | 1 (7%) | 0 |
| Mikhail | 13 | 1 (7%) | 0 |
| Petr Pisar | 13 | 0 (0%) | 0 |
| Peter Robinson | 11 | 0 (0%) | 0 |
| satellit | 11 | 1 (9%) | 0 |
| Giulio ‘juliuxpigface’ | 10 | 0 (0%) | 0 |
| leinfeva | 9 | 3 (33%) | 0 |
| Petr Schindler | 7 | 2 (28%) | 1 |
| Dan Horák | 6 | 0 (0%) | 0 |
| Ilya Gradina | 6 | 0 (0%) | 0 |
| Kamil Páral | 5 | 0 (0%) | 0 |
| Leslie Satenstein | 5 | 3 (60%) | 0 |
| Miroslav Grepl | 5 | 0 (0%) | 0 |
| Lukas Vrabec | 4 | 0 (0%) | 0 |
| Mikko Tiihonen | 4 | 0 (0%) | 0 |
| Stephen Gallagher | 3 | 0 (0%) | 1 |
| Bruno Roberto Zanuzzo | 3 | 0 (0%) | 0 |
| Christian Stadelmann | 3 | 0 (0%) | 0 |
| Diogo Campos | 3 | 0 (0%) | 0 |
| Jan Kurik | 3 | 0 (0%) | 0 |
| Michal Toman | 3 | 0 (0%) | 0 |
| Petr Lautrbach | 3 | 0 (0%) | 0 |
| Pinkie Pie | 3 | 1 (33%) | 0 |
| Rex Dieter | 3 | 0 (0%) | 0 |
| Scott Robbins | 3 | 0 (0%) | 0 |
| Ville Skyttä | 3 | 1 (33%) | 0 |
| …and also 101 other reporters who created less than 3 reports each, but 117 reports combined! | |||
Conclusion
We can definitely see some MVPs in the numbers: big thanks to juliuxpigface, satellit, and leinfeva (Carlos Morel-Riquelme) for their all-around contributions! These numbers reflect a phase which went pretty smoothly, with the early testing, improvements to openQA coverage, and anaconda’s own automated testing all helping us keep things in shape quite effectively. Check in soon for part 2, covering the Beta phase!
Recent Comments