QA happenings, post-F23

Hi folks! I haven't blogged for a while, so I thought I'd write up a few notes on what's going on in QA now Fedora 23 has been released.

We've been working for the last few years to improve ongoing validation testing outside of the Alpha/Beta/Final TC/RC system, so of course, we're testing Fedora 24 already. openQA is testing Rawhide nightly, and we're getting nightly builds nominated for manual validation testing regularly. These validation test events are announced on test-announce and you can always find the current validation event summary. We've found several release blocker and showstopper bugs already, and gotten several of them fixed; today we saw that all the openQA tests had failed, so I checked it out and found that a new dracut build was causing the installer images not to boot.

We also wrote the Fedora 23 Common Bugs page, and have been keeping it up to date (along with the Fedora 22 Common Bugs page). It's always a good idea to check out those pages if you're running into what seems like a major bug - you may find more information and help there.

We've been working on an improved deployment of openQA. As I wrote about before, the current 'semi-official' Fedora openQA instance is running on a machine inside Red Hat's firewall, so only Red Hat folks can see the tests. This isn't what we want, of course, so we've been working for some time to make it possible to deploy openQA in the Fedora infrastructure. This is almost complete, now - there is in fact a staging openQA instance running in the Fedora data centre now, running tests nightly; we're only waiting on some firewall rule changes to make it publicly visible. We should have both staging and production openQA instances fully up and running pretty soon, which will have two major benefits: more capacity (the new production instance should be able to handle 3-4x as many tests as the current instance) and of course allowing non-RH folks to see the tests (which means we can link to them in report emails, Bugzilla reports and so forth as well).

We've also been working on improving the blocker bug process, specifically the handling of 'special blockers'. For some years now, we've been finding release blocker bugs for which the fix doesn't actually go onto the new release media; we've informally referred to these bugs as 'special blockers' and had some ad hoc workarounds for dealing with them, but we really need something better. There are two groups of 'special' blockers: bugs where we need a fix to go into the 0-day updates for the new release (the set of updates which is already in the update repository on release day), and bugs where we need a fix to go out as an update for the previous release(s) (so, for the release of Fedora 23, we had a couple of cases where we needed to ship updates for Fedora 22 and Fedora 21). The current process gives us a strong guarantee that the release media won't be built without fixes for all blocker bugs that need to go on the media, but we don't really have any process in place for making sure either kind of 'special blocker' actually gets fixed in time - we often say something like "we'll have to make sure we send out an update for that before the release", but we don't have any process in place to make sure that actually happens, and sometimes it doesn't.

So we have a mailing list discussion going at present (sorry, I can't link to it, as archives haven't been imported to Hyperkitty yet...) about how we can better track those 'special' blockers, and how the release process could be adjusted to ensure the fixes actually appear in the appropriate places in time for the release date. These changes should start happening pretty soon, well in time for Fedora 24 Alpha.

Taskotron work continues at a good pace, with disposable clients now almost ready for deployment, and several interesting plans for new tests that should help ensure consistent repository quality.

Of course, things like update testing roll on as always, with many QA team members volunteering lots of their time to test updates - this is always a great way to help out the project if you have a little time! If you dropped out of testing while Fedora Easy Karma was broken by the Bodhi 2 transition, good news - it's working again now!

The Fedora 24 Test Day cycle should start ramping up soon with a call for Test Days, and the Heroes of Fedora posts for the Fedora 24 cycle should be coming soon too. Before Fedora 24 testing really ramps up in earnest, we're hoping to be able to automate even more of the release validation tests - in openQA, taskotron, Beaker, or whatever else works best! We're looking at some of the simpler Desktop validation tests, the Base tests, and perhaps some of the Server tests as targets for this cycle.