Fedora 28, broken comments, QA goings on...

Long time no blog, once more!

Fedora 28

So of course, the big news of the week is that Fedora 28 was signed off yesterday and will be coming out on 2018-05-01. If you examine the Fedora 28 schedule, you will observe that this was in fact the originally targeted date for the release. The earliest targeted date.

Yes. It's a Fedora release. Coming out on time. That noise you hear is the approaching meteor that will wipe out all life on Earth. You're welcome. ;)

We've always said the schedules for Fedora are really estimates and we don't consider it a problem if there's a week or two delay to fix up bugs, and that's still the case. We may well wind up slipping again for F29. But hey, it's nice to get it done "on time" just once. I did, in fact, check, and this really is the very first time a Fedora release has ever been entirely on time. Fedora 8 was close - it was only a day late, if you discount a very early draft schedule - but still, a day's a day!

There are, as always, a few bugs I really wish we'd been able to fix for the release. But that's pretty much always the case, and these are no worse than ones we've shipped before. We have to draw a line somewhere, for a distro that releases as often as Fedora. This should be another pretty solid release. My desktop and main laptop are running it already, and it's pretty fine.

Comments: Yes, They're Broken

Quick note for people who keep emailing me: yes, posting comments on this blog appears to be broken. No, I'm not particularly bothered. I actually have been meaning to convert this into an entirely static blog with no commenting for years, I just don't want to deal with Wordpress or any dynamic blog framework really any more. But I never have time to do it, as I want to include existing comments in the conversion, which isn't straightforward. I'm gonna get it done one of these days, though.

openQA news: upgrade tests for updates, aarch64 testing...

I've been doing a lot of miscellaneous stuff I haven't blogged about lately, but here's one thing I'm pretty proud of: Simo and Rob from the FreeIPA team asked if it would be possible to test whether Fedora package updates broke FreeIPA upgrades, as Simo had noticed a case where upgrading a server to Fedora 27 didn't work. We already had tests that test deploying a FreeIPA server and client on one Fedora release, then upgrading both to the next Fedora release and seeing if things still worked - but we weren't running it on updates, we only ran it on nightly composes of Branched and Rawhide. So effectively we know all the way up until a given release comes out whether upgrading works for it, but once it comes out, we didn't know if upgrading is suddenly broken by a later update.

These tests are some of the longest-running we have, so I was a bit worried about whether we'd have the resources to run them on updates, but I figured I'd go ahead and try it, and after a day or two of bashing was able to get it running in staging. After a week, staging seemed to be keeping up with the load, so I've pushed this out into production today. If you look at recent openQA update tests, like this one, you'll see an updates-server-upgrade flavor with a couple of tests in it: these are testing that installing the previous Fedora release, deploying a FreeIPA server and client, then upgrading them to the release the update is for, with the update included, works OK. I'm quite happy with that! I may extend this basic mechanism to also run the Workstation upgrade test as well. Note that these tests don't run for updates that are for the oldest current stable Fedora, as we don't support upgrades from EOL releases (and openQA doesn't keep the necessary base disk images for EOL releases around, so we actually couldn't run the tests).

Aside from that, the biggest openQA news lately is that we got the staging instance testing on aarch64. Here's the aarch64 tests for Fedora 28 Final, for instance. This isn't perfect yet - there are several spurious failures each time the tests run. I think this is because the workers are kind of overloaded, they're a bit short on RAM and especially on storage bandwidth (they each just have a single consumer-grade 7200RPM hard disk). I'm working with infra to try and improve that situation before we consider pushing this into production.

Other QA goings on

One thing that's been quite pleasant for me lately is I'm no longer trying to do quite so much of...everything (and inevitably missing some things). Sumantro and coremodule have done a great job of taking over Test Day co-ordination and some other community-ish tasks, so I don't have to worry about trying to keep up with those any more. Sumantro has been bringing a whole bundle of energy to organizing Test Days and onboarding events, so we've had lots more Test Days these last two cycles, and more people to take part in them, which is great. We've also had more folks taking part in validation testing. It's made life a lot less stressful around here!

I've been mostly concentrating on co-ordinating things like release validation testing, doing a bit of mentoring for the newer team members, and keeping openQA ticking over. It's nice to be able to focus a bit more.