Hi folks! It's Fedora Test Day time once again! Tomorrow, 2016-09-28, will be the Fedora 25 i18n Test Day. i18n is internationalization; i18n and its buddy localization (l10n) (together known as 'g11n', for 'globalization') cover all the work needed to make Fedora usable in languages other than U.S. English and countries other than the U.S.

i18n specifically covers things like the special 'input methods' used to input languages which need something beyond a simple 1:1 key-to-character system, and the fonts and font engine capabilities needed to render non-Latin characters.

The test day page has all the information you need to get testing, and you can enter your results on the result page. Please, if you're familiar with using anything but U.S. English to work with your computer, come along and help us test! As always, the event will be in #fedora-test-day on Freenode IRC. If you don’t know how to use IRC, you can read these instructions, or just use WebIRC.

Hmm, I didn't blog for a while and now I feel like I have three posts all coming along at once...

Best thing I read today: The Register poking the Apple PR department with a series of blunt sticks

Best thing I read yesterday: Jenn Schiffer on blog comments. It took me years to figure out what the article illustrates so perfectly: sometimes (especially if you're male...) you will want to comment on something not because you can actually enlighten, inform or entertain the author or other readers in any way, but just because, well, you're reading it and you have opinions and you're sure everyone else wants to know about that, because you're pretty great.

Don't.

All her other posts are pretty hilarious as well, but the comment sections take the cake. So many angry nerds with STRONG OPINIONS and absolutely zero sense of irony...

Anyhoo! In less random news: I'll be at the Libre Application Summit in Portland next week, in case anyone's interested. There may be some sort of openQA BOF there, depending on who turns up. It'll be fun! Come along! Also Portland is great, so long as you make sure not to get run over by a moustachioed hipster on a fixie on his way from the artisanal coffee shop to the craft beer bar. It's like New York - one of the few places that precisely lives up to its stereotypes...

I find myself with a bit less to blog about lately because I've sort of gradually shaded over to working more on a constant cycle of figuring out why tests are blowing up and trying to help keep the unstable branches composing and working every day, and less on 'community facing' events like Test Days and the like. And - much to my constant surprise - the hideous framework of hacks, bodges and StackOverflow-derived code that constitutes the release validation system seems to keep working more or less smoothly so I don't have to think about it very much.

Nevertheless, the Fedora QA community continues to rock and it's not like Test Days and manual tests and all that other fun stuff isn't happening! Geoff (aka coremodule) did a great job of writing up the Heroes of Fedora 24 blog posts:

• Alpha
• Beta
• Final

It looks like there are some encouraging upward trends from Fedora 23, so that's great news - the wave of new contributors in the last few months has really been doing great work, so thanks to everyone.

Meanwhile, Sumantro and Petr are working on the Fedora 25 Test Days (plus I have to find some time to wedge a Wayland one in there at some point). We should be having events on cellular modem support, storaged - the replacement for udisks, and Fedora Media Writer, so look out for those coming up soon!

Fedora 25 Beta is slated for October 11th, with the freeze coming on September 27th, so those are the next dates we're looking towards for the release process. Current Beta nightly compose testing is looking pretty good so far.

openQA continues to expand its empire by degrees. In the last few weeks we've added tests for text installation, the Fedora Server database server role, KDE and GNOME default browser, the FreeIPA web UI, NFS installs and updates.img sourcing and upgrading an encrypted system. We also now have the capability to forward openQA test results to ResultsDB.

I did quite a lot of work in the last few weeks to try to fix some issues - including bugs in openQA, bugs in the tests, and bugs in Fedora itself - which could cause invalid failures and inconsistent results from day to day, with the aim of making the test results more reliably comparable between runs. For the last few days the results have been pretty consistent, pace a few remaining bugs which occur only intermittently (so sometimes a test will suddenly fail not because of a change in that day's compose, but just because it happened to run into an intermittent bug). This makes it a bit more feasible to think about plans like gating Rawhide pushes on test results, or running openQA on candidate updates and providing feedback via Bodhi.

Hi folks! We are having another 'onboarding' video call to help new Fedora QA recruits get started tomorrow. Sumantro will be leading the call, I'll try and stop by if I can. To join the call, just keep this piratepad open. The call agenda is shown there and it will be used for notes when the call is happening, plus there's a chat panel. Ten minutes before the call starts, Sumantro will post the URL for people to join. Then just join the call and follow along! Please make sure to mute yourself on the call when you're not talking.

Thanks everyone, and welcome to the group, new members!

This is a bit late, but I got sidetracked by the microcode_ctl bug...for anyone who's interesting in getting started with Fedora QA, we'll be holding an 'onboarding' video call at 1700 UTC today (2016-07-08). To join in, just load up the piratepad where we're keeping the agenda and text chat; the URL for the video call will be posted there shortly before it starts. We'll be talking through the various ways you can join in with Fedora QA. Hope to see you there!

Hi folks! So in the last couple of days a significant issue in all Fedora releases has come to our attention, affecting (so far) several systems that use the Intel 'Skylake' hardware platform. Systems that appear to be affected so far - at least with some system firmware versions - include: Lenovo Thinkpad T460, Lenovo Thinkpad x260, Lenovo Yoga 260, ASUS Zenbook UX305CA, Asus Zenbook UX303UB, Samsung Notebook 9.

The problem appears like this: you install a kernel update, and the new kernel fails to boot, failing very early in the boot process (right after the boot loader). Older kernels boot fine.

For those who don't want to read much, there are a few workarounds available if this is the bug you're hitting:

1. Boot an older kernel.
2. Boot with the kernel parameter dis_ucode_ldr.
3. Downgrade the microcode_ctl package to version 2.1-11 or earlier and force an initramfs rebuild, either with dracut or by reinstalling the packages for the kernels that don't boot.

You may also find that a system firmware update is available from your system manufacturer, and updating the system firmware makes the bug go away. So do please check with your manufacturer's site and try updating your system firmware if there's an update available.

We're sorry for the inconvenience, and we're looking at better fixes at present.

The story behind this bug is that it's not actually a kernel bug at all. It's a bug in microcode_ctl . This is a package which contains both processor 'microcode' updates and a loader for such updates, for Intel processors. You can think of processor microcode as being kind of like firmware for your processor; this mechanism lets Intel correct bugs and improve behaviour in processors after they've been released and shipped out. It also occasionally lets them break stuff, like in this case. :)

The way this mechanism works on most Linux distros (this bug is affecting other distros as well as Fedora, btw) is that if there's a microcode update for the CPU in your system at the time an initramfs is built, the update and a loader mechanism for it are built into the initramfs in such a way that they load very early during initramfs initialization, which is as early in the boot process as we can manage. If there is no microcode update for your CPU, you get an initramfs without this trickery.

Prior to microcode_ctl-2.1-12 there was no microcode update for the affected CPUs. microcode_ctl-2.1-12 has added a microcode update for these CPUs, so all initramfs'es built after microcode_ctl is updated to that version will include the update. The bug seems to be that on some system firmwares, the microcode load fails and hangs the system. On other system firmwares the microcode loads fine and the system boots.

The reason the bug appears when you update your kernel - rather than when you update microcode_ctl - is simply that updating microcode_ctl does not trigger any initramfs rebuilds; your existing installed kernels will still have initramfs'es with no microcode update loader mechanism. But when you install a new kernel, a new initramfs is generated for it, and now it will include the microcode update, and thus hit the bug.

This is why you can work around the issue by downgrading microcode_ctl and then regenerating the initramfs for affected kernels. It also means that if you regenerate the initramfs for a kernel that was working fine after microcode_ctl has been updated, that kernel will stop working.

The dis_ucode_ldr kernel parameter simply disables this microcode loading mechanism, which obviously avoids the bug happening.

So I've had a couple of issues with my personal FreeIPA deployment in the last couple of months that I never managed to dig into properly because of work on Fedora 24. But now F24 is done, I had time to figure them out, finally.

The first bug is simpler and probably has a wider impact. Fedora 24 ships with sudo 1.8.16. Unfortunately, there seems to be an issue between this version of sudo and FreeIPA for sudo rules that use FreeIPA host groups. If you have a FreeIPA sudo rule which says 'allow user foo to sudo on all systems in host group bar', it probably won't work with sudo 1.8.16. You can fix it for the short term by downgrading to sudo 1.8.15. The bug is fixed upstream in sudo 1.8.17, so hopefully we'll either get a 1.8.17 update or a backport of the fix to 1.8.16 for Fedora 24 soon.

The second bug was a hell of a lot more complex to figure out, but we got there in the end! There was a change to FreeIPA between Fedora 21 and Fedora 22 whereby certificate profiles for the CA portion of FreeIPA are now stored in LDAP, rather than on the filesystem, by default. Things are supposed to be set up such that when you upgrade an installed system to the FreeIPA and pki-core versions that implement this change, the FreeIPA upgrade process changes the pki-core config file to use the LDAP storage system, and imports the profiles into the LDAP database.

I upgraded my server to Fedora 23 right when it came out, and the upgrade process was pretty rough at that time. I don't have enough logs left to figure out exactly what went wrong, but it looks like one of my runs of the upgrade process happened to break between the config file change and the import of the profiles to the database, so pki-core was set up to use the LDAP storage system, but the profiles weren't actually in the LDAP database.

This is obviously broken, but it didn't actually prevent FreeIPA working (in most ways - I imagine certificate management might have been broken, if I used it more) until quite recently. A change in pki-core results in it failing to start entirely if you're in this situation. That change landed in Fedora in pki-core-10.2.6-14 (for F23 and F24) and pki-core-10.2.6-10.fc22 (for F22). From that build on, pki-tomcat will fail to start up if you have the messed up configuration, rather than starting up but being a bit broken.

We did figure out that since the relevant pki-core packages landed in F22+, upgrades of FreeIPA servers from Fedora 21 are broken. The problem there is that the upgrade process tries to edit the configuration file, restart pki-tomcat, then import the profiles - but when it tries to restart pki-tomcat that fails because of the bug, and the profile import can never happen. So the upgrade process fails and leaves the system in the broken state.

So if you tried to upgrade a Fedora 21 (or earlier) FreeIPA server recently and found FreeIPA would not start after the upgrade, this might be the reason. The workaround is actually not too hard, though.

First, downgrade all pki-core packages to the version right before the change landed in Fedora. For instance, for Fedora 23, the easiest way to do this is - in an empty directory - run koji download-build pki-core-10.2.6-13.fc23, then dnf downgrade *.rpm. For F22 or F24, just use pki-core-10.2.6-13.fc24 or pki-core-10.2.6-9.fc22; on F24 you may also need to do rpm -e --nodeps pki-base-java before the downgrade (it's a new package that was added in later pki-core builds, and dnf downgrade can't really cope with removing packages that were added by the new build, so you have to hit it with the big stick). Second, edit /etc/pki/pki-tomcat/ca/CS.cfg and replace subsystem.1.class=com.netscape.cmscore.profile.LDAPProfileSubsystem with subsystem.1.class=com.netscape.cmscore.profile.ProfileSubsystem. Finally, run ipa-server-upgrade as root. It should run successfully, and at this point the certificate profiles will be imported - so you can re-update the pki-core packages to the latest version, and things should work properly.

I'm hoping to implement some FreeIPA upgrade tests in openQA this cycle; now we have clean install FreeIPA testing working, extending it to test upgrades shouldn't be too hard, and hopefully would help us avoid some upgrade issues in the future...

NOTE: this post is entirely personal. The opinions are my own and do not represent Fedora or Red Hat. The facts, however, are all 100% truthy. ;) Just to make it 100% clear for any visiting journalists etc. who don't know me: I work for Red Hat, on Fedora. I am not unbiased and am not claiming to be, but I am claiming that the concrete stuff I say below is true.

You may have read some stuff this week about an application delivery mechanism called Snappy and how it's going to unite all distributions and kill apt and rpm!

This is, to put it diplomatically, a heaping pile of steaming bullshit. You may not be surprised to learn that said pile has been served by the Canonical press department.

The source of all this press was this gem of a press release, which has been widely covered in a fairly...uncritical way by several outlets. Even Ars Technica, which is usually fairly good at doing actual journalism rather than just unquestioningly paraphrasing press releases, gave it a pretty anodyne write-up.

The press release and the stories together give you the strong impression that this thing called Snappy is going to be the cross-distribution future of application delivery, and it's all ready for use today and lots of major distributions are buying into it. In the press release you'll find stuff like this:

"Developers from multiple Linux distributions and companies today announced collaboration on the “snap” universal Linux package format, enabling a single binary package to work perfectly and securely on any Linux desktop, server, cloud or device."

The stories have headlines like "Adios apt and yum? Ubuntu’s snap apps are coming to distros everywhere" and "Snap Packages Become Universal Binary Format for All GNU/Linux Distributions" (jeez, I particularly love that one).

So what are the problems with this happy-clappy story? Several of them!

First let's be clear: Snappy is a Canonical project. The press release was issued, I think, sort of as if it came from some sort of independent or cross-vendor project, and there's the snapcraft.io site to back up that impression, but every Snappy committer is a Canonical employee, and contributions to Snappy require signing the notorious Canonical CLA:

"Contributions are always welcome! Please make sure that you sign the Canonical contributor licence agreement at http://www.ubuntu.com/legal/contributors"

Now, does Snappy actually have the cross-distribution buy-in that the press release claims (but never outright states) that it has? No. The press release sure sounds superficially impressive:

"Developers from multiple Linux distributions...Snaps now work natively on Arch, Debian, Fedora, Kubuntu, Lubuntu, Ubuntu GNOME, Ubuntu Kylin, Ubuntu MATE, Ubuntu Unity, and Xubuntu...Together, these distributions represent the vast majority of common Linux usage on the desktop, server and cloud."

but it's a pretty big mis-representation. The other distributions cited have not actually declared their support for Snappy and said 'yes, this is how we want applications to be distributed in future'. Canonical employees have independently built and released Snappy packages for those distributions. This is the basis of all the press release's claims. For instance, the Snappy packages for Fedora are in a COPR - COPR is a system like PPA that lets anyone build packages - maintained by a Canonical employee. The sum total of communication between Canonical and Fedora before the release of this press release was that they mailed us asking about the process of packaging snappy for Fedora, and we told them about the main packaging process and COPR. They certainly did not in any way inform Fedora that they were going to send out a press release strongly implying that Fedora, along with every other distro in the world, was now a happy traveler on the Snappy bandwagon.

There is in fact another system with very similar goals, which is now called Flatpak and was previously called xdg-app. To be as fair as I can, I'll say that Flatpak is quite heavily Red Hat influenced: the main author of Flatpak is Alex Larsson, a Red Hat employee. It is not, however, a "Red Hat project" to anything like the extent Snappy is a Canonical project. There are more than 20 other committers to Flatpak, most of whom are not RH employees (and including contributors to several other distributions). Flatpak is not under any corporate CLA. Insofar as Fedora is supporting one of these systems, it's behind Flatpak. No other distribution besides Ubuntu is particularly committed to either system, so far as I can tell. Flatpak and Snappy both began, so far as I could find from internet research, in December 2014. Canonical's press release, of course, doesn't even acknowledge Flatpak's existence...which is kind of par for the PR course, but you'd think at least some journalists might go out and do a bit of independent research.

UPDATE: since writing this post I've also been made aware of another system, AppImage, which has been around somewhat longer than Flatpak or Snappy (though not necessarily their forerunners). I know little about it so I will say little, but one thing to note is that - so I've heard - it does not attempt to do sandboxing like Snappy and Flatpak do, which is a major feature of those two implementations. It's purely an app bundle format. But hey, it's a choice! And it's been around a while!

Neither Snappy nor Flatpak is at all close to being 'done', in the sense of being a credible system for cross-platform app distribution with buy-in from software publishers and distributions. The PR's claim that Snappy enables "a single binary package to work perfectly and securely on any Linux desktop, server, cloud or device" sounds lovely, doesn't it? Let's take a look at the truth. Taking Fedora as an example, the Snappy install instructions for Fedora - go to the Snappy site and click the Fedora logo - say:

# SELinux support is in beta, so on Fedora 24 you currently have to:
sudo setenforce 0

well, that doesn't seem terribly 'secure' or 'perfect' now, does it? Along the same lines, the Fedora packages are actually compiled with Snappy's confinement disabled. Confinement being the entirety of what's supposed to be secure about this form of app distribution. If confinement isn't turned on, you've basically just got a big blob with uncontrolled access to the system. Good luck with that.

AIUI, the builds for other distributions are in similar states.

Note that neither Snappy nor Flatpak can possibly provide meaningful confinement of apps running under X11, as mjg59 illustrated. Flatpak will only provide meaningful confinement with Wayland. Snappy, of course, is designed to work with Mir, though they claim it can/could (not sure which) also work with Wayland. But the point here is that neither Wayland nor Mir is out there in real widespread use by Linux users at present, yet here's Canonical happily glossing over that point while they talk about how Snappy right now allows "a single binary package to work perfectly and securely on any Linux desktop".

At the time this Panglossian PR was sent out, there were exactly two actual useful applications available as snaps: LibreOffice and Krita. Phoronix quickly found that the LibreOffice snap was huge (over 1GB in size) and buggy. The size issue was quite quickly resolved, but this just goes to show that reality is vastly different from Canonical's claims. This stuff is in early Alpha or proof-of-concept state. It is not remotely 'done' and ready for practical use in the real world outside of the very limited contexts where Canonical was already using it.

Neither is Flatpak, of course. But this is why Flatpak's developers have been communicating with technical conference presentations and blog posts and trying to build a dialog with application developers and distributors, rather than issuing press releases trumpeting how great Flatpak is and how it's ready to kill apt RIGHT NOW.

Here's another interesting thing about Snaps: the server end (the 'app store' bit of the equation) is closed source, and Canonical have been refusing to tell anyone how to run their own 'app store' - see the comment from Cassidy James Blaede, of Elementary. If you want to distribute your snaps, your choices are 1) publish it through the Canonical store, entirely under Canonical's control, 2) upload it as a file and tell people to use the CLI to install it, or 3) try to figure out how to reconfigure the snap client to use a different server by reading the source code, then write your own server end from scratch, and tell your users to do that. Hmm.

So: Snappy is, like Flatpak, a heavily-under-development, interesting attempt to provide an app store-like app provision mechanism for Linux. It is not finished, it is not close to finished. It is not independent or cross-distribution, it is entirely controlled by Canonical. It does not have, so far as I can tell, meaningful buy-in from a single major distribution outside of Ubuntu. It does not work properly on other distributions yet and it likely will not do so in the near future.

But apart from that, sure, it's all ready to kill apt and dnf tomorrow!

sigh

Now I'm sure I will get criticized for being mean and nasty and cynical and attacking Canonical instead of being constructive and all they want to do is make things better for everyone, Adam why are you such an ass?

Well, if Canonical actually wanted to work constructively with others, the way to do so would be to talk to them. We have forums for cross-distribution and cross-project collaboration. Lots of them. We have tech conferences where you can go and talk about your project and try to get buy-in for it. Canonical could have come to other distributions and said, hey, how about we all try to get together behind a single format and a common delivery mechanism for this kind of a confined app bundle?

But they didn't. They just decided to send out a wildly misleading press release and actively encourage the specialist press to report that Snappy was all set to take over the world and everyone was super happy with that.

That's not being constructive or working together with others. That's being a bunch of asshats and trying to present the rest of the community with a fait accompli - and notably, a fait accompli in which Canonical holds all the strings (by means of the Canonical CLA controlling contributions to the client end, and the closed source, closed shop server end that is owned entirely by Canonical).

You probably read a more sensationalized version of this on Phoronix already, but we had the Fedora 24 go/no-go meeting today and decided to slip for a week. The sole reason for this was this blocker bug, which is to do with booting Windows from the Fedora boot menu on a UEFI system, wasn't fixed in time. If you don't need to do that, you could frankly go ahead and grab a Fedora 24 nightly and use it. It'll be fine. Go ahead, have a ball.

Aside from helping our awesome new interns Sumantro and company with validation testing and running test days, I've been spending most of my time lately teaching openQA new stuff. The robot's getting pretty good now! We're testing install in Russian, running a terminal in KDE and GNOME, the Server default firewall configuration, Cockpit, FreeIPA enrolment via Cockpit, installing to an iSCSI target, and - well, I just finished this today - several NFS install tests.

Getting inter-dependent tests working in openQA has been a big help in letting us extend coverage, it's been fun trying to knock out as many of the validation tests as possible, and we're doing pretty well now. It's definitely very nice not to have to painfully re-do all the setup for stuff like the FreeIPA and iSCSI tests over and over again.

I also reported and helped fix quite a few bugs that were found by these tests, including this fun keyboard layout bug which took a couple of days of research to figure out (well, I say 'research', one whole afternoon of it was manually bisecting the offending systemd commit with Fedora 21 RPM builds...)

Here, children, gather round and let Uncle Adam tell you a story. Are you ready? Here we go!

Once upon a time there were some test cases for a thing called FreeIPA. And once upon a time, two poor people called Adam and Stephen did have to run these tests manually, all the goddamn time.

And lo, Adam and Stephen were sad, because running these test cases manually sucked big-time.

"I know!", thought Adam. "Let's automate them!"

But Adam had doubts. "Hmm," he thought, "I'm pretty sure that company with the funky hats already has some automated tests for this. Maybe I should wait until Beaker is finally deployed properly for Fedora and use those tests instead, to save duplicating work?"

Then Stephen found himself moving on to different work, and Adam realized he would have to do the tests himself all the damn time. And Adam ran the tests manually one more time and halfway through remembering how to do static IP configuration via dracut on the kernel command line he didst think to himself, "screw this for a game of soldiers".

And so Adam didst go and read to himself the openQA advanced networking documentation. "Bloody hell," he thought, "this is tough going".

Yet he did persevere, and at length, he conquered the mountains of Open vSwitch, including the feared climb up the Iptables Face, and he did have a couple of PoC tests working. And he was happy!

His travails were not yet ended, however. Next didst he, with much wailing and gnashing of teeth, manage to get the entire Open vSwitch configuration working via Ansible for the official openQA instances in Fedora Infrastructure.

Finally didst he, with much trial and error, refine and extend the tests until they covered several of the validation tests, and actually worked. And lo, didst he dump two huge pull requests on the tracker, in the manner of the famed "mic drop", and go for a beer.

Seriously, though, this is pretty cool and I'm happy I got it working. openQA by default uses qemu user-mode networking for the VMs that run the tests, which is fine for a lot of stuff - the VM can see the outside world just fine - but means they can't talk to each other. If you want to test something like FreeIPA where you need two or more tests (here, the server and a client) running simultaneously and talking to each other, you need to do some custom networking config.

openQA has some integration with Open vSwitch and it's what the SUSE folks use, so I went with that. You basically have to create a tap device for each worker instance and use something like OVS to connect those devices together with a virtual bridge or whatever so the test VMs can communicate. The VMs also need to access the per-job web server that os-autoinst runs for the worker to upload logs to and download scripts to run from (in some cases), so in the reference set up you have that bind to the bridge interface and ensure the firewalling is set up so the VMs can reach it. And if you need the VMs to have access to the external network, as we do for FreeIPA testing (dnf and rolekit just do not want to work without access to the repositories), you have to basically set up NAT routing for the traffic from the VMs. It's lots of network configuration fun!

But I worked it all out. So both the tests are configured as 'children' of the install_default test for the Server DVD ISO, which just runs a default installation then uploads the hard disk image of the installed system.

Once that test is done, both the server and client tests boot up from that hard disk image. The client test recognizes it has to wait for the server, and just sits at the boot menu till the server signals that it's ready (this is all part of openQA's parallel test support). The server logs in, sets its hostname to an appropriate one for a FreeIPA server (FreeIPA does not like localhost.localdomain, it requires an FQDN), and reboots.

Then it re-configures the network - since we're effectively on a private network now, with no DHCP server or anything - to use a static IP. Then it copies /etc/resolv.conf from the worker host, which sounds like a terrible idea at first but is actually a pretty good way to ensure it can resolve names properly (I first thought about doing this as a joke, then found it was how SUSE do it and realized it's actually probably the best option). Now it deploys the 'domain controller' role (i.e. sets itself up as a FreeIPA server) and runs the first set of required rolectl sanity checks from QA:Testcase_Server_role_deploy. Then it sets an enrolment password for the client, creates a couple of user accounts, tweaks the login policy a bit, and then it sends the signal (actually a mutex lock) to the client that it's ready and the client can go ahead. Then it waits for the client to complete.

The client wakes up, does a kickstart install with client enrolment using the password the server set up and an appropriate static IP networking configuration. The FreeIPA server acts as a DNS server, so the client doesn't need to fiddle about with that. The installed system boots, and we log in as root, then we check that we did indeed become a member of the domain properly (as per QA:Testcase_realmd_join_kickstart). Then we kinit as each test user and set a permanent password, then we try logging in as each user, expecting success with the first and permission denied with the second (as was the policy we set on the server earlier).

At this point the client has fulfilled the requirements of the kickstart joining test case, QA:Testcase_domain_client_authenticate and QA:Testcase_FreeIPA_realmd_login, so it finishes up. The server sees that the client has finished, wakes up, does the rest of the rolectl sanity tests, and completes.

And Adam says "thank God I don't have to all that crap manually again", and goes for a beer. :)

Time for another of my increasingly irregular blog posts :)

I was at LinuxFest NorthWest 2016 last weekend. I've been going to LFNW for several years now, and I look forward to it every year - it's just a great conference, which has managed to grow to nearly 2000 registrations this year while keeping its community/grassroots feel. The talks are always widely varied and interesting, and there's a great feeling that you could run into anyone doing anything - I spent an hour or two at the social event talking to a group of college students who run a college radio station entirely on F/OSS, which was awesome.

As always, Jeff Sandys and Jeff Fitzmaurice did a great job setting up and manning the booth, and preparing for the pre-event games night, which has been sponsored by Fedora for the last few years and is now I think bigger than the whole conference used to be six or seven years ago! We had a games theme for the booth this year, and the Super Tux Kart big screen TV setup was a big hit - every time I was at the booth there were at least two people playing. People were also impressed at how easy it is to install Steam on Fedora, which was great.

LFNW is extremely well run for a non-profit volunteer-organized free-to-attend conference (in fact it's well-run compared to some extremely non-free to attend conferences I've been to!) and this year had every talk recorded, so I can give you YouTube links to all the good ones I saw!

On Saturday morning I saw Adam Monsen's The command line - a versatile, future-proof computing environment, which was awesome - it's basically 45 minutes of cool tools and tricks for efficient terminal use. I think everyone learned at least one new thing, it's absolutely worth watching and you'll probably find something that'll make your life easier! Adam's source and a follow-up blog post are on the talk page.

The infamous Bryan Lunduke gave a new talk, Linux is [censored] Weird, which was a lot of fun but a bit more lightweight than his equally-infamous "Linux sucks" talk series. The highlight of the talk was definitely the Linux-powered cow milking system, though - and the fact that a guy in the audience had done some work on it and had a photo of the admin interface!

Corey Quinn gave an excellent talk called Docker Must Die, which sold me on the basis of the title alone but turned out to be a really tight and extremely well-presented rundown of the inconvenient real-world truths that are skated over by the more hardcore devops evangelists - all the ways in which development is never really quite the same as production, no matter how shiny your containers. Corey's a funny guy, too. And he wasn't done yet! He gave another talk on Sunday morning, Terrible Ideas in Git, which covered some really awful things he's seen people doing with git, and by extension talked about some best practices. Definitely worth watching for anyone collaborating on development projects with git, and again, very well presented.

E. Dunham did a great session called Rust's Community Automation, which was a really interesting look at how Rust, as a fairly new project backed by an organization with a lot of community experience, has been able to build its community from the ground up in a way that avoids some of the issues older F/OSS communities have experienced, using social tools like a strong code of conduct with clearly defined enforcement and modern tools and process design that really help new community members come on board in a welcoming way without compromising the project's goals or quality requirements. Definitely recommended for anyone with an interest in community management - lots of great ideas.

Sriram Ramkrishna did a talk called Application sandboxing, which was a kind of overview of xdg-app in terms of what it's meant to achieve instead of its technical details. It's great to see people getting the word out on xdg-app, which is probably going to be a big story in the future. I thought the talk was slightly 1.0 and could be fleshed out a bit more in future revisions, but it's definitely worth a watch if you're currently thinking "what the hell is this xdg-app thing?"

And of course what was clearly the best talk of the conference was saved for last: my amazing talk on openQA, in which I graciously let some jackwagon called Richard Brown talk for a bit ;)

No, Richard - the openSUSE Chairman, no less - and I did a joint talk, which I really enjoyed. It was mostly Richard's slide deck, as he already has a pretty polished openQA talk, but we put in a substantial 'openQA on Fedora' section and I did the talking for some of the 'general introduction' section, since Richard and I are very much on the same page about what the focus of distribution QA should be and how openQA goes a long way to addressing those. We tried to focus too on the ways in which the SUSE and Fedora teams have been collaborating on openQA work. It seems like people really liked to see two distributions working together for the benefit of both, we had lots of feedback to that effect, and I think the talk worked well as an introduction to openQA as well - at least a couple of folks told us afterwards that they were interested in testing it out for their own use. You can find the slides in Richard's github, and the Fedora slides have notes with links to lots of background info.

The Saturday social event was at the Hotel Bellwether's ballroom, which is in a really nice location but is really just a big room - people definitely enjoyed being able to take in the view outside, but I did hear quite a few people saying it was fun in previous years to have some kind of venue (like the Spark Museum to look around while chatting and drinking. It's a bit tricky for the organizers now, though, as the conference is so big - there are very few venues in Bellingham which can fit everyone in! Still, the appetisers were good and we had nice beer courtesy of Boundary Bay (IIRC). BTW, here's a PSA: folks, when you go to a conference and the beer is free, that doesn't mean you can stiff the servers! They're working late for not a whole ton of money, and for Pete's sake, you don't even have to pay for the beer! Tip them.

As is almost an annual tradition by now, after the conference I hung out with the awesome Jakob Perry and Emilie Nouveau, who've been involved in organizing the event for years now, and a few of their cohorts. They're super awesome people and they do a great job on the event. I take the train home on the Sunday evening and there's always a few hours between the end of the conference and the train leaving, so they're kind enough when they have the time to drive me to their board game meetup and then drop me at the station later. It's the one time a year I play board games and I always get killed, but I always have a lot of fun, so if you're reading this, thanks again folks!

I meant to talk about Fedora 24 Beta and some fun work I've been doing lately to get openQA testing FreeIPA (ooh look, stealthy sneak preview!), but the LFNW write-up got longer than I expected, so I'll leave it here! Stay tuned for those posts next week :)