Flock 2014, and other happenings

I had a really great time at this year's Flock in Prague. As always it was a packed event with a really positive and productive vibe - clusters of people all over the place hacking on things. I posted a lot about it on YAY Diaspora and BOO Google+, but here's a long-form post-event wrap-up.

I got into town in the afternoon of day 0, and did the thing I like to do, which is to find a pool and go for a swim - it's an interesting way to see places, and gives you a bit of a feel for the local infrastucture - it's like Van Halen's brown M&Ms, a sort of handy little benchmark for how the whole system works. Last time I was in Prague I wound up in the weird quiet little pool in the downtown YMCA, but this time I found a public pool a 20 minute bus ride north of the Flock hotel. It was interestingly similar to a typical British public pool - one modestly sized 25m pool, clean, functional, pretty spartan, very 1970s public institution. Though they use what I think of as the Hong Kong system for lap swimming, where there aren't many marked lanes and instead everyone swims in a big mass, trying to do on-the-fly collision avoidance. It gets interesting when you get a clash of algorithms...

The Red Hat QA folks went out for dinner on the first day at one of Kamil's signature 'it's just a five minute walk, guys' restaurants, which turned out to have a very nice location and outdoor seating when we arrived after a brutal five day trek (I exaggerate only slightly) but also to have the typical Middle European slightly fuzzy notion of 'vegetarianism' - it's all very well, but surely a bit of chicken can't hurt, right?

Note before jumping into the event proper - you can find the recorded versions of all the presentations (though not hackfests) on Youtube here, many thanks to the folks who set up all that recording.

I saw the day one keynote on F/OSS in the EU (more or less) which had some interesting notes on fairly significant deployments besides the ones that have really made the tech press, then watched the QA overview talk which was kindly and expertly presented by Amita Sharma, who's been contributing some energy to QA lately. I saw Matthias Clasen give an update on the status of Wayland. I've actually been checking up on that for myself lately - there's been some pretty significant fixes landing in Fedora 21 / Rawhide in the last month or two, and it's now at the point where I can run this laptop on Wayland and just about everything I use works fine except for VAAPI video playback acceleration and suspend (or at least, suspend on lid close - I need to look into that one some more). Matthias gave a quick no-nonsense 'here's where we are and here's where we're going' infodump talk, which was probably the best line to take for the audience.

After lunch I was sitting in Christian Schaller's talk on Workstation, but I was actually mostly working on stuff by that point - I think I was still busy cleaning up a bunch of stale wiki bits I'd noticed from screenshots in Amita's talk, ancient test plans and the like. You can see the stuff I cleaned up on August 6 in my contribution log (at least until it gets scrolled out). After that I caught Luke Macken's "Evolving the Fedora updates process" talk - nice job of hiding the now-traditional "Bodhi 2.0" talk, Luke ;) Seriously, Luke mostly outlined the current work on Bodhi 2.0, which looks like it's really truly actually happening real soon now, and is full of awesome. Really can't wait for that to arrive. Some of the hook-ups with Taskotron and other bits that are planned look like they'll be really great.

Next I sat in the usability methods talk, but again I was mostly working on something else - that may have been around the time I started work on getting a test MozTrap instance running, on which more later. Afterwards I believe I was in Dennis Gilmore's release engineering talk, and I think I was splitting my attention again. Finally I sat in on the Badges talk, which mostly served to remind me what an awesome job Marie Nordin has been doing on Badges this year, both as part of the Outreach Program for Women run by GNOME and later. It was great to meet Marie and nice to have the chance to thank her for all the work - Badges is still one of my favourite Fedora things, though it's lost out to Fedlet and ownCloud in the passion project stakes lately, and without her we probably wouldn't have anywhere near as many badges out there!

The evening's festivities involved a pub which gamified the consumption of alcohol - each table has some beer pumps and tracks the amount consumed by each seat and the table as a whole. The table consumptions are plotted up in nice big bar charts and projected on the wall. This concept I think rather terrified those of us who've been going to Flocks/FUDCons for a while (and particularly the folks who've been around long enough to have been at the still-legendary early Boston ones, about which...that was probably enough said already), but obviously the crowd has gotten mellower and/or saner over the years, as I don't think anyone was carted off to the hospital or attracted the attentions of local law enforcement. Fedora QA formed a pretty strong table, but we were narrowly beaten out by a table with a strong contingent of Anaconda developers and release engineers. I don't think this result surprised anyone. It was also where I snapped the now-legendary F/OSS axis of evil photo - I think the Workstation table (with special guest star Lennart) was too busy plotting world domination to challenge for the drinking title... Before the pub, Tim Flink and I decided to eat separately to forestall any more issues over the precise definition of 'vegetarian'. The pasta place I'd seen turned out to be closed, and so was everywhere else on that street (presumably it's for the lunch crowd), but somehow we turned a corner and stumbled across an Asian vegetarian restaurant, which has to be some pretty long odds. Had some very nice fried rice and eggplant and other bits and pieces there, at highly reasonable prices.

On the second morning I saw the neatest 'surprise' talk of the week - the Review Server talk, proposing a new way to do package review, which sounds about fifteen thousand times better than the old way and which I hadn't heard of yet. It may have been floated on devel@, I'm still like 1500 mails behind on that list right now, but it was a new and pleasant surprise to me anyway, and I really hope it gets done. I then gave my UEFI talk (in a sleeveless t-shirt, striking just the right professional attitude - in my defense, it was frickin' hot, and I was running out of clean laundry). It went fine and I got some good feedback, but I forgot to update the talk after I last gave it at LFNW and so I made the same mistakes over again, not fully explaining fallback path and the EFI system partition and not talking about Secure Boot at the right time. I've now updated the slide deck, so if I give it again, it should run better. I watched the keynote from the awesome Sean Cross about the Novena open laptop he's working on. Open hardware is really, really hard, but it looks like they're going to be one of the more successful projects out there. It was really nice to see just a few hours later that John Dulaney, one of QA's awesome volunteers (who also contributes to many other areas of Fedora) had helped Sean get Fedora up and running on the demo hardware, a challenge Sean had issued during the talk!

Then I dropped in on the unit testing talk for a while, but it was heavily coder-focused so I ducked out and worked on something else for a bit. Following that, we put together an impromptu QA hackfest. Amita had suggested doing a triage hackfest during her talk, and we liked the idea, so we went ahead and did it. Unfortunately due to the usual conference communication fubars, Amita didn't make it, but most of the other QA folks and few people who couldn't escape before we nailed them to their chairs (including the awesome Jon Stanley, making a triumphant triage comeback) got together with Richard Hughes and worked to do some triage of the GNOME Software bug list. It's an illustration of the surprisingly time-consuming nature of triage that a roomful of QA folks and the package maintainer couldn't blow through the whole bug list in the 90 minutes we gave it, but there ya go - triage is hard. We did make a dent! For the last half hour or so of the hackfest, I gave a very quick overview of my MozTrap demo install. People were sufficiently interested that we planned to have a more detailed look at it later in the conference.

The revelry that evening was aboard a boat, again an entirely sensible and safe idea with no possible drawbacks whatsoever, but unaccountably no-one wound up being pulled from the Vltava and a good time was had by all (well, at least, no-one complained to me). Prague is an amazing city, in case you didn't know, and from a riverboat is a great way to see it - probably worth a couple of hours if you're ever there.

On Day 3 I unfortunately missed Arun Sag's Docker talk - the title was a bit generic so I sort of figured it was the 'hey, look! Docker!' talk, the one you can find six people capable of giving by just hurling a brick at any F/OSS conference these days, but apparently it was much cooler than that. I still need to check the video. I think I then stopped in on the systemd daemon integration talk, but it seemed a bit more basic than I was expecting and mostly going over stuff I knew, so I worked on MozTrap stuff for a bit (still more coming on that, be patient!). I dropped by the Fedora.next joint session for a bit - it seemed to be mostly a retrospective on how the Fedora.next changes had been handled so far and where we could improve, which was a pretty good idea, and I think we moved things forward a bit. I got some good feedback on how we can maybe get more input from other groups into QA planning, which should help out with doing the validation testing process updates for Beta and Final.

After lunch I caught Arun's second talk, on rapid deployment to bare metal systems, which was really great. It's an interesting area - there are lots of things out there to help you do rapid deployments to VMs, like virt-install and so on, but not so much for bare metal. The approach Arun's been taking at Yahoo is a neat one, extending anaconda's existing support for deploying a disk image 'payload' (which is used for live installations) to support installing a tarball containing such an image, and he did a nice live demo. There were a bunch of anaconda devs and other heavy hitters in the room, and I think we came up with a good half dozen suggestions for improvements to anaconda that would help out this workflow (and incidentally make things nicer for other users too). Should be some really good work there. I also abused the talk to nail one of the anaconda devs to his chair - Vratislav Podzimek was my unfortunate victim - and trace out an install crasher we were hitting during Flock, which eventually turned out to probably be effectively a cunningly-disguised dupe of the bad live image compose bug we knew about already. We were able to come up with a couple of improvements to anaconda's error checking though, which should make it much more apparent when something like this is happening in future.

I managed an hour in the Documentation hackfest, where Jared Smith very kindly gave me the crash course on Docbook, and I got about halfway through turning my post on NetworkManager bridging into a proper guide - I hope to find time to finish that up. After that we ran a second impromptu QA hackfest, this time spending the whole time on MozTrap. Well, first we went for dinner because we were really hungry, THEN we did the hackfest.

We got most of the QA folks together, put up a demo instance on a laptop so we were all on the same subnet, and then just played with it for a bit, setting up a few dummy products and test suites and environments and playing around with the reporting and querying interfaces.

MozTrap is a TCMS - basically it keeps track of test cases and their results. Fedora essentially uses Mediawiki as a TCMS; that's what we're doing when we write test cases as wiki pages based on a template. Things like the release validation matrices are what a TCMS might refer to as a 'test suite' or 'test plan' or 'test run' or similar, and also a (very bad) result storage system. We 'query' our 'TCMS' by, well, looking at matrix pages, using wiki categories, and writing cute nonsense hacks. Up to Fedora 20 we were able more or less to keep up with what we needed to do, but the seams have really started showing with Fedora 21; the amount of inventive judo I have to do with wiki table layouts to encode the concept of 'run this test case on these images in these configurations' is getting unmanageable. Reporting results has always been kind of a sucky experience in the current system - basically you have to edit a page, insert a bit of text like {{result|fail|adamwill|123456}} which is kind of gibberish, hope you got it right and in the right place, then save the page and hope you didn't just blow up the wiki, which is hardly a welcoming experience. 'Querying' results has always been a bit of a pain point - you can usually find the absolutely essential stuff like "what bugs did we find at this Test Day?" or "what release validation tests failed on the last run?" if you knew what you were doing, but it's always a bit disjointed, and you just can't really do stuff like "what results have we had for this test case, regardless of the context, for the last three months?" Finally, we've started running into things we'd like to do that we just can't figure out an entirely satisfactory way to do with the current system. For me, at least, the 'let's get a real TCMS at last' idea seems to be becoming rather more compelling as of late.

We've evaluated real TCMSes in the past; the obvious one has long been Nitrate, the F/OSS release of RH's in-house TCMS. But for various reasons, we've always found Nitrate just doesn't line up very well for Fedora; it makes quite a few assumptions that just aren't quite in line with how we do testing. We found a similar story with the other TCMSes we've looked at in the past.

MozTrap is a fairly new TCMS written by Mozilla, and used in production by them since 2012 - they use it for pretty much all their testing, I believe. It's fully open-source of course, and seems to have a decent bunch of developers behind it who are open to outside suggestions and contributions. It's a Django webapp, which is something we're reasonably familiar with and isn't, you know, Java or raving insanity (wait, aren't those the same thing?) And most importantly, its basic design lines up rather more closely with how we do Fedora testing than any other TCMS I've come across. You write 'test cases' and organize them into 'test suites', then associate them with 'builds' of 'versions' of 'products' and create 'test runs' (which are more or less 'this is a single instance of this test suite executed against this build of this version of this product'). People actually running tests see the 'test runs', pick one, and get a nice interface where they see all the test cases and can easily provide their results, without running the risk of blowing up the system somehow. And there's one other concept which I really like - 'environments' act as a sort of multiplier on the 'test runs'. An environment can be, well, anything, but basically it's that confounding factor of 'well, running this test against ARM is a different result from running it against x86_64, and we need to do both and track them' - in that context the architecture is the 'environment', it can also easily be the product 'flavour' (e.g. Server or Workstation for Fedora.next), or even a version (say the product you're testing is ownCloud, the distribution version may function as an 'environment', depending on exactly how you want to organize your testing). Environment elements are organized into 'environment groups', so you can have multiple 'environment' factors apply to a given test run, and you can track and query all the possible combinations of all the environment elements in the results. The whole setup is very flexible and configurable - you can have all kinds of different products and versions and test suites and environments, and hence track very different forms of testing all in one TCMS.

So for the hackfest, as I said, we just played around with a test deployment for a while, looking for pain points. We wrote down our notes as we went along in a Piratepad. I think in general we all liked the basic design and layout of the system, but we did identify various things that worried or us or might need work. The design lends itself to massive multiplication of test runs and test cases, and the query interface gets a bit 'idiosyncratic' when that happens; if you look at Mozilla's production deployment, they have literally thousands of test runs listed in a five-item-high scrollbox in their query interface. I think there are ways to use the 'smart' query search box to make querying manageable with a huge corpus of test cases and test runs and results, but it's not as nice and obvious an experience it is when you just have a small, manageble amount of test data. We also noted that for production we'd need to jump the usual packaging hurdles - I'm currently looking at exactly how much of a job this would be, how bad the dependencies are (it's always the dependencies, for webapps) - integrate the authentication with FAS somehow, manage the import and fix up of our existing test cases, and a bunch of other little things like checking that the API is sufficient to our Grand Plans for integration with the blockerbugs webapp and so on.

We're currently kicking the idea around a bit more on the mailing list, and the next step will probably be a slightly more permanent staging deployment in Fedora OpenStack or similar - still not suitable for production use, but persistent enough that we could do a bit more of a convincing build-out of test cases within it, maybe duplicate some results from the mediawiki tables, and see how it goes in that kind of use. It'd also be useful for us to do any development we want to submit to upstream against.

There wasn't an official party on that evening, but I think that was the one where the QA team agreed to meet up at a Czech pub down the street from the hotel only to find that the anaconda team was already there, as was Kanarip, and we wound up forming one big happy party, taking over half the tables in the pub in dominoes fashion, and probably driving the rest of the patrons crazy. But it was a fun night.

On the final day I attended Michal Toman's ABRT talk, which was another very useful session - it was another 'past/present/future' infodump, but that was just what I needed, particularly the bits about their immediate future plans, they have some nice stuff in the pipeline. I was able to pitch the importance of improving the UIs for the bits of the system people get to interact with, as well - there's such a mess of workflows and use cases in the gnome-abrt tool and the web UI that it's hard or impossible to really 'fix' it by filing one bug at a time, it really needs the more fundamental work of a proper UX designer sitting down with some users and figuring out the various use cases and workflows, and coming up with a fundamentally sane design.

I then went to Peter Robinson's talk on Fedora's ARM support status, which was one of the most uplifting of Flock because the news was basically all good - our support as of now is pretty damn good and about to get way more so. Fedora 21 should support a huge range of ARM hardware, and especially the new aarch64 platforms that'll really be coming on stream around the time it comes out. Fedora's still primarily focused on dev boards and the new crop of server hardware, so we're still not really talking Fedora-on-cellphones here, but it's awesome news for people playing with Beagleboards and Cubietrucks and Utilites and so forth and so on, and particularly for people interested in the brand-new 64-bit ARM server platforms. Definitely a highly recommended talk for video catch-up (and I don't know how long the cameras ran, but there was some fun stuff at the end where several people who were clearly under NDAs were trying to figure out how much they could say about exciting new shiny hardware without getting in trobule). After that I hit up Cole Robinson's talk on virtualization, which was also very useful because it made me realize I've been a giant idiot for years by not using the offline snapshotting feature of Fedora's virt stack, which is super easy to use and built right into virt-manager since (IIRC) Fedora 20. I think when I looked at 'snapshotting' I wound up looking at online snapshots, which are apparently a completely different and much more finicky creature, and didn't realize this whole easy-as-falling-off-a-log alternative mechanism existed which is just ideal for QA purposes. So that'll help me out a whole heck of a lot in future. The talk also has lots of other interesting tips for improving your use of virtualization, so check it out!

In the afternoon I attended the governance hackfest, which produced about as much controversy and hot air as you'd expect, but also produced at least the skeleton of a possibly-viable plan for revising the makeup of the Fedora Board, which could be interesting. I was the principal note-taker for the session, and you can still find the Piratepad notes at least for now (I don't know how long PP keeps them). I've emailed them off to Haïkel, who ran the session, and I believe the idea now is that the plan gets fleshed out a bit and sent off to the lists for more comprehensive public review.

After that hackfest I sat in on the spins session, which I think was pretty productive, and developed a general consensus to try and focus in harder on higher quality spins as a part of Fedora 21 and later, and move on from the very open and inclusive vision that Spins initially had but never quite achieved - it's mostly become more of a delivery mechanism particularly for alternative desktops, and similar things.

Again there was nothing official for the evening, so I walked around town for a bit with Amita (who had never visited Prague before, so hadn't seen the sights) and then spent the evening doing justice to a bottle of Glenmorangie with Peter, which was thoroughly enjoyable!

Once again a really fun and productive Flock, and as always I have about two dozen possible avenues for doing interesting stuff which I'll never get time to explore fully (the handwritten notes on possible next things to do on Badges that I wrote after the last Flock are still lying all over my desk, where I'm going to get around to them real soon now, honest). But that's a good thing, not a bad one - I'll start getting really worried when I don't see anywhere interesting to go! Since Flock I've been taking a few vacation days, travelling across Europe the slow way and spending a night in London before pitching up back here at my family's home for a few days before I return home and get back to work. I'm hoping by then it'll finally be full speed ahead on Fedora 21 Alpha testing! Thanks to all the folks I saw at Flock, both old and new, for making it such a fun, productive and informative event as always - I love this community.