Fedora 21 Alpha release, common bugs, relval, and wiki work

Source

I've had quite the packed week, somehow. I started off on Monday with the regular QA meeting, then wrote up the initial Common Bugs page for Fedora 21 - many thanks to bitlord for adding a note on the ARM issue I didn't really understand!

As part of that, I did the usual thing where I copied the page header from an early version of the previous page, but it kind of irritated me - I felt like there should be a better way to it. As it happens, there is: I was reminded of the existence of substitution, a use of Mediawiki templates different from the transclusion typically used on the Fedora wiki - where you enter {{somethingorother}} and the contents are 'dynamically' replaced from another page each time your template is rendered.

With substitution, the replacement is done just once, at the time the page is first created. This is handy for something like the Common Bugs. We want to use the FedoraVersion and FedoraVersionNumber templates, but we don't want the page for Fedora 21 to suddenly start referring to Fedora 22 the next time the 'current' release changes. So substitution's just the thing: when you create the page from the template the correct value will be included, and it won't change afterwards. Now there are templates designed to be substituted to create the pre-release and stable release headers for Common Bugs pages via substitution.

Once I'd been reminded of the system, I had the idea of using it to improve the process of creating the release validation results pages. You know the ones - pages like Fedora_21_Alpha_RC1_Install, which we use for recording the results of validation testing. I've been looking into using Mozilla's Moztrap to replace our Wiki test 'system', but it's not turning out to be such a straightforward swap, and in the meantime I wound up getting stuck into this.

Creating the pages the way we've done it up till now is a bit of a pain. We had pages called 'templates' which weren't actually Mediawiki templates, but just sort of base pages that you would copy and paste, then go through by hand, somewhat painstakingly changing version numbers and updating link locations and so on. Now we have five 'mandatory' pages and one optional one, this was getting to take a lot of time.

So instead, I decided to look into creating the pages via templates. It got a bit tricky, because we also want to use transclusion to share some content dynamically between the different results pages. We want the test instructions to be transcluded so we can improve them at any time, but we want the result tables to be substituted, of course, because it'd hardly work to share their content between results pages!

But it's not impossible! The system is quite capable, and has some neat tricks. The design I wound up with uses:

The main template page itself is designed to be substituted to create the actual result pages - it handles substitution of the release identification information (release, milestone, compose).

Hold on, though! If you're paying attention, you're wondering how one template can substitute another - wouldn't the content just wind up in the main template at the time it's created?

To avoid this we use a trick that takes advantage of the <includeonly> tag, indicating content should only be displayed when the page is substituted or transcluded, not when it is displayed directly. If you wrap a use of {{subst:}} - a substitution - on a template page in <includeonly> tags, then each time that template itself is substituted, the substitution within the template will be rendered and hence will take effect. The template itself, though, remains unaltered. Cunning!

The upshot of all this jiggery-pokery is that if you create a Wiki page and pass this magic incantation as its only content:

{{subst:Validation_results|testtype=Base|release=21|milestone=Alpha|compose=RC1}}

your page will become a fully-formed Fedora 21 Alpha RC1 Base release validation results page. If you check it in the editor after creating it, you'll see the contents are very different from what you typed.

I was pretty happy with that trick. In my announcement email I rather recklessly offered a bit of a hostage to fortune:

"I have vague plans to write a little script using one or other of the zillion python mediawiki interfaces out there that would make creating the pages and handling the categories for a new TC/RC build a one-line operation."

So on Tuesday I had to go ahead and make good on that promise. And rather to my surprise, I seem at some point to have osmosed the ability to write some sort of passable imitation of object-oriented Python. I'm really not at all sure how that happened. Anyhow, I hereby present Relval.

Relval spits out release validation pages. Basically it's a command line tool for generating those magic {{subst:Validation_results}} lines, in appropriately named Wiki pages. It has to do other little things, like discover the available test types, validate user input, and solve captchas (not as impressive as it sounds, the wiki uses an extremely weak captcha system) and it's slightly over-engineered for its task at present, but that's because I plan to make it do more stuff, like do Rawhide monthly result pages, handle the categories and maybe generate announcement emails. It uses the mwclient Mediawiki API library, which seems to work fine - it's a bit under-documented, but pretty easy to follow from the source.

It's already pretty capable at generating TC/RC validation pages. If you want to try it out, pass the --test parameter which makes it work against the staging wiki rather than the production one. So when we hit the Beta test composes next week (next week? oh God), we can fire off:

./relval compose --release 21 --milestone Beta --compose TC1 --username adamwill

and all the release validation pages will appear. I think I'll be on the winning side of that ol' automation graph somewhere in the middle of F22, if we haven't moved on to a different TCMS by then...

I should gratefully acknowledge Mike Ruckman's advice and suggestions on relval, particularly on the captcha issue - all errors are mine, of course.

I sent relval out into the world on Tuesday, and was all set to do some cleaning up and extending of it on Wednesday, but after the release blocker meeting in the morning, I was making the freeze exception and blocker process pages share more content using templates (TEMPLATE ALL THE THINGS) and actually doing a modicum of testing Fedora (it's a crazy idea, I know, but it might just work) when I fell into a fateful conversation with David Shea and Samantha Bueno in #anaconda:

<davidshea> adamw: re: blocker review, if you want us to remember what the
commit/freeze/no-freeze policy is at any given time, I dunno maybe write it down?
<adamw> davidshea: it is written down...
<davidshea> where we can find it?
<davidshea> like, on a schedule?
<adamw> https://fedorapeople.org/groups/schedule/f-21/f-21-devel-tasks.html
<adamw> the 'change deadline' dates
<adamw> are the milestone freezes
<adamw> https://fedoraproject.org/wiki/Updates_Policy#Branched_release has some other bits
<davidshea> ok so I just need this big chart that's kind of giving me a
headache, plus the special fedora word translation ring
<sbueno> see, when i read "change deadline" i read that as "accepted changes"
 e.g. things for which a feature page has been written

and then...I don't know, I might have gone a little crazy. It all started sensibly enough, with a modest plan to maybe clarify a few pages, and a proposal to rename the 'change deadlines' and the 'branch freeze'.

Then I started poking a few 'what links here' pages, and one thing led to another, and it's now 36 hours later and I seem to have rewritten half the Wiki.

So, allow me to present:

  1. The Milestone freezes: Alpha freeze, Beta freeze, Final freeze (yes, people with long memories, everything old is new again)
  2. The revived and updated Fedora Release Life Cycle
  3. The completely new Repositories page - yes, now you can actually point somewhere to explain Fedora repositories.
  4. The extensively overhauled package maintainer update HOWTO
  5. The melded-from-three-sources-and-heavily-rewritten Package maintenance guide
  6. The overhauled Release Engineering Overview
  7. The carefully tweaked Updates Policy (no changes to the actual Policy, just updates to the links and some adjustment of concepts like the milestone freezes, Change freezes, and the 'Bodhi enabling point')
  8. The somewhat revised Branched page (I was running out of steam by that point...it should probably be redone with templates to share a lot of content with Rawhide, but I don't have the heart.)

There was a whole bunch of detail work that went along with the above - I kind of subsumed the concept of the "branch freeze" policy into the Updates Policy (and the new concept of the "Bodhi enabling point"), and clarified the relationship between it, the milestone freezes, and the Change freezes. I did small pokes to various pages as I went around, and updated just a metric assload of outdated links and redirects and things.

There's some more detail in my announcement mail, and you can look at my contributions page for the full list. I think I need to go and watch the Ryder Cup with a nice glass of Scotch around about now...

Comments

No comments.