Yak shaving, again: ownCloud 7 for Fedora 20, and TLS/SSL certificate generation in Fedora and Red Hat

Introduction

I seem to have been setting yak shaving world records all over the place this week. This last one has been particularly fun, though.

I've been working on and off all week on my ownCloud testing deployment kickstarts. These are kickstarts that can be used to deploy completely configured and working (though highly insecure) ownCloud instances with no interaction needed: you can install Fedora 20, 21 or 22 with inst.ks=https://www.happyassassin.net/ks/oc/oc7-httpd-mysql.ks, for instance, and out pops a completely configured ownCloud 7 instance running on Apache with MariaDB as the database. You browse to http://hostname/owncloud and up pops the ownCloud interface (even the OC first time wizard is automated).

I'm kinda proud of these anyway as it's a neat thing, but the actual point was to make it easy for me to run ownCloud smoke tests. Now I actually have the kickstarts for all three databases our packages support (SQLite, MariaDB / MySQL, PostgreSQL) working, I was able to build ownCloud 7.0.2 packages and test both clean deployment and upgrade from 6.x for all three databases in about half an hour. That made me confident enough to submit 7.0.2 as an update for Fedora 20 as well as sending to Fedora 21 and Rawhide. Please do test it out and submit feedback! Take backups first, of course.

That's where the yak shave of the last day and a half kicked off. I thought it'd be nice to ask upstream to promote or at least mention the native Fedora packages instead of or alongside their own OBS packages in the upstream documentation. Of course, that works best if there's a nice attractive page they can link to. So I went to see if we had a Fedora wiki page for ownCloud, and lo and behold, there it is!

It was a bit outdated, though, and missing some useful information I figured it should have, so I sat down to rewrite it. (I've done that now - it should be all nice and updated and comprehensive, do let me know if you see any problems). Now, wouldn't you know it, part of the page dealt with configuring TLS/SSL for Apache. "Huh," I thought, "surely we have a generic page explaining how to do that?"

Guess what? Here's the Fedora wiki Apache page as it existed before I started poking it. I bet you didn't know about that, huh? I actually at first assumed that it was an ancient thing no-one had touched for years and so I hacked into it with wild abandon, but now I see it was only written in May of this year - I hope I didn't hurt any feelings by editing it.

So I spent some time improving (I hope) and extending the Apache page. When it came to the TLS/SSL stuff, somehow - I'm not exactly sure how - I got the idea that the genkey command the previous version of the page recommended using was boring old stuff, and /etc/pki/tls/Makefile was shiny and new. I think it's because the page recommended using openssl directly if you needed to bump the certificate serial number, and the Makefile has a parameter for that, so I just figured, hey, the Makefile must be better, and switched to it.

So last night I finished tweaking the ownCloud and Apache wiki pages, and also the PostgreSQL one, and was feeling pretty pleased with myself. Then this morning I woke up to an innocuous little discussion between Stephen Gallagher and Dan Mossor about the TLS/SSL certificate generation stuff, in relation to the Apache web page (I'd asked a few people to check my changes to it). Dan found make testcert hadn't worked for him (though I think it was just that he had a pre-existing certificate), and there was some discussion of other methods. This caused me to wonder exactly what the deal was with all these different SSL/TLS certificate generation thingies we have anyway, and, well, now it's 4am the next morning.

Fedora / Red Hat TLS/SSL certificate generation tools

First I found out that both /etc/pki/tls/Makefile - shipped as part of the openssl package - and /usr/bin/genkey - shipped as part of the crypto-utils package - are damn ancient, dating back to at least March 2000 (when it was moved into openssl) and November 2000 respectively. /etc/pki/tls/make-dummy-cert, also part of the openssl package, is nearly as venerable, dating back to June 2001. It's not clear from the changelog why it was invented at all.

So we've had (at least, I'm sure there are more) three different Red Hat/Fedora-ish tools for producing TLS/SSL certificates in RHL/Fedora for, oh, a little over 13 years now. Yay duplication of effort!

make-dummy-cert

make-dummy-cert is the simplest: it takes a filename as an argument, and (using openssl) writes a self-signed certificate and the associated private key together into a single file named whatever you told it to call it. That's all it does. In Fedora, since June 2009 (openssl-0.9.8k-6) it produces a 2048-bit key; prior to that it produced a 1024-bit key. The hash algorithm is mostly irrelevant for self-signed certificates, but it uses openssl's default, which was changed from MD5 to SHA-1 in Fedora in 2005 (and is changed to SHA-256 in Fedora 21). Its companion command renew-dummy-cert simply renews the certificate, because hey, it'll expire some time and you'll want to do that. make-dummy-cert doesn't tell OpenSSL what to do about the certificate's serial number; since 0.9.8 in 2004, if not instructed otherwise, OpenSSL has used a large random number. Prior to that, it used 0, so all certificates generated with make-dummy-cert would have had 0 as their serial number.

/etc/pki/tls/Makefile

The Makefile is slightly more complex/powerful. It can be called in different ways to write a private key only, a key and a CSR, or a key and a self-signed certificate - the latter either as separate files or as a single combined file, the way make-dummy-cert does it. You can specify the filenames. Apart from the 'single combined file' case, the key will be saved to /etc/pki/tls/private and the certificate or CSR (if any) to /etc/pki/tls/certs. This is the layout commonly used with the mod_ssl Apache extension, which is powered by OpenSSL (mod_ssl can also use the 'single combined file' format output by make-dummy-cert).

By default it always uses a serial number of 0, which can cause issues if you generate a new certificate for the same site: you can pass it SERIAL=1 (or any other integer) to change the serial number. As with make-dummy-cert, its default key length was changed to 2048 in June 2009 and it uses openssl's default hash function, whatever it is in the version of openssl you have installed when you run it.

/usr/bin/genkey

genkey is more powerful again than the Makefile, and rather different to the other two. It's not a simple non-interactive script, but an interactive (Newt-based) console application - basically a step-through wizard. It has a few different modes you can call with parameters. It can generate a self-signed server certificate, a self-signed CA certificate, or a CSR (but, ironically given its name, not a key alone). And it can generate these either in mod_ssl style - as described above - or in NSS style. NSS stores keys and certificates in separate flat-file databases. When using the mod_nss Apache extension (which is powered, obviously, by NSS) these are typically named /etc/httpd/alias/keyX.db and /etc/httpd/alias/certY.db, where X and Y are numbers (in current Fedora releases, it's usually key3.db and cert8.db, and these are the filenames genkey will use unless instructed otherwise).

Also unlike the other two tools (which use openssl), genkey uses NSS for generating keys and certificates (even when generating 'OpenSSL-compatible' certificates). A program called keyutil is included alongside genkey in the crypto-utils package. It provides (on top of NSS) some functions genkey needs to produce OpenSSL-style certificates which are not available in NSS itself.

genkey explicitly sets the key length (which you choose interactively) on both NSS and OpenSSL paths. On the OpenSSL path, keyutil (not genkey itself) explicitly specifies the hashing function to use for certificate signing, it does not rely on NSS' default. On the NSS path, the default NSS hashing function is used (though one of the patches I've worked on today might change that). Since January 2014 the default key length (the value pre-selected when you enter the screen) in genkey has been 2048-bit, and the hashing function used when signing certificates and CSRs on the OpenSSL path has been SHA-1. Prior to that, the default key length was 1024-bit and the OpenSSL path hashing function was MD5.

Things I did, and thoughts on future improvements

Early as I was figuring all this out, I came across a bug report requesting genkey use SHA-2 hashing functions instead of SHA-1. I'm now reasonably sure it's not that big a deal, but that's only after working through a lot of this stuff, and writing a patch to fix that problem.

I also noticed an odd place in NSS where an MD5 hashing function is specified but never used - I spent quite a long time making sure it really isn't used, and eventually found that someone else had been there before, and confirmed their bug report.

I confirmed a bug report from Kai Engert, who ran into some of this stuff recently too, that the parameter that lets you specify a hashing function for the NSS certutil command is undocumented, which is sure a hurdle when you're trying to patch this kind of thing.

I submitted another documentation bug report to OpenSSL - hasn't shown up anywhere linkable yet - to report that the manpage for the 'req' command claims the default serial number for new self-signed certificates is 0, when it has not been for ten years.

I wrote this damn blog post. ;) I hope it's useful to someone.

It seems silly to me to have two fairly simple scripts for doing certificate generation (in the same package, no less). Makefile is a superset of make-dummy-cert; I think it would probably make sense to turn /etc/pki/tls/make-dummy-cert into a tiny thing that simply calls the Makefile with the correct parameters. Or, you know, we could put a binary script or wrapper around the Makefile in an actual binary directory, I know it's radical, just a thought. We ought to be able to get down to one non-interactive script-y thing and one interactive app, at least.

It does rather amaze me that there's no simple light tool for doing this sort of thing shared across distributions.

I should ask Stephen exactly what he meant about his new method for self-signed certificate generation that reduces the possibility of MITM attacks using stolen self-signed certs.

I now actually more or less understand how x509 public key cryptography works (that is, the 'who signs what and encrypts what and why' of it, I'm not claiming to understand the damn maths). And I know for damn sure I didn't know that this morning. Both of those things feel like worthwhile upgrades.

Tomorrow I'll edit the Apache wiki page back to recommending the use of genkey, and now I'm thoroughly prepared to explain it comprehensively and correctly :) And that should bring this whole shave to an end...