Further sysadmin adventures: where's my FreeIPA badge?

So, hey, I have a working FreeIPA deployment for happyassassin.net. After I put up my last post about how to do unified authentication for the various services I run through happyassassin.net, I got like five tweets and a couple of comments telling me to use FreeIPA. It's the obvious choice!, they said. It's easy!, they said. It only takes half an hour!, some jackass said. To these people, I blow a gigantic raspberry. But first, I figured I was smarter than them anyway, and didn't need it. The services I'm mostly interested in having unified authentication for are mail - which for me means dovecot and postfix - and owncloud. I already had postfix configured to auth via dovecot's SASL (which I'd totally forgotten about), so they were already in lockstep, and owncloud has an IMAP authentication option. So it's easy as pie, right? Turn on IMAP authentication in owncloud and proceed directly to Go. One problem: owncloud's IMAP authentication does not freaking well work. You will not convince me otherwise. I've read every goddamn Google result for 'owncloud imap', carefully negotiated all the gotchas documented therein, and it still doesn't goddamn work. Nothing logs anything useful anywhere. All I have to go on unless I feel like breaking out wireshark (here's a clue: I DON'T EVER FEEL LIKE BREAKING OUT WIRESHARK) is a 500 error from the server, and a 500 error from an HTTP server is roughly equivalent to "because I don't FEEL like it, that's why." So. That was a bit of a dead end. Having cursed for a bit and set fire to some things, I decided, screw it, I can set up FreeIPA. It's a bit like using a large atomic bomb to crack a nut, but it's all shiny and I can probably convince people that it is somehow work-related and it'll, I don't know, save me about ten seconds of sshing into boxes and running 'passwd' when I want to change my passwords, or something. And hey, it's only going to take me half an hour anyway, right? So, cue that raspberry. I don't know how half hours work in CERTAIN PEOPLE's worlds, but in mine, it takes me half an hour to get through the damn introduction to the FreeIPA installation guide, never mind the rest. I've been up past 4am every day this week so far fiddling with it. But hey, now I actually have it working, and it is pretty shiny. I don't even remember all the hurdles I leapt on the way any more, but here are some of them. One, DNS. You kind of want to let FreeIPA do your DNS for you, you see. If you're going to have a 'real' deployment of it on a domain with hundreds or thousands of machines and services, which will change quite a lot, you really want to let FreeIPA do your DNS for you. But I don't want to let FreeIPA do my DNS for me, because of REASONS. Well, okay, because it's just not a very good fit for my setup, right? I'm running a small home network with a bunch of systems which won't ever be part of the FreeIPA domain, and there are issues with how I have my VPN access set up, and it just...didn't look like it was going to work out very well at all to have my network's DNS run out of the FreeIPA server. It makes a lot more sense to run it out of my router, which is what I'm doing. You can set up a subdomain with the hosts that are actually in the FreeIPA domain and just let FreeIPA manage the DNS for that, but ehhh, that would've been even more finicky in some ways. And the practical drawbacks to running DNS outside of FreeIPA once you get it working don't really apply to me: I don't need my boxes to be able to change their hostnames and push that info to the DNS server, and I don't need to keep moving services around between boxes and have the SRV records updated automatically when that happens, my network just isn't that big; I don't use SRV auto-discovery at all. So. External DNS it was going to be. Setting up FreeIPA to work with external DNS is kind of a pain in the ass, though. Well, if you know a lot about FreeIPA and a lot about DNS I'm sure it's a walk in the freaking park, but for monkeys, no. That one took me most of one evening, including all the time I spent learning what the hell an SRV record is. To save you the trouble, an SRV record is pretty much just a DNS record which says 'hey, on happyassassin.net, the LDAP server lives HERE'. Or something like that. You specify a service - which is something like LDAP, or Kerberos, or IMAP, or something like that - and list the name of the server that provides that service, the port and protocol, and optionally a couple of numeric parameters that are used for prioritizing multiple servers that provide the same service on big, grown-up domains (used for load balancing and fallback). In 'stuff I'm kinda glad I learned as a side benefit of this whole project', this is actually how mail auto-discovery works (well, it's the way it ought to work, I think Microsoft has their own ridiculous way of doing it too or something): that's RFC 6186 (on which the Evolution maintainer, Matthew Barnes, has some interesting thoughts, but I digress). Anyhoo. If you're going to run FreeIPA with an external DNS server, you basically need to stick a whole whack of these SRV records in your DNS configuration. If you let FreeIPA handle DNS for you it does all this for you during server deployment, of course. When you run the server deployment script, FreeIPA will spit out an example configuration file in bind format that you can use. Unfortunately, the DNS server on my router's firmware is DNSmasq, so I had to figure out how to transpose the bind config lines to DNSmasq format, which took an hour or so before I grokked it. DNSMasq's man page and example config are invaluable here. And the format is: srv-host=_kerberos._udp,id.happyassassin.net,88,0,100 srv-host= is the directive for dnsmasq that says 'add this arbitrary SRV record', _kerberos is the name of the service (always has a _ before it in SRV-speak for some reason), _udp is the protocol (ditto), id.happyassassin.net is the server, 88 is the port, and 0 and 100 are those numerical values I mentioned earlier, not really important unless you have multiple servers. This record also illustrates one of the bear traps I fell into during this process: note the FQDN, id.happyassassin.net. According to the docs I read on the SRV format, just the hostname 'id' is perfectly valid so long as your DNS server will further resolve that to an IP, which mine does. If I set that bit of the line to be just 'id' and query the server with dig, the response I get is sensible. But for FreeIPA purposes, well, it's gotta be an FQDN. That simple. You want FreeIPA to work, make damn sure the server names in your SRV records use FQDNs. Public service announcement right there, folks. edit: The FreeIPA devs would like me to point out that this is really a Kerberos requirement, not a FreeIPA requirement. Fine, folks, consider it pointed out. I wasn't trying to suggest that FreeIPA was being ridiculous here, just giving appropriate emphasis to the topic. To the monkey who never set up either before, the point is moot: Kerberos is just one of those scary bits that I'm counting on FreeIPA to deal with for me, and save me the trouble of knowing how the heck it works. On that topic: actually, if you want FreeIPA to work, just generally make sure everything that can possibly be an FQDN is an FQDN. As part of this project I have learned that hostnames are far more psychopathically complicated than I ever wanted to know, but here is the very short version: for every machine that's going to be part of your FreeIPA domain, run 'hostnamectl set-hostname (fqdn)', and check that just plain 'hostname' returns the FQDN for that machine. This: [adamw@id ~]$ hostname id [adamw@id ~]$ hostname -f id.happyassassin.net is not good enough for FreeIPA (edit: fine, Kerberos). No. It needs this: [adamw@id ~]$ hostname id.happyassassin.net [adamw@id ~]$ hostname -f id.happyassassin.net So just give it that, and don't worry about it too much. If you are trying to be clever and get your hostnames set by DHCP, but when you do that, 'hostname' only gives a shortname, then give up on your clever DHCP thing and just whack the hostname with a hammer. Really. It's easier this way. (Please don't ask me why there is a 'hostname' and a 'hostname -f' because it will make me cry.) Oh, and also, make sure every machine has a /etc/hosts line like this (for its own IP address and hostname): 192.168.1.XX machine.your.domain machine make sure the FQDN comes before the shortname. You can even leave the shortname out. But don't have the shortname before the FQDN. Another bear trap: it's very nice of the FreeIPA server install script to spit out an example BIND config when you run it with its own DNS configuration disabled, but it rather spoils the effect by going on to run the client install script automatically immediately after the server install script. Here's the problem: the client install script won't work properly unless your DNS records are in order. So inevitably it'll fail, because you haven't stuck all the DNS records in your DNS server's config yet. Sigh. Somewhere around here I found that the freeipa client deployment script doesn't always clean up after itself perfectly when it fails. Bear trap: if you run the client install script once, and it fails, and you fix whatever was causing it to fail, and you run it again, and now it doesn't goddamn well auto-discover the IPA server even though it goddamn discovered it JUST FINE the first time and WHY?, the answer is 'rm -f /etc/ipa/ca.crt'. Bug #1011396. You're welcome. You might also get to a point where it gets so far as to actually register the system with the FreeIPA server but then fails later. If you do, then to do a complete setup once you fix whatever was causing the failure, you'll need to pass --force-join . Once I finally worked out all THOSE bear traps, I got into a pretty good swing. I had a test client box which I used to get familar with the web UI and do a few test policies and things, and it was all working out nicely. So I was about ready to make the jump to actually registering my real desktop systems as clients and using FreeIPA in anger, to manage my HUGE RANGE of login accounts (which is...er...two, plus a 'facebook' account I use to sandbox facebook in a really janky way). Only one thing: sudo. If you're going to centralize your auth and privilege policies you probably want it to include sudo, right? Bit pointless otherwise. When you find the obviously available instructions for enabling sudo with FreeIPA, you may, like me, be liable to run off screaming. It's not quite as complex as it looks at first glance, but it is still kinda complex. More significantly, AIUI, that approach has the rather significant drawback that it won't work if the system is offline. (Most bits of a typical FreeIPA setup do, thanks to sssd, which does credential caching; you can assume anything in your FreeIPA setup which runs through sssd will work just fine offline, I think). Fortunately, there is another approach which is both simpler to implement and works offline, so what you should do is just use that. It's documented very concisely here: that is, indeed, all you need to do, on each machine in the domain. You will also want to add this line to /etc/rc.d/rc.local: nisdomainname your.domain or else group-based policies won't work, I think. So once I found that out, I got on a nice roll of enrolling most of my machines in the domain and setting up a few basic policies. For each machine I'd tidy up so each existing local user matched the UID and GID for that user in the FreeIPA config, enrol it as a client, add the host to the appropriate host group so my policies took effect, set up the sudo stuff, check everything was working, and then delete the local user accounts. It was all working nicely all along the way. Then I thought 'hey, I'm basically nearly done. I guess I should shut down the FreeIPA server VM and take a backup copy of it at this point.' So I did. Then I booted up the FreeIPA server again. And found...no FreeIPA server at all. Just a failed ipa.service for no particularly obvious reason, in the logs. Not much to get my teeth into at all. So I debugged that for four hours this evening, and let me tell you, that wasn't a fun one. It was complicated by the slightly weird way FreeIPA initializes itself: most of the relevant systemd units are disabled, and ipa.service actually starts them up itself. I'm guessing Lennart isn't a fan. I thought the fact that they were disabled was the bug and wasted hours fiddling about trying to enable them 'correctly', which wasn't the problem at all. No, it was this GIANT bear trap: Sometimes, it seems, when you boot a FreeIPA server machine, the directive in /etc/tmpfiles.d/dirsrv-YOUR-DOMAIN.conf which tells systemd to create a directory called /var/lock/dirsrv/slapd-YOUR-DOMAIN...just doesn't work. That causes 389 Directory Server (FreeIPA's LDAP server) to fail to start. edit: Thanks to viking-ice and nkinder, I find that this was recently re-reported, and the new bug reports have been far more fruitful. This is getting fixed, in multiple places (not just 389-ds has the problem, it turns out): FreeIPA #996716 pki-core #996847 389-ds #1008306 Once I finally found that ticket, fixing it was easy: just manually run 'systemd-tmpfiles --create', restart ipa.service, and Bob's your uncle. Well, he would be, except for this: [adamw@adam ~]$ ssh id System is booting up. Connection closed by UNKNOWN Ladies and gentlemen, what the freaking freak? Maybe some of you ran into that error before. I don't know. But judging by Google, a lot of you are like me, and never have: Googling turns up some results, but not many, and they're mostly Arch users stumbling around and kinda fixing it but not really knowing what they did. I found one reference which clued me in to exactly what was going on, though. If you ever see that message, what it means is that the file /var/run/nologin exists. This file seems to be as old as the hills, but I'd never come across it before. The idea is that it's created early in system startup and deleted at the end, then again created early in system shutdown, and while it exists, the system will deny any kind of login to any user but root. In a modern-day Fedora-y system, this is implemented by systemd (creates and destroys the file) and PAM (denies you access if it exists). The contents of the file are what is displayed when you are denied login; that's the "System is booting up." message in my output: systemd creates the file with the content "System is booting up." So here's what happened: that file is actually created during early boot by systemd-tmpfiles, on a systemd system. It's then deleted at the end of boot by systemd-user-sessions.service (which more or less exists solely to do this). But if you have a problem with systemd-tmpfiles and run it manually to correct that problem...hey, you just created /var/run/nologin, and ain't nothing gonna delete it. You just locked yourself out. Congratulations. The dumb fix for that is 'rm -f /var/run/nologin' after you run systemd-tmpfiles --create, but the smart fix is just to use a better systemd-tmpfiles command: systemd-tmpfiles --create /etc/tmpfiles.d/dirsrv-YOUR-DOMAIN.conf that tells tmpfiles to only run the directives from that specific config file, which is the only one we need, and avoids the whole /var/run/nologin problem, as that file is listed in a different tmpfiles config file. So for now I just disabled ipa.service and have an icky /etc/rc.d/rc.local which runs systemd-tmpfiles then starts ipa.service, and that's working well enough for now. And...whew, that brings me pretty much up to RIGHT NOW. I have a FreeIPA server which works, and survives a reboot, and all my systems but one are members of the domain, with no local user accounts any more, using the FreeIPA server for authentication and such. Oh, yeah, forgot to mention, one rather useful feature of FreeIPA is that you can store ssh pubkeys for each user, for 'authorized_keys' purposes: instead of having to create ~/.ssh/authorized_keys on every goddamn machine you ever configure, if you use FreeIPA, authorized_keys for any given user will just be read from the IPA server, so for any machine that's a member of the domain it'll "just work". Nice. Just one thing: I haven't enrolled my mail server yet. Which is the original goal of the whole bloody exercise. But that's because it'll be the most complex bit - as postfix and dovecot are using the system user database for auth at the moment, I'll have to FreeIPA-ify those services as well, when I register it. So I was saving it for last. owncloud is using its own user db ATM, so I could happily register www in the FreeIPA domain and wipe local user accounts without breaking owncloud, but for mail, I'll have to work through the guides for setting up postfix and dovecot to use the FreeIPA server. But once I've done that, I should be home freeeee. And then, with luck and a following wind, I can pretty much grant anyone access to the main happyassassin.net services just by creating them a user on the server and sticking them in a particular group. Everything else should happen magically. Neato.