Notes on a couple of FreeIPA bugs: host group sudo rules and failure to start with recent pki-core on older, upgraded installs

So I've had a couple of issues with my personal FreeIPA deployment in the last couple of months that I never managed to dig into properly because of work on Fedora 24. But now F24 is done, I had time to figure them out, finally.

The first bug is simpler and probably has a wider impact. Fedora 24 ships with sudo 1.8.16. Unfortunately, there seems to be an issue between this version of sudo and FreeIPA for sudo rules that use FreeIPA host groups. If you have a FreeIPA sudo rule which says 'allow user foo to sudo on all systems in host group bar', it probably won't work with sudo 1.8.16. You can fix it for the short term by downgrading to sudo 1.8.15. The bug is fixed upstream in sudo 1.8.17, so hopefully we'll either get a 1.8.17 update or a backport of the fix to 1.8.16 for Fedora 24 soon.

The second bug was a hell of a lot more complex to figure out, but we got there in the end! There was a change to FreeIPA between Fedora 21 and Fedora 22 whereby certificate profiles for the CA portion of FreeIPA are now stored in LDAP, rather than on the filesystem, by default. Things are supposed to be set up such that when you upgrade an installed system to the FreeIPA and pki-core versions that implement this change, the FreeIPA upgrade process changes the pki-core config file to use the LDAP storage system, and imports the profiles into the LDAP database.

I upgraded my server to Fedora 23 right when it came out, and the upgrade process was pretty rough at that time. I don't have enough logs left to figure out exactly what went wrong, but it looks like one of my runs of the upgrade process happened to break between the config file change and the import of the profiles to the database, so pki-core was set up to use the LDAP storage system, but the profiles weren't actually in the LDAP database.

This is obviously broken, but it didn't actually prevent FreeIPA working (in most ways - I imagine certificate management might have been broken, if I used it more) until quite recently. A change in pki-core results in it failing to start entirely if you're in this situation. That change landed in Fedora in pki-core-10.2.6-14 (for F23 and F24) and pki-core-10.2.6-10.fc22 (for F22). From that build on, pki-tomcat will fail to start up if you have the messed up configuration, rather than starting up but being a bit broken.

We did figure out that since the relevant pki-core packages landed in F22+, upgrades of FreeIPA servers from Fedora 21 are broken. The problem there is that the upgrade process tries to edit the configuration file, restart pki-tomcat, then import the profiles - but when it tries to restart pki-tomcat that fails because of the bug, and the profile import can never happen. So the upgrade process fails and leaves the system in the broken state.

So if you tried to upgrade a Fedora 21 (or earlier) FreeIPA server recently and found FreeIPA would not start after the upgrade, this might be the reason. The workaround is actually not too hard, though.

First, downgrade all pki-core packages to the version right before the change landed in Fedora. For instance, for Fedora 23, the easiest way to do this is - in an empty directory - run koji download-build pki-core-10.2.6-13.fc23, then dnf downgrade *.rpm. For F22 or F24, just use pki-core-10.2.6-13.fc24 or pki-core-10.2.6-9.fc22; on F24 you may also need to do rpm -e --nodeps pki-base-java before the downgrade (it's a new package that was added in later pki-core builds, and dnf downgrade can't really cope with removing packages that were added by the new build, so you have to hit it with the big stick). Second, edit /etc/pki/pki-tomcat/ca/CS.cfg and replace subsystem.1.class=com.netscape.cmscore.profile.LDAPProfileSubsystem with subsystem.1.class=com.netscape.cmscore.profile.ProfileSubsystem. Finally, run ipa-server-upgrade as root. It should run successfully, and at this point the certificate profiles will be imported - so you can re-update the pki-core packages to the latest version, and things should work properly.

I'm hoping to implement some FreeIPA upgrade tests in openQA this cycle; now we have clean install FreeIPA testing working, extending it to test upgrades shouldn't be too hard, and hopefully would help us avoid some upgrade issues in the future...