Fedora 19/20 logfile explosions

PSA: if you're running Fedora 19 or 20, I highly recommend you stop what you're doing right now and do this instead. There are a couple of unfortunate bugs in F19/F20 right now which may well be screwing the hell out of your log files. systemd-journald bug combines with new rsyslog to eat your CPU time, make /var/log/messages huge, break journalctl and generally spread woe and misery SELinux denial happening every three seconds to GNOME users Chapter One 1) Run top and check if 'rsyslogd' is sucking a bunch of your CPU time 2) If so, immediately stop rsyslog.service, do 'yum downgrade rsyslog', and restart it 2b) If not, breathe a sigh of relief, and make a note not to update to rsyslog 7.4.0 until this blows over. But still read the following chapters EDIT 2.5) There's now a systemd update which should both prevent any further buggy journal entries being written, and cope with reading existing buggy entries better: systemd-204-8.fc19. After updating to that systemd, the infinite loop in buggy journals should no longer occur. If you prune your journal as described in the rest of this post, and update to systemd 204-8, you should then be able to update back to rsyslog 7.4.0 without further problems. Once we get systemd 204-8 pushed stable, new F19 installs should no longer ever be affected by this problem in any way, so please upkarma it. 3) If you got bitten, /var/log/messages is now probably infeasibly huge (mine was 8GB). The easiest thing to do is probably just nuke it. If you have valuable logs, they can probably be extracted from journalctl. ... ON WHICH TOPIC Chapter Two If you got hit by chapter one, you're now in for some joyous fun. If your system logs are of no particular important to you, you can probably just go to /var/log/journal and blow *.journal* away, and you should now be more or less happy (though read Chapter Three too). The files in /var/log/journal are the Fabled Systemd-Journald Binary Logfiles themselves: you wipe one, you lose all the logs in it. If your system logs are important to you, you get to have some fun. Run 'journalctl --no-pager' and wait a while. It'll likely start looping around some very small period of time, endlessly. ctrl-c it, go to /var/log/journal , do 'ls -ltr' and spot the file(s) with a very close timestamp to where the loop happened. Move those files somewhere else (if you want to retain maximum data, you can move them one at a time, doing 'journalctl --no-pager' after each move until you hit the exact combo of files you need to move to unblock it. You may need to rinse and repeat: I had five different loops in mine. You may now be able to update rsyslog to 7.4.0 again and be okay, but personally, I'm leaving it on the old build till things quiet down a bit. Chapter Three You may well still have a giant assload of SELinux alerts about bug #973849 in your logs - "SELinux is preventing /usr/libexec/accounts-daemon from 'read' accesses on the directory /var/log." accounts-daemon tries this read about every three seconds, spamming the logs each time it fails. So first order of business, install selinux-policy-3.12.1-52.fc19 to fix it. Then you can clean up your logs, if you like. I just gave up and wiped all the really large files in /var/log/journal - if you do an 'ls -lSr' in that directory you should see that most files are a few MB at most, but a few recent ones are tens or hundreds of MB in size. I killed all of those. You can leave 'em, but they eat space and make journalctl really slow. Your choice. Chapter Four While doing all this you might notice a bunch of rather old files with the extension "journal~" in /var/log/journal . It seems that journald doesn't rotate these (they're journal files that were uncleanly closed on shutdown). I manually deleted all the really old ones I had. Chapter Five If you have gnome-shell-extension-fedmsg installed and you're not really using it, you might want to consider removing it, because it spams the hell out of the journal: see bug #974429. Just something else I noticed on my log sanitization quest. I am now running a sweepstakes on the first slashdot "systemd SUCKS!" post to cite this little mess as justification for the 'binary logs are awful!' meme...