How to kill a server with logrotate
 

Howdy y'all.

I had to do some sleuthing on an inherited server and thought someone out there might find it interesting...

We have recently been given a half dozen RHEL & Oracle RAC servers to run and a few weeks into production we're getting automatic tickets raised about excession usage of /dev/sda on a system that's only running RAC from a NAS, suggesting minimal disk activity should occur locally.

So looking at the reports we see a spike, or a table-top mountain, in disk IO between about 03:50 and 04:20 every morning. Sound like a familiar time? cron.daily! But there was nothing interesting in the cron directory, or /etc/crontab, but the personal root crontab actually contains a "logrotate -f oraclelogs" kicking off around 3:40, so there we go... Why is that there?? Why didn't whatever muppet implemented that use cron.daily?? Slack. Now looking at the logrotate file there are about 20 different directories, the entire contents of each is being rotated and eventually we find one single log directory containing 540000 files. Roughly. Nice.

These log files are called something like css84952.log where the number appears to be a pid. And these appear to be client connections which spawn a new process to do its thing, so each connection creates a new, countably unique, log file with 2 lines in it.

Now about here I start laughing as the default logrotate config uses the "create" directive, touching a new file in place of the one it just rotated and gzipped.

SO... joining the dots, every day approximately 200-300 new log files were created, with minimal content. And due to the "create" happening, once a file exists, it will always exist forevermore, and then have 8 files per original file to process, with the whole number of files, and work, getting bigger and bigger and bigger everyday, until the server would probably just run out of inodes on that filesystem.

Solutions... add "nocreate" to this config file, and kick whoever set this up in the first place in the nuts. And tell them to NEVER use wildcards in a logrotate file!

There may well have been an Oracle level solution too, but that's none of my business really...

Logrotate
 

Thanks man...
I haven't had this issue but it makes you think outside the box...
It's a great thing to have a post like this, just to tell us what we might face one day or maybe just to help us not to make that mistake.

We're a RHEL + Oracle RAC shop too. Good to be aware of.