Scalr running out of disk space if sendmail not running!
Recently had another quirky issue with Scalr.
Scalr generates a lot of email notifications via the Cron jobs. On my instance I have sendmail turned off so that these email notifications don’t go out.
After running continuously for about 4 months, the Scalr instance started failing with an “Out of diskspace” error. This was very strange since there was still about 25GB free on the disk.
After checking all kinds of configurations and permissions, a google search yielded lack of inodes as a possible error!
A quick “df -i” confirmed the error, IUse% was 99 and there were a handful of inodes free! Tracing this back to find out where all the inodes had been consumed took longer. After a lot of hunting I ended up with the /var/spool/clientmqueue directory. An “ls” in this folder ended up with my terminal “hanging” as there were so many files created in that folder. Even an “rm -rf *” at the clientmqueue folder will not work and you will see the “Argument list too long” after some time.
Instead use the “find . -type f -print0 | xargs -0 rm” command in the /var/spool/clientmqueue folder. On my instance this command ran for a few hours due to the large number of files. (You may want to execute the command in the background in a screen command).
Once done the inode utilization fell to 7% and the system is again fine. In order to avoid future issues, I added a cron job to delete the files in the clientmqueue folder periodically.
Note: Scalr is a fantastic software that works great. This blog post only addresses one of the issues faced in running Scalr and has nothing to do with the Scalr software itself.