We have a web server that’s running out of memory about once every couple of weeks. The problem is that when it happens the swap file is totally full, the system is unresponsive and it usually needs a hard reboot. So it’s a bit difficult to debug. To avoid digging through log files I don’t understand I elected to put a script in /etc/cron.hourly which checks the total amount of free memory (including swap and physical). If there is less than 256mb free (this server has 512mb of ram and a 1gb swap so at this point the situation is serious), it dumps the process list to /tmp/processes.txt and sends me an email with it attached.
Note that mutt must be installed (‘apt-get install mutt’ on Debian/Ubuntu, or ‘yum install mutt’ on RedHat/CentOS).
#!/bin/bash
free=`free -mt | grep Total | awk '{print $4}'`
if [ $free -lt 256 ]; then
ps -eo %mem,pid,user,args >/tmp/processes.txt
echo 'Warning, free memory is '$free'mb' | mutt -a /tmp/processes.txt -s "Server alert" email@me.com
fi
Then of course make it executable and symlink to cron.hourly:
chmod +x /etc/scripts/memalert.sh ln -s -t /etc/cron.hourly/memalert.sh /etc/scripts/memalert.sh
Hello,
Just a few tips for the next version:
free -mt | grep Total | awk ‘{print $4}’
is the same that:
free -mt | awk ‘/Total/{print $4}’
Also when I parse commands output, I use to set LANG=C into the script, to avoid locales (“Total” is not said “Total” in all languages)
And in this case, we can avoid the mutt depend to attach files, and insert the txt into the mail body:
Greetings
arf, bad editing, little typo corrected:
mmm it seems that it is wordpress that eats the code, if you want to edit the first comment, it should look like this: http://pastie.org/1264823
Cheers, your version is definitely an improvement!
Hey Alex,
I’ve used mutt too, as I don’t know how to use mailutils. However in my server I have a laptop HDD installed in the name of noise, whenever memory is running low and it uses swap, everything slows down a lot, so I have a warning system for when memory is being used.
#!/bin/bash
memfree=”`cat /proc/meminfo | grep MemFree | cut -d: -f2 | cut -dk -f1`”
memfreemb=$(echo “scale=5; $memfree/1024″ | bc -l| awk -F. ‘{if(($2/10^length($2)) >= .5) printf(“%d.0n”,$1+1);else printf(“%d.0n”,$1)}’)
if [ $memfree -lt 204800 ]; then
ps -eo %mem,pid,user,args >/tmp/processes.txt
echo ‘Warning, free memory is ‘$memfreemb’mb’ | mutt -a /tmp/processes.txt -s “Server Alert” — email@me.com
fi