We have a web server that’s running out of memory about once every couple of weeks. The problem is that when it happens the swap file is totally full, the system is unresponsive and it usually needs a hard reboot. So it’s a bit difficult to debug. To avoid digging through log files I don’t understand I elected to put a script in /etc/cron.hourly which checks the total amount of free memory (including swap and physical). If there is less than 256mb free (this server has 512mb of ram and a 1gb swap so at this point the situation is serious), it dumps the process list to /tmp/processes.txt and sends me an email with it attached.
Note that mutt must be installed (‘apt-get install mutt’ on Debian/Ubuntu, or ‘yum install mutt’ on RedHat/CentOS).
#!/bin/bash
free=`free -mt | grep Total | awk '{print $4}'`
if [ $free -lt 256 ]; then
ps -eo %mem,pid,user,args >/tmp/processes.txt
echo 'Warning, free memory is '$free'mb' | mutt -a /tmp/processes.txt -s "Server alert" [email protected]
fi
Then of course make it executable and symlink to cron.hourly:
chmod +x /etc/scripts/memalert.sh ln -s -t /etc/cron.hourly/memalert.sh /etc/scripts/memalert.sh
Hello,
Just a few tips for the next version:
free -mt | grep Total | awk ‘{print $4}’
is the same that:
free -mt | awk ‘/Total/{print $4}’
Also when I parse commands output, I use to set LANG=C into the script, to avoid locales (“Total” is not said “Total” in all languages)
And in this case, we can avoid the mutt depend to attach files, and insert the txt into the mail body:
Greetings :)
arf, bad editing, little typo corrected:
mmm it seems that it is wordpress that eats the code, if you want to edit the first comment, it should look like this: http://pastie.org/1264823
Cheers, your version is definitely an improvement!
The http://pastie.org/1264823 is no longer valid link
Hey Alex,
I’ve used mutt too, as I don’t know how to use mailutils. However in my server I have a laptop HDD installed in the name of noise, whenever memory is running low and it uses swap, everything slows down a lot, so I have a warning system for when memory is being used.
#!/bin/bash
memfree=”`cat /proc/meminfo | grep MemFree | cut -d: -f2 | cut -dk -f1`”
memfreemb=$(echo “scale=5; $memfree/1024” | bc -l| awk -F. ‘{if(($2/10^length($2)) >= .5) printf(“%d.0n”,$1+1);else printf(“%d.0n”,$1)}’)
if [ $memfree -lt 204800 ]; then
ps -eo %mem,pid,user,args >/tmp/processes.txt
echo ‘Warning, free memory is ‘$memfreemb’mb’ | mutt -a /tmp/processes.txt -s “Server Alert” — [email protected]
fi
So what do you do once you get the notification? What process is the memory hog?
Wow this seems old now. Cringing at my lack of bash (and Linux) skills.
Kernc, the -o option tells you this, the first column is “%mem”, i.e %age of memory used. Now I would probably do ps -eo %mem,pid,user,args | sort -rn, so that the processes are sorted in order of memory usage. And of course pipe it straight to mail as suggested by poisonbit rather than attach a text file with mutt.
I used your code, to build an auto-deploy gist
https://goldmanmalka.com/2019/03/19/bash-script-to-alert-when-memory-gets-low/