Tag Archives: script

Safely running bulk operations on Redis with lua scripts

This article was also posted on the Gumtree devteam blog

If there was one golden rule when working with redis in production, it would be

“Don’t use KEYS”

The reason for this is that it blocks the redis event loop until it completes, i.e. while it’s busy scanning its entire keyspace, it can’t serve any other clients.

Recently, we had a situation where code was storing keys in redis without setting an expiry time, with the result that our keyspace started to grow:
Continue reading

Pausing Spotify and playing a random video in Python – A party trick for Halloween

For a Halloween party last weekend I wrote a python script to pause Spotify, play a random video and start music playback again. The videos were basic ogg files I cobbled together which showed a scary image and evil laughs or screaming with OpenShot. I can’t really share them, as I don’t have rights to the media, but it’s pretty simple to recreate them yourself.

The code for this script is on Github, and I’ve reproduced the latest snapshot below. Feel free to fork and improve if you want to scare your guests, or add support for other OS’s. Presently it only supports Linux because I used dbus to perform the play/pause actions.

[python]
#!/usr/bin/python

”’
This is a Halloween party script which pauses Spotify and plays a video
at random intervals.
”’

import random
import subprocess
from subprocess import call
from time import sleep
import os
import datetime

start_time = datetime.time(21, 0, 0)
stop_time = datetime.time(23, 0, 0)

video_dir = ‘/home/alex/Videos/scream/’
videos = { ‘scream1_nofade.ogg’: 30,
‘happy.ogg’: 1,
‘evil_laugh.ogg’: 5,
}

def time_in_range(start, end, x):
“””Return true if x is in the range [start, end]”””
if start <= end:
print(“start<end”)
return start <= x <= end
else:
print(“end<start”)
return start <= x or x <= end

def weighted_choice(weights):
total = sum(weights for video in weights)
r = random.uniform(0, total)
upto = 0
print(“total: %s\nrandom: %s” % (total, r))

for video in weights:
w = weights
if upto + w > r:
return video
upto += w
assert False, “shouldn’t get here”

def spotifyPause():
command = “dbus-send –print-reply –dest=org.mpris.MediaPlayer2.spotify /org/mpris/MediaPlayer2 org.mpris.MediaPlayer2.Player.Pause”
print(“pausing spotify”)
os.system(command)

def spotifyPlay():
print(“playing spotify”)
command = “dbus-send –print-reply –dest=org.mpris.MediaPlayer2.spotify /org/mpris/MediaPlayer2 org.mpris.MediaPlayer2.Player.PlayPause”
os.system(command)

def play_video(video_file):
print(“Playing %s” % video_file)
#call([‘/usr/bin/mplayer’, ‘-fs’, video_file], stdout=None, stderr=None)
#result = subprocess.Popen([‘/usr/bin/mplayer’, ‘-really-quiet’, ‘-fs’, video_file])
result = subprocess.check_call([‘/usr/bin/mplayer’, ‘-really-quiet’, ‘-fs’, video_file], stdout=None, stderr=None)
return result

def playBuzz(buzzfile):
print(“Buzz…”)
result = subprocess.check_call([‘/usr/bin/mplayer’, ‘-really-quiet’, ‘-ss’, ’18’, buzzfile], stdout=None, stderr=None)
return result

def infiniteLoop():
while 1:
current_time = datetime.datetime.now().time()
#if current_time > stop_time or current_time < midday:

choice = weighted_choice(videos)

random_time = random.randrange(1200,2400)
random_time = 3

video_file = video_dir + choice
print(“Chose video %s after %s seconds” % (video_file, random_time))
sleep(random_time)

# Whether to play buzz
buzz = False
if random.randrange(0,100) > 90:
buzz = True

# Continue if outside time range
if not time_in_range(start_time, stop_time, current_time):
print(“Not playing video, outside time range”)
continue

# Do it
spotifyPause()
if buzz:
playBuzz(‘/home/alex/Videos/scream/audio/buzz.mp3’)
play_video(video_file)
spotifyPlay()

if __name__ == “__main__”:
infiniteLoop()
[/python]

Photo workflow script

After trying several photo management tools I’ve concluded that the best way to manage a photo library is the simplest – copy the files yourself and name the directory.

I also often shoot in RAW+JPG mode on my camera and like to keep the files separate to make viewing saner, so under each event directory I will have a RAW and JPG directory as well.

After downloading, the first step is always to delete the photos that didn’t come out. With both raw and jpeg files to delete this process is a bit of a pain, especially when they’re in separate directories, so I wrote a script that deletes the raw file when the jpg is missing. That way I can simply browse through the JPG directory with Eye of Gnome, delete the photos I don’t want, and the script handles cleanup of the RAW directory.

https://github.com/al4/scripts/blob/master/bash/rawdel.sh

The code is below but you should use the github link above in case it gets improved at some point in the future.

[shell]
#!/bin/bash

# rawdel.sh, a photo workflow script to delete raw files when the jpg has been removed
# By Alex Forbes
#
# I frequently shoot in RAW+JPG mode and when downloading from the camera I like to separate
# the raw and jpg files into separate directories. I then go through the jpg directory and
# delete the rejects. It is a pain to have to manually delete the corresponding raw files as
# well, so I wrote a script to do it for me.
#
# It simply removes RAW files from a directory when the corresponding JPG file has been removed.

# Set these
rawextn=”CR2″ # raw file extension (Canon is CR2)
rawdir=”./RAW” # directory where raw files reside
jpgdir=”./JPG” # directory where jpg files reside
# rawdir and jpgdir can be the same

# Working variables, leave as-is
list=”” # list of files that have been deleted
rawlist=”” # the list of raw files that we will delete
filecount=”” # number of files we will delete

# Operate on each raw file
for f in $(ls -1 $rawdir/*.$rawextn); do
# Corresponding JPG file is:
jpgfile=$(basename $f | sed “s/\.$rawextn$/.JPG/”)

# If this JPG file doesn’t exist
if [ ! -f $jpgdir/$jpgfile ]; then
# Add to our list of files that have been deleted
list=$(echo -e “${jpgfile} ${list}”)
fi
done

# Convert jpg filenames back to corresponding raw filenames
rawlist=$(echo ${list} | sed ‘s/\.JPG$/.CR2/g’)
filecount=$(echo -e ${rawlist}| awk ‘END{print NF}’)

if [ $filecount == 0 ]; then
echo “No files to delete”
exit 0
fi

echo -e “About to remove $filecount files:\n${rawlist}”
read -p “Continue? [Y/N] ” prompt

if [[ $prompt = “Y” || $prompt = “y” ]]; then
# Delete all files in the list
for f in ${rawlist}; do
rm -v $rawdir/$f
done
exit 0
else
echo -e “\nAborted.”
exit 1
fi
[/shell]

I Sincerely Hope I Never Write a Script like this Again

This sort of stuff destroys your soul:
[shell]
#!/bin/bash
logfile=~/slave-watcher.log

while true; do
status=$(mysql –execute=”show slave status\G”|grep “Seconds_Behind_Master:”|awk ‘{print $2}’)

if [ $status == “NULL” ]; then
mysql –execute=”show slave status\G” | grep “Last_SQL_Error:” | tee -a $logfile
mysql –execute=”set global sql_slave_skip_counter=1; start slave;”
fi

sleep 1
done
[/shell]
What it’s doing is looking at MySQL’s slave status and skipping over any statement that causes an error. There are so many reasons why you should never do this, and I won’t go into detail on what necessitated it here, but you’ll be glad to know the slave I set this loose on was not in production!

Bash script to alert when memory gets low

We have a web server that’s running out of memory about once every couple of weeks. The problem is that when it happens the swap file is totally full, the system is unresponsive and it usually needs a hard reboot. So it’s a bit difficult to debug. To avoid digging through log files I don’t understand I elected to put a script in /etc/cron.hourly which checks the total amount of free memory (including swap and physical). If there is less than 256mb free (this server has 512mb of ram and a 1gb swap so at this point the situation is serious), it dumps the process list to /tmp/processes.txt and sends me an email with it attached.

Note that mutt must be installed (‘apt-get install mutt’ on Debian/Ubuntu, or ‘yum install mutt’ on RedHat/CentOS).

#!/bin/bash

free=`free -mt | grep Total | awk '{print $4}'`

if [ $free -lt 256 ]; then
        ps -eo %mem,pid,user,args >/tmp/processes.txt
        echo 'Warning, free memory is '$free'mb' | mutt -a /tmp/processes.txt -s "Server alert" [email protected]
fi

Then of course make it executable and symlink to cron.hourly:

chmod +x /etc/scripts/memalert.sh
ln -s -t /etc/cron.hourly/memalert.sh /etc/scripts/memalert.sh

Automatically update DansGuardian Filter Groups List from LDAP

Update 20th March 2011: Heath has made some modifications to the original script and made it more efficient, see the comments below.

Here’s a script I wrote today, which updates the filtergroupslist file of Dansguardian. If you’re using LDAP authentication and want to give different levels of protection to certain groups of users, you need to update the list somehow, as Dansguardian doesn’t support LDAP groups. See this page for more info on filter groups.

The school I wrote this for is a Novell eDirectory site, and it will require a bit of modification to work on other sites. In particular you will need to alter the parameters of the ldapsearch command (filter string, server name, user credentials). Other LDAP servers may not support the ufn attribute, which this is based on. If your directory is well maintained and up to date you would probably be better of using the uid attribute, but this particular school hasn’t populated it for all users yet (only users created with ConsoleOne and iManager populate the uid attribute by default). If you do use uid, be sure to remove the cut command.

ldapsearch outputs data in ldif format, which is difficult to use in scripts. The tool to use to convert this is awk, which unfortunately is a language I haven’t learnt yet. So I found a premade awk script which converts ldif2csv (from here), removed out all the attributes and replaced them with just ufn (you may want to use uid instead).

If you use this script and modify or improve it, I’d appreciate you contributing the modifications back, as they may be useful to others (myself included)!

updateFilter.sh

#!/bin/bash 
#
# Dansguardian filter group update script
# Alex Forbes, Edtech Ltd
# Updated 9th September 2009
#

## Variables
# Dansguardian filtergroupslist file
DESTFILE=/root/filtergroupslist-test
LOGFILE=/var/log/dansfgupdate.log

# LDAP settings
LDAPFILTER="(&(objectClass=Person)(|(groupMembership=cn=ALL-TEACHERS,ou=TCHR,o=HWK)(groupMembership=cn=ALL-ADMIN,ou=ADM,o=HWK)(groupMembership=cn=OESAdmins,o=HWK)))"

# Which filtergroup do you want the users to be a member of
FILTERGROUP=filter2

# Path to the awk script (converts the ldif file to parseable text). I modified one from
# http://www.yolinux.com/TUTORIALS/LinuxTutorialLDAP-ScriptsAndTools.html
AWKSCRIPT=/opt/ldif2csv.awk
TMP=/tmp

# Dansguardian filter group list file
# Temp path, creates folder for the temp files. There are probably better ways of doing it.

# Make temp directories
WIP=$TMP/dgFilterUpdate
mkdir -p $WIP

# Header message
echo "## This file is automatically updated, any changes will be overwritten" > $WIP/4final
echo "## See /opt/edir2dansg.sh" >> $WIP/4final
echo "" >>$WIP/4final

# Perform LDAP search. Outputs ldif file.
ldapsearch -uxvA -H ldaps://fs2.howick.school.nz -b "o=HWK" -S '' -s "sub" -D cn=ldapauth,o=hwk -w password "$LDAPFILTER" ufn > $WIP/1ldif

# Picks out the ufn attribute using a modified awk script I found:
awk -F ': ' -f $AWKSCRIPT  $WIP/2txt

# Picks the first field of the ufn attribute to generate a clean list of users
cut -d, -f1 $WIP/2txt > $WIP/3userlist

# Add the values required to meet the dansguardian filter format
for u in `cat $WIP/3userlist`; do
	echo "$u=$FILTERGROUP" >> $WIP/4final
done

# Finally, copy the file to overwrite the dansguardian list.
# I've done a simple check to make sure the file isn't too small in case of error, but it could be handled better.
SIZE=`stat -c %s $WIP/4final`
if [ $SIZE -gt 2500 ]; then
	cp $WIP/4final $DESTFILE
	echo $(date +"%Y/%m/%d %H:%M"): Updated filter groups list "("size $SIZE bytes")" >> $LOGFILE
else echo $(date +"%Y/%m/%d %H:%M"): Output file is too small, list not updated >> $LOGFILE
fi

# Gentle reload of dansguardian
dansguardian -g

And the modified awk script, ldif2csv.awk:

BEGIN {
        ufn = ""
      }
/^ufn: /              {ufn=$2}
/^dn/ {
        if(ufn != "") printf("%sn",ufn)
        ufn     = ""
      }
# Capture last dn
END {
        if(ufn != "") printf("%sn",ufn)
}

Update 9-9-09: Fixed a few dumb mistakes.

Simple File Backup to Email Script

Here’s a file backup script I installed for a client. The original outline came from a post on the ubuntu forums (I forget where exactly), but it’s simple enough. It creates an archive in /tmp, zips it up, emails it then deletes the archive. If your target is a linux computer then it makes more sense to gzip it by adding a “z” to the tar options (i.e. tar -czf) and removing the zip line.

#!/bin/bash
#
# Simple file backup script, creates archive in /tmp and emails it.
#
# Software required:
#  zip
#  tar
#  mutt

# Variables
[email protected]
SOURCE="/home/user1 /home/user2"
SERVERNAME=server.example.com
MAIL=`which mutt`
ZIP=`which zip`
DATE=`date +%Y_%m_%d`
FILE=myfiles-$DATE.tar
DESTINATION=/tmp/$FILE
ZIPFILE=$DESTINATION.zip

# Actions
tar -cf $DESTINATION $SOURCE 2> /dev/null
$ZIP $ZIPFILE $DESTINATION
$MAIL -a $ZIPFILE -s "Backup for $DATE" -s "$SERVERNAME backup $DATE" $MAILADDR < /dev/null
rm $ZIPFILE $DESTINATION

For mutt to work you need an MTA (mail transport agent) such as postfix. If it’s not installed and you don’t need it for anything else, configure it as a satellite system (the Ubuntu/Debian packages prompt you on install and satellite system is an option). This prevents spammers from using it as a relay, and ensures the mail goes to your real mail server.