Category Archives: Sysadmin

Ubuntu Home Server 14.04

I had grand intentions.

This home server article was to be a detailed masterpiece, a complete documentation of my home server setup.

It hasn’t turned out that way, and many pieces are missing. Turns out, that writing a detailed article on setting up a server is much harder than just doing it! So what you see here is what I finally managed to publish, 5 months after actually building it. I hope you find it useful, and I don’t rule out the possibility that I may update parts of it in future. Continue reading

Ubuntu Home Server 14.04 – A DIY NAS

It’s been more than 4 years since I wrote about home servers, but my Ubuntu Home Server article was, for a while, the most popular post on this blog. Since moving to the UK though, I’ve taken a more appliance-based approach to my home network. For the last few years I’ve been using a Boxee Box for media playback, and a 4-bay Netgear ReadyNAS duo NV2+ for storage, mainly to keep the bulk of my possessions to a minimum.

The appliance approach does have advantages. It is power efficient, easy to setup, and very low maintenance. But after getting an internet connection with decent upload speed, I wanted to run CrashPlan on the NAS without having to have another PC running. I managed to get it running by following directions I found here.

There’s just one problem:

3.3 months to upload 350GB is a little too long

3.3 months to upload 350GB is a little too long

Performance is abysmal, and I’ve only selected the most important data – my photos. I’m limited not by my internet connection, but by the NAS’s anaemic CPU and lack of ram (just 256Mb). Furthermore, it’s always had very slow read and write speeds – generally around 2Mb/sec, and loading a large directory via its Samba shares can take a while.

So I started to look for a replacement. My requirements:

  • Minimum 2GB ram
  • Strong CPU, preferably x86
  • 4+ drive bays
  • Linux based OS
  • Root access to said OS

The best pre-built option I could find which meets those requirements is the Thecus N5550, but at £383 it is a long way from cheap. And it barely meets the specs; an Atom CPU is strong for a NAS but not by modern x86 standards.

While the customised software shipped with a NAS does offer some conveniences, it also gets in the way of using newer Linux features such as BTFS RAID 5/6 (which is currently not considered stable but should be within the next 12 months). You’re also reliant on the vendor for distribution upgrades, and the priority is going to be shiny features which consumers will appreciate, not keeping the foundation OS up to date. The ReadyNAS NV2+ is currently running Debian Squeeze, and will be until the day support ends.

At this point I realised that a pre-made NAS with the level of power and flexibility I wanted doesn’t exist at a realistic price point. And with the end of Boxee support its days as a useful device are numbered, so a HTPC could be on the cards as well. It’s time to build my own server again.

Continue reading

They used to call this /.’d

Maybe these days it’s “hackernews’d”.

Some kind person posted a link to this article, which resulted in an email alert from Linode about outgoing traffic at midnight this evening.

This blog runs on a single wee Linode instance, but fortunately it’s over-engineered for its usual traffic volume, and served by nginx, php-fpm and the WordPress totalcache plugin.

It seemed to weather the storm really comfortably with load hovering around 0.2.

Network Graph

Wordpress Graph

Safely running bulk operations on Redis with lua scripts

This article was also posted on the Gumtree devteam blog

If there was one golden rule when working with redis in production, it would be

“Don’t use KEYS”

The reason for this is that it blocks the redis event loop until it completes, i.e. while it’s busy scanning its entire keyspace, it can’t serve any other clients.

Recently, we had a situation where code was storing keys in redis without setting an expiry time, with the result that our keyspace started to grow:
Continue reading

Fixing Puppet 3.2 symlinks on OSX Mavericks

Received the following error when running puppet after upgrading to Mavericks:

/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:45:in `require': cannot load such file -- puppet/util/command_line (LoadError)
	from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:45:in `require'
	from /usr/bin/puppet:3:in `<main>'

Solution is to symlink the packages to the new ruby 2.0.0 directory:


sudo ln -s /usr/lib/ruby/site_ruby/1.8/puppet /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/site_ruby/2.0.0/puppet
sudo ln -s /usr/lib/ruby/site_ruby/1.8/puppet.rb /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/site_ruby/2.0.0/puppet.rb
sudo ln -s /usr/lib/ruby/site_ruby/1.8/semver.rb /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/site_ruby/2.0.0/semver.rb
sudo ln -s /usr/lib/ruby/site_ruby/1.8/facter /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/site_ruby/2.0.0/facter
sudo ln -s /usr/lib/ruby/site_ruby/1.8/facter.rb /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/site_ruby/2.0.0/facter.rb
sudo ln -s /usr/lib/ruby/site_ruby/1.8/hiera /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/site_ruby/2.0.0/hiera
sudo ln -s /usr/lib/ruby/site_ruby/1.8/hiera.rb /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/site_ruby/2.0.0/hiera.rb

Should be fixed in the next major version.


Calling mysqldump in Python

Python is a fantastic tool to know, and despite being a beginner I find myself using it more and more for everyday tasks. Bash is great for knocking together quick scripts, but when you want to something a little more complex such as interfacing with APIs or other systems over a network, you really need a more fully-featured programming language.

The topic of this post, however, is the kind of task that bash is perfect for. Thanks to mysqldump, a database backup script can be written in a few lines and dump/restores are easily automated. So why on earth would we do this in Python?
Continue reading

How not to program Bash

Came across this gem today:


if [ -e $COMP ]; then
    echo "Please supply a competition_id"

On first read it looks backwards – “if exist $COMP then exit”, and the programmer clearly wanted to do the exact opposite. However it actually works as desired. Mostly.

If no argument is supplied, $COMP will be a blank string, i.e. “”. Thus [ -e $COMP ] will parse to [ -e ], which returns 0 (effectively true). It’s basically like saying “does nothing exist”.

This is because the -e argument tests whether a file exists. So if an argument is supplied, [ -e $COMP ] will return 1 (effectively false), which does what the programmer intended UNLESS the argument supplied is a valid file path.

In this case a number was expected, so sure it’s unlikely to fail in this way, but it’s still an incredibly bad way to test if an argument is set. Not to mention confusing to read!

The correct way to test this by the way would be to use -z, which simply tests if a string’s length is zero:


if [ -z "$COMP" ]; then
    echo "Please supply a competition id"

Or better still, use getopts. For more info run ‘man test’ from a bash terminal.

Xen Server “The SR failed to complete the operation”

Had this problem when trying to start a newly created VM and install from certain ISO files. Some ISO images would work and others would not.

Removing the ISO image allowed the VM to start but obviously there was no OS image to install from.

The error message is generic and not very helpful, but if you do encounter it check that the ISO is readable by the Xen server. In our case it was on an NFS share but the permissions were read-only to all but the owner…. so a simple `chmod a+r *.iso` fixed it!

Percona Monitoring Plugins for Cacti and Redis auth

In implementing the Percona Monitoring Plugins for Redis on our Cacti server we discovered that they don’t support authentication. This creates a problem when your servers require authentication to issue the “INFO” command.

The Percona templates use ss_get_by_ssh.php to fetch the data, and there are functions specific to Redis in this file. So I added a variable to store the password, and modified the redis_get function to run an AUTH command on the socket before INFO.
Continue reading

I Sincerely Hope I Never Write a Script like this Again

This sort of stuff destroys your soul:


while true; do
	status=$(mysql --execute="show slave status\G"|grep "Seconds_Behind_Master:"|awk '{print $2}')

	if [ $status == "NULL" ]; then
		mysql --execute="show slave status\G" | grep "Last_SQL_Error:" | tee -a $logfile
		mysql --execute="set global sql_slave_skip_counter=1; start slave;"

	sleep 1

What it’s doing is looking at MySQL’s slave status and skipping over any statement that causes an error. There are so many reasons why you should never do this, and I won’t go into detail on what necessitated it here, but you’ll be glad to know the slave I set this loose on was not in production!