Category Archives: Linux

ZFS compression and encryption

Up until a recent overhaul, I was using btrfs in raid1 to manage the 4 drives I had in my NAS. However it’s been clear for a while that the momentum is behind zfs. It has more features, better stability, and generally inspires much more confidence when things go wrong. btrfs still has its place in managing single-device boot volumes, but for multiple physical devices, I would definitely recommend zfs over btrfs.

When I added a couple of new 16TB disks, I opted to create a new pool with a single mirror vdev. If I need to expand it in future, I’ll add another mirrored vdev to the pool.

Continue reading

Home Server – new HBA edition

Some long time readers of this blog may remember my home server articles, the most recent being “Ubuntu Home Server 14.04 – A DIY NAS“. There haven’t been any more recently because there’s not been much to report. The server described in that article, built in 2014, has been backbone of my home network ever since.

Since then, I have swapped out hard drives a couple of times (it now contains 2x16TB Seagate Exos and 4x4TB Seagate IronWolf), doubled the ram to 8GB, and added a NVME riser card (along with a cheap 128GB NVME SSD), so I could have a separate boot drive while using all 6 SATA ports for hard drives.

Along the way it also lost HTPC and media player duties to an Apple TV, so now it’s little more than a file and backup server with Plex Media Server, Syncthing, and Duplicati installed. And the operating system has been upgraded from Ubuntu 14.04 to 16.04, 18.04, 20.04 and now 22.04.

A couple of weeks ago though, it failed. And by failed I mean, all I got was blank screen when powering on. No post, and no signs of life other than spinning fans.

My immediate thought was a loose connector, or possibly memory or motherboard failure, so I disconnected everything, blew the dust out and plugged everything back in. With the hard drives unplugged, everything worked. With 4 hard drives plugged in it still worked. Then it failed again when I connected the last two.

By now I figure I’m looking at a dodgy SATA cable, SATA port, or hard drive, but the core components are obviously fine. So why not give it a minor overhaul at the same time?

Continue reading

Domain Expert vs Generalist

When should you use a blunt generalist tool, and when should you use a sharper domain-specific tool?

I posted a question on Serverfault recently, and received a relevant answer that wasn’t quite what I was looking for:

Systemd – How do I automatically reload a unit, when another oneshot service is fired by timer?

My reply to the answer thanked him for it, but mentioned that I think systemd is the right place to do this “sort of thing”. In reply to my reply, he told me that systemd is “absolutely the wrong place” to do this sort of thing, which is pretty strong language!

I think we’re approaching this from different perspectives here, so let’s break the problem down in general terms.

Continue reading

Building and Packaging a Python command-line tool for Debian

python-logo-notext-svg

Python packaging has a chequered past.

Distutils was and still is the original tool included with the standard library. But then setuptools was created to overcome the limitations of distutils, gained wide adoption, subsequently stagnated, and a fork called distribute was created to address some of the issues. Distutils2 was an attempt to take the best of previous tools to support Python 3, but it failed. Then distribute grew to support Python 3, was merged back in to setuptools, and everything else became moot!

Unfortunately, it’s hard to find reliable information on python packaging, because many articles you might find in a Duckduckgo search were created before setuptools was reinvigorated. Many reflect practices that are sub-optimal today, and I would disregard anything written before the distribute merge, which happened in March 2013.

Continue reading

Ubuntu Home Server 14.04 – A DIY NAS

It’s been more than 4 years since I wrote about home servers, but my Ubuntu Home Server article was, for a while, the most popular post on this blog. Since moving to the UK though, I’ve taken a more appliance-based approach to my home network. For the last few years I’ve been using a Boxee Box for media playback, and a 4-bay Netgear ReadyNAS duo NV2+ for storage, mainly to keep the bulk of my possessions to a minimum.

The appliance approach does have advantages. It is power efficient, easy to setup, and very low maintenance. But after getting an internet connection with decent upload speed, I wanted to run CrashPlan on the NAS without having to have another PC running. I managed to get it running by following directions I found here.

There’s just one problem:

3.3 months to upload 350GB is a little too long

3.3 months to upload 350GB is a little too long

Performance is abysmal, and I’ve only selected the most important data – my photos. I’m limited not by my internet connection, but by the NAS’s anaemic CPU and lack of ram (just 256Mb). Furthermore, it’s always had very slow read and write speeds – generally around 2Mb/sec, and loading a large directory via its Samba shares can take a while.

So I started to look for a replacement. My requirements:

  • Minimum 2GB ram
  • Strong CPU, preferably x86
  • 4+ drive bays
  • Linux based OS
  • Root access to said OS

The best pre-built option I could find which meets those requirements is the Thecus N5550, but at £383 it is a long way from cheap. And it barely meets the specs; an Atom CPU is strong for a NAS but not by modern x86 standards.

While the customised software shipped with a NAS does offer some conveniences, it also gets in the way of using newer Linux features such as BTFS RAID 5/6 (which is currently not considered stable but should be within the next 12 months). You’re also reliant on the vendor for distribution upgrades, and the priority is going to be shiny features which consumers will appreciate, not keeping the foundation OS up to date. The ReadyNAS NV2+ is currently running Debian Squeeze, and will be until the day support ends.

At this point I realised that a pre-made NAS with the level of power and flexibility I wanted doesn’t exist at a realistic price point. And with the end of Boxee support its days as a useful device are numbered, so a HTPC could be on the cards as well. It’s time to build my own server again.

Continue reading

13″ Retina Macbook Pro (late 2013) – Buyer Review

As with all my reviews, this is a totally subjective personal view and not an in-depth technical analysis. For more mainstream reviews, check out Engadet, Pocket Lint, Expert Reviews, and Casey Johnston’s Air vs Pro comparison on Arstechnica if you’re also considering an Air.

Retina vs Dell

My old faithful Dell E4300 has done its dash. Actually it still works; it runs Ubuntu well, it has an SSD and 4GB of ram which makes it pretty nippy for web browsing and lightweight tasks, but what sealed its fate was my work laptop – a 15″ Retina Macbook Pro. After getting used to that gorgeous 2880×1800 screen, I found I just couldn’t go back to the Dell any more with its 1280×800 TN LCD (ugh), horrible touchpad and 2009-era performance.

Continue reading

It’s nice to be right some times

Five short years ago I wrote an article about my desire for a Nokia N900. I was extremely enthusiastic about the device, which I saw as the future of computing and a sign of things to come. I also said:

Personally I think Linux usage overtaking Windows on personal computing devices is inevitable, and this is how it’s going to happen (although the capabilities of the N900 will have to move down to a much lower price point first). We’ll see if I’m right in 5-10 years time.

It’s now 4 years and 4 months later. I was right about Linux overtaking windows on personal computing devices, but I was wrong about how, and it happened far more quickly than I could have imagined.

Continue reading

Pausing Spotify and playing a random video in Python – A party trick for Halloween

For a Halloween party last weekend I wrote a python script to pause Spotify, play a random video and start music playback again. The videos were basic ogg files I cobbled together which showed a scary image and evil laughs or screaming with OpenShot. I can’t really share them, as I don’t have rights to the media, but it’s pretty simple to recreate them yourself.

The code for this script is on Github, and I’ve reproduced the latest snapshot below. Feel free to fork and improve if you want to scare your guests, or add support for other OS’s. Presently it only supports Linux because I used dbus to perform the play/pause actions.

[python]
#!/usr/bin/python

”’
This is a Halloween party script which pauses Spotify and plays a video
at random intervals.
”’

import random
import subprocess
from subprocess import call
from time import sleep
import os
import datetime

start_time = datetime.time(21, 0, 0)
stop_time = datetime.time(23, 0, 0)

video_dir = ‘/home/alex/Videos/scream/’
videos = { ‘scream1_nofade.ogg’: 30,
‘happy.ogg’: 1,
‘evil_laugh.ogg’: 5,
}

def time_in_range(start, end, x):
“””Return true if x is in the range [start, end]”””
if start <= end:
print(“start<end”)
return start <= x <= end
else:
print(“end<start”)
return start <= x or x <= end

def weighted_choice(weights):
total = sum(weights for video in weights)
r = random.uniform(0, total)
upto = 0
print(“total: %s\nrandom: %s” % (total, r))

for video in weights:
w = weights
if upto + w > r:
return video
upto += w
assert False, “shouldn’t get here”

def spotifyPause():
command = “dbus-send –print-reply –dest=org.mpris.MediaPlayer2.spotify /org/mpris/MediaPlayer2 org.mpris.MediaPlayer2.Player.Pause”
print(“pausing spotify”)
os.system(command)

def spotifyPlay():
print(“playing spotify”)
command = “dbus-send –print-reply –dest=org.mpris.MediaPlayer2.spotify /org/mpris/MediaPlayer2 org.mpris.MediaPlayer2.Player.PlayPause”
os.system(command)

def play_video(video_file):
print(“Playing %s” % video_file)
#call([‘/usr/bin/mplayer’, ‘-fs’, video_file], stdout=None, stderr=None)
#result = subprocess.Popen([‘/usr/bin/mplayer’, ‘-really-quiet’, ‘-fs’, video_file])
result = subprocess.check_call([‘/usr/bin/mplayer’, ‘-really-quiet’, ‘-fs’, video_file], stdout=None, stderr=None)
return result

def playBuzz(buzzfile):
print(“Buzz…”)
result = subprocess.check_call([‘/usr/bin/mplayer’, ‘-really-quiet’, ‘-ss’, ’18’, buzzfile], stdout=None, stderr=None)
return result

def infiniteLoop():
while 1:
current_time = datetime.datetime.now().time()
#if current_time > stop_time or current_time < midday:

choice = weighted_choice(videos)

random_time = random.randrange(1200,2400)
random_time = 3

video_file = video_dir + choice
print(“Chose video %s after %s seconds” % (video_file, random_time))
sleep(random_time)

# Whether to play buzz
buzz = False
if random.randrange(0,100) > 90:
buzz = True

# Continue if outside time range
if not time_in_range(start_time, stop_time, current_time):
print(“Not playing video, outside time range”)
continue

# Do it
spotifyPause()
if buzz:
playBuzz(‘/home/alex/Videos/scream/audio/buzz.mp3’)
play_video(video_file)
spotifyPlay()

if __name__ == “__main__”:
infiniteLoop()
[/python]

Changing boot order in Ubuntu 13.04 (or Debian) – the easy way

I wanted nice, concise instructions on changing the boot order in Ubuntu 13.04, which uses Grub 2. Being a newbie focused OS however, Googling “ubuntu boot order” results in SEO blogs with lots of fluff, and then the actual instructions start out with “install package from ppa”…

What the hell, I just want to change the boot order!

Continue reading

Calling mysqldump in Python

Python is a fantastic tool to know, and despite being a beginner I find myself using it more and more for everyday tasks. Bash is great for knocking together quick scripts, but when you want to something a little more complex such as interfacing with APIs or other systems over a network, you really need a more fully-featured programming language.

The topic of this post, however, is the kind of task that bash is perfect for. Thanks to mysqldump, a database backup script can be written in a few lines and dump/restores are easily automated. So why on earth would we do this in Python?
Continue reading