Category Archives: IT

IT related posts, technical stuff

How not to troubleshoot an unexplained server reboot

We asked our provider to investigate why one of our servers rebooted last night. In the process they accidentally rebooted it again… this is root’s bash_history just before it happened, note line 971:

  954  2011-08-17_15:10:39 sar -q
  955  2011-08-17_15:10:59 sar -q|less
  956  2011-08-17_15:11:09 sar -r|less
  957  2011-08-17_15:11:24 last -x|less
  958  2011-08-17_15:11:49 history |grep -i shutd
  959  2011-08-17_15:11:21 history
  960  2011-08-17_15:11:32 date
  961  2011-08-17_15:13:52 cd /var/log/
  962  2011-08-17_15:13:53 ls
  963  2011-08-17_15:13:54 ls -lah
  964  2011-08-17_15:13:58 less audit/
  965  2011-08-17_15:14:04 less audit/audit.log
  966  2011-08-17_15:14:25 less secure
  967  2011-08-17_15:15:15 grep -v nagios secure | less
  968  2011-08-17_15:16:11 dmesg
  969  2011-08-17_15:17:57 sar -r
  970  2011-08-17_15:18:19 dmesg
  971  2011-08-17_15:18:30 dmesg | reboot
  972  2011-08-17_16:20:20 [LOGOUT]: xxxx     pts/2        2011-08-17 15:27 (xxx.xxx.xxx.xxx)

Continue reading

Upgrading my E4300

Two years is quite a long time for me to keep a computer, but this Dell has lasted surprisingly well. It lags a bit in the 3D graphics department, but CPU wise it’s still perfectly acceptable and 4gb of ram is still a decent amount. And the 13″ form factor is perfect for my needs.

One of the major advantages of business models is the long product life cycle, which means an ample supply of spare parts and accessories. In the case of the Dell E series the docking stations and some of the other accessories are compatible with all other E series laptops. The greater amount of resources that go into the design and the higher build quality are also apparent – once you’ve had a good business laptop it’s hard to go back to disposable consumer machines!

Recently I’ve had a few gripes with it however:

  • The lack of bluetooth has become an inconvenience as the E4300 is chronically short of USB ports. Two really isn’t enough, and I want to get a bluetooth mouse to free up one of them.
  • I moved to the UK last year and the USA keyboard layout lacks the keys for € and £ symbols.
  • The fan started to buzz loudly now and again about six months ago. It was relatively infrequent and only annoyed me perhaps once a week, but it was a sure sign that the bearings were on the way out.

When the fan finally outright stalled I was able to get it to move again after a rest, but I decided it was time to kill 3 pigs with 1 bird and give the old girl a new lease of life.

I picked up the following parts on ebay: Continue reading

Keyboard layout switching to USA in Ubuntu 11.04

I recently upgraded the keyboard in my Dell E4300 from a standard USA model to a backlit UK model. All went great however I noticed that the keyboard layout kept switching back to the USA layout. It seemed to happen after every reboot, and also randomly in the middle of the session.

There’s even a bug in launchpad about it which I commented on: https://bugs.launchpad.net/ubuntu/+bug/762111

There were two factors causing the layout to switch and the USA layout to be reinstated.

Firstly when logging in the session was set to the USA layout. To fix this logout, select your user name and then look for the keyboard symbol down the bottom of the screen.

The second feature causing seemingly random switches during sessions was the “Separate layout for each window” option in Keyboard Preferences (Keyboard Preferences is in system settings and the option is on the layout tab).

If you deactivate this, remove the USA layout, apply system wide and make sure the GDM session is set to the correct layout on login you shouldn’t have any more problems with randomly switching keyboard layouts!

Getting the most out of Terminator

Terminator in action

Terminator is a must-have tool for Linux administrators. It’s a terminal emulator that supports multiple terminals via tabs, but also by dividing up its window with horizontal and vertical splits.

The user documentation is a bit sparse, in fact what you see in the man page is what you get. In this post I’ll take you through the features that I think make Terminator the best terminal emulator around.
Continue reading

Setting up a secure Ubuntu LAMP server

Disclaimer: This article is provided for your information only, and simply following this guide will not make your server “secure”. As the server administrator you are ultimately responsible for its security!

Intro

Having recently been through the process of setting up a few Ubuntu LAMP (Linux, Apache, MySQL, PHP) servers lately I thought I’d make an article out of my notes and provide a starters guide to setting up the LAMP stack on Ubuntu.

It goes without saying that the only truly secure computer is one with no network connection, no ports or input devices and is locked in a bank vault, but such a machine is not terribly useful. Regretfully, compromises must be made to allow functionality! Besides presuming insecurity, there are a lot of things you can do to make your server more secure and keep out the vast majority of would-be hackers running port scans, meta-exploit scripts and dictionary attacks.
Continue reading

A quick look at Unity in Ubuntu 11.04

I usually jump on the latest Ubuntu release before it hits the final release stage, but this time it was with a bit more trepidation than usual. You see they’ve replaced the shell with a completely new one – Unity. And to say that not everyone likes it would be a minor understatement.

The good news is that Unity is undoubtedly a step forward in practical user interface terms. The classic Gnome panel is really showing its age, and doesn’t lend itself to wide screen formats due to having panels on both the top and bottom of the screen. SUSE addressed this by created a start menu like launcher, but it was always a bit clunky and barely an improvement over Windows XP.

So how is Unity different? It is perhaps more informative to talk about how it is similar…
Continue reading

Recovering a RAID5 mdadm array with two failed devices

Update 1/11/2019

If you’ve reached this article by Googling how to recover a RAID array, I suggest you don’t use this guide. The Linux RAID wiki has much more correct, complete, and authoritative information. In retrospect I was very lucky not to destroy the data, but this article is an interesting account of how I did it.

Update

Before reading this article you should know that it is now quite old and there is a better method – ‘mdadm –assemble –force’ (it may have been there all along). This will try to assemble the array by marking previously failed drives as good. From the man page:

If mdadm cannot find enough working devices to start the array, but can find some devices that are recorded as having failed, then it will mark those devices as working so that the array can be started.

I would however strongly suggest that you first disconnect the drive that failed first. If you need to discover which device failed first, or assemble doesn’t work and you need to manually recreate the array, then read on.

I found myself in an interesting situation with my parents home server today (Ubuntu 10.04). Hardware wise it’s not the best setup – two of the drives are in an external enclose connected with eSATA cables. I did encourage Dad to buy a proper enclosure, but was unsuccessful. This is a demonstration of why eSATA is a very bad idea for RAID devices.

What happened was that one of the cables had been bumped, disconnecting one of the drives. Thus the array was running in a degraded state for over a month – not good. Anyway I noticed this when logging in one day to fix something else. The device wasn’t visible so I told Dad to check the cable, but unfortunately when he went to secure the cable, he must have somehow disconnected the another one. This caused a second drive to fail so the array immediately stopped.

Despite having no hardware failure, the situation is similar to someone replacing the wrong drive in a raid array. Recovering it was an interesting experience, so here I’ve documented the process.
Continue reading

Quick analysis of a phishing attack

Twice in three days I have received emails purporting to be from ASB and ANZ Bank. Both are New Zealand banks, and the fact that I’ve received two of them clearly indicates that my email address is on a spam database somewhere and geographically tagged New Zealand. Easy enough – my .co.nz domain uses it as the registration address, and it has a New Zealand residential address on it.

I see these all the time, but the execution of this particular attack struck me as unusually slick however, hence the blog post.

The email

The ANZ email subject was “Please remove your Online Banking Limitation! Last warning!”, whereas the ASB email was titled “Online banking suspension warning!”. Both are clearly designed to panic the user into clicking the link and entering their banking details. The ANZ subject has a hint of ESOL and the grammar in the emails is sub-standard, I suspect the origin is a country where English is not the first language.
Continue reading

Splitting files with dd

We have an ESXi box hosted with Rackspace, it took a bit of pushing to get them to install ESXi it in the first place as they tried to get us to use their cloud offering. But this is a staging environment and we want something dedicated on hardware we control so we can get an idea of performance without other people’s workloads muddying the water.

Anyway, I’ve been having a bit of fun getting our server template uploaded to it, which is only 11GB compressed – not exactly large, but apparently large enough to be inconvenient.

In my experience the datastore upload tool in the vSphere client frequently fails on large files. In this case I was getting the “Failed to log into NFC server” error, which is probably due to a requisite port not being open. I didn’t like that tool anyway, move on.

The trusty-but-slow scp method was also failing however. Uploads would start but consistently stall at about the 1GB mark. Not sure if it’s a buffer or something getting filled in dropbear (which is designed to be a lightweight ssh server and really shouldn’t need to deal with files this large), but Googling didn’t turn up much.
Continue reading