Disclaimer: This article is provided for your information only, and simply following this guide will not make your server “secure”. As the server administrator you are ultimately responsible for its security!

Intro

Having recently been through the process of setting up a few Ubuntu LAMP (Linux, Apache, MySQL, PHP) servers lately I thought I’d make an article out of my notes and provide a starters guide to setting up the LAMP stack on Ubuntu.

It goes without saying that the only truly secure computer is one with no network connection, no ports or input devices and is locked in a bank vault, but such a machine is not terribly useful. Regretfully, compromises must be made to allow functionality! Besides presuming insecurity, there are a lot of things you can do to make your server more secure and keep out the vast majority of would-be hackers running port scans, meta-exploit scripts and dictionary attacks.

The target audience here is the casual administrator with some Linux knowledge. I assume you know how to edit config files, what SSH & sudo are, how to change your password on the command line and a vague idea of how to setup user accounts. Having said that, nothing here is terribly complicated and I’d encourage anyone interested in LAMP to give it go. I’ve tried to include all the details but may have missed some things – feel free to leave a comment if you find you get stuck.

The guide assumes you’re using a virtual private server, such as those provided by the “cloud computing” services of Amazon and Rackspace (I use inverted commas because cloud computing is currently one of the most misused terms in the computing field), but there’s absolutely no reason why you couldn’t apply most of this to a local physical server. There are in that case more factors to consider such as local network security, and additional configuration steps to setup the network interfaces, DNS servers etc, but that is beyond the scope of this article.

A small plug…

I’ve signed up to the Rackspace Cloud affiliate program, so if you like this article and do decide to go with Rackspace I’d be very grateful if you used the links above or the banner at the bottom before signing up. :)

I’m using Ubuntu 10.10 for this guide as it’s the latest image on Rackspace. The changes between server releases tend to be quite subtle and the concepts are fairly generic as far as Ubuntu is concerned so you should be able to apply this to 11.04 and later versions.

One final note – while using root is a very bad habit for day to day sysadmin work, when setting up a server most commands needs root privileges so I do tend to run a root shell. Thus sudo commands are omitted, and you should assume root unless stated otherwise (you can gain a root shell from an account that has the appropriate access rights with sudo -i).

By the end of the article you should have a functioning LAMP server which is secure by any reasonable measure. In a future article I will cover the installation of WordPress using this as a base.

Before you begin

It is a very good idea to have a valid DNS record for your server, one that you don’t use for anything else. Often this is overlooked by novice admins, and can cause problems for software such as mail servers which assume the system’s host name is a valid record. As an example, say you want to create a server to host a website called www.yourdomain.com. Many would simply setup the server’s host name as www.yourdomain.com and be done with it, or worse still “yourdomain”, which is not a valid record. However this can cause headaches if at some point you want to move the web site to a different server, as you’d have to change the original server’s name to something else.

Another example is email – you wouldn’t email [email protected] now would you ;)

It also creates “ambiguous” and messy configurations if you add additional web sites to the host. Why is blog.apples.com hosted on www.oranges.com? Thus I strongly recommend creating a new subdomain that is used to refer to your server and only your server, for example server1.yourdomain.com, and not any web site or service. Any additional records such as “blog.yourdomain.com” or “www.yourdomain.com” can be CNAME records to server1.yourdomain.com in DNS (basically an alias).

Pedantic tech note: CNAME records actually cause an extra DNS round-trip the first time the client visits your site because it has to lookup first the A record and then the CNAME record. The penalty is tiny unless DNS is being painfully slow, but if performance is critical all DNS records should be A records pointing directly to the IP address.

Initial Setup

I’m assuming you already have the OS installed – all hosts do this for you. After commissioning a new image on Rackspace Cloud you should receive the root password via email.

By default Ubuntu server has you setup an administrator account at the time of install and does not set a root password, which prevents the root account from being used for console and ssh access. This is a very good thing as the majority of script kiddies will usually try to hit the “root” account. On cloud environments we don’t get that option, so the first thing you need to do is reset the root password and create an administrator account for yourself to use.

Hopefully you know how to set passwords:

• passwd root

This password should be long and complex. To generate secure passwords I use a little tool called pwgen, which is available in the Ubuntu repositories (apt-get install pwgen):
e.g.:

Hopefully you are never going to use this password again so you may as well make it long and secure. Rackspace provide the facility to reset the root password if you ever need it, and if you really want you can store it in a password database such as the free (and open source) KeePassX.

Creating an account is a little more complex (but only a little), as we need to ensure it has enough rights to use sudo:

useradd -m -G sudo,adm -s /bin/bash charlie

An explanation of the options here:

• -m is for make home directory. This also copies the contents of /etc/skel so if you want to make a template for your users alter the files there.
• -G sudo,adm is the supplementary groups to add the account to. By default the primary group is the same as the username which is fine for most purposes. sudo should be obvious, it allows the use of the sudo command as defined in /etc/sudoers. adm is the administrators group, and by default many log files in /var/log are readable by this group. I find it is useful to assign additional permissions to this group where I want to avoid using sudo for a trivial task.
• -s /bin/bash simply sets the shell to bash. The default is /bin/sh which is not as powerful as bash and lacks many of its features.
• charlie – the username :-)

Set a password for the account as before:

• passwd charlie

Logout (exit or Ctrl+D) and reconnect using your new credentials.

If you’re using a Linux desktop to connect, now is a good time to setup an SSH key and copy the public key to your new server. This allows you to login without entering a password each time, and gives you the option of disabling password authentication (which, assuming you have passworded protected your private key, makes your server more secure). SSH keys are not the focus of the article so I’ll gloss over the process here and you can Google for more info if needed:

Generate key (only do this once): ssh-keygen -t dsa

Authorize key on server: ssh-copyid -i ~/.ssh/id_dsa.pub charlie@<server address>

If you’re unfortunate enough to be on Windows, Putty also supports SSH keys, but I don’t use it so don’t ask me how!

Once you’ve logged into your server again set your hostname if you didn’t enter it when the machine was setup. This is simply accomplished by editing /etc/hostname, and rebooting.

Install all updates:

• apt-get update && apt-get --yes upgrade

If you get warnings about perl failing to set the locale:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LANG = "en_GB.UTF-8"
    are supported and installed on your system.

… you need to setup the language (in my case en_GB). Asumming your language is one of the variants of English, installing language-pack-en-base should do it:

• apt-get install language-pack-en-base

Hardening SSH

In this section we will disable root logins and restrict access to only our admin user. Disabling root instantly defeats 90% of dictionary attacks, and restricting access to your own user means that accounts created for other purposes don’t unintentionally get ssh access (many hackers attempt to guess passwords for “apache”, “mysql” etc).

The sshd configuration file is /etc/ssh/sshd_config, so open this in your favourite editor, e.g. vim /etc/ssh/sshd_config (if you don’t know how to use vim nano is easier, but you may have to install it first). Note there is also an ssh_config, but that is for the ssh client and does not relate to the server daemon.

First change the line that says PermitRootLogin yes to PermitRootLogin no. Note that this makes the Rackspace root password reset ineffective as far as SSH goes, although you can still login via their Java console.

Next we want to explicitly allow users so that any additional users are not granted ssh access by default. A sensible group to restrict it to is the adm group, as you can’t really do much administrating without ssh access, so add the following lines to /etc/ssh/sshd_config:

AllowUsers charlie
AllowGroups adm

The lists are space delimited so if you need to allow more users or groups just add a space between them.

If you intend to grant non-administrators ssh access to your server, you may want to grant the “users” group access as well. Just remember to add them to the users group when creating, or you can a user to a group after creation with:

• usermod -a -G users charlie

The -a here stands for append, were we to omit this option it would overwrite his list of supplementary groups, and charlie would be a member of users but no longer in the sudo or adm groups.

NTP Setup

NTP stands for network time protocol, and ntpd is the daemon keeps the time in sync. This is important on any server! By default it uses an ubuntu.com time server which is fine by me, so to set this up do apt-get install ntp, enable it with update-rc.d ntp enable and you’re done. If you prefer you can find more time servers at pool.ntp.org.

Firewall

Ufw, or “uncomplicated firewall”, makes this part easy. In the past you’d have to wrestle with complicated iptables rules, but ufw generates these rules for you and makes managing a command line firewall as easy as it could possibly be.

To install it and allow ports 80, 22 and 443:

apt-get install ufw
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow 22/tcp
ufw enable

Be sure you have enabled port 22 before enabling the firewall or you may find you won’t be able to ssh (when doing “ufw enable” from an ssh session it gives you a warning to this affect).

To delete a rule, do exactly the same but with delete in front, i.e. ufw delete allow 22/tcp.

If you want to restrict access to certain IP addresses (great if you have a static IP and don’t require access away from home), you can do ufw allow proto tcp from <IP address> to any port 22.

If you do this, remember to delete the allow 22/tcp rule!

Show current rules with ufw status.

Apache

A web server isn’t very useful without a web server:

• apt-get install apache2

Security of Apache (and MySQL) is a HUGE topic, so the key here is keeping things simple. Ubuntu makes sensible choices by default and releases security fixes quickly so by using the distribution packages you’re quite secure out of the box. Thus what’s important is what you do afterwards.

You need to create the directories that are going to serve your site. I like to put server content in /srv as that’s what this folder is intended for (see this for a more authoritative stance on the topic). You certainly don’t have to, many people use /var/www. /home/sites is also quite common, and the most incorrect of all. So I’m going to recommend /srv. A good format to use is /srv/servicename/sitename, e.g. /srv/www/example.com or /srv/ftp/example.com.

Make any directories required (I am going to create for a fictional site called charlie.example.com), and assign permissions to adm so you can edit with having to sudo:

• mkdir -p /srv/www/charlie.example.com
• chown -R root.adm /srv/www
• chmod -R 775 /srv/www

Echo a test file so that you can see when it’s working:

• echo "Charlie was here" > /srv/www/charlie.example.com/index.html

Next we need to setup the Apache config. By default the Ubuntu package sets up a default site which simply says “it works”.

You can use the default site and place content in /var/www, but it is much cleaner (and proper) to create your own config. But there’s no harm in using the default site as a starting point. After making sure the default site is working, remove it with a2dissite default. The files in sites-enabled are simply links to files in sites-available, so we’re not losing the default config by doing this.

Copy the config and edit:

• cd /etc/apache2/sites-available
• cp default charlie.example.com
• vim charlie.example.com

Below is an appropriate config which you can copy-paste and edit:

<VirtualHost *:80>

        ServerAlias charlie.example.com
        ServerAdmin [email protected]

        DocumentRoot /srv/www/charlie.example.com
        <Directory />
                Options FollowSymLinks
                AllowOverride None
        </Directory>
        <Directory /srv/www/charlie.example.com>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>

        ErrorLog ${APACHE_LOG_DIR}/error.log

        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn

        CustomLog ${APACHE_LOG_DIR}/charlie.example.com_access.log combined

</VirtualHost>

• AllowOverride is set to none because .htaccess files are a security risk. If an attacker is able to drop a .htaccess file in your web server through a security oversight in a PHP script or other exploit, they can overwrite the options and potentially access other parts of the file system (e.g. allow symlinks via htaccess, create a symlink to other files on your system and download those). This is one of the reasons why you NEVER run apache as root. Some software (such as WordPress), does make use of .htaccess files for rewrite rules and the like, however there is nothing to stop you from moving these rules from the .htaccess file to the virtual host configuration, and that is what I suggest you do.
• I’ve removed the cgi-bin entries because most Apache installations will use a module rather than cgi binaries. PHP has mod_php, python has mod_wsgi. Using modules is I think the simplest way to get started, but high-performance sites these days will frequently use proxied daemons such as php-fpm or uWSGI.

After modifying the config to suit and saving the config (make sure you have altered ServerAdmin, DocumentRoot and CustomLog), create the symlink to sites-enabled to enable the site:

• cd /etc/apache2/sites-enabled
• ln -s ../sites-available/charlie.example.com ./charlie.example.com

After altering an apache configuration it’s always a good idea to do a configtest with apachectl configtest. This ensures that you’re not restarting your web server with a bad apache config.

Restart apache with /etc/init.d/apache2 restart, and hit your domain/IP address in a web browser. You should see the contents of the file you echo’d earlier:

Password protecting a site

While setting up a site I like to have a password set so I can see the results of my work but others can not. Note that Apache’s basic authentication is NOT a secure method for daily use because unless you’re speaking to the server in https the password will be transmitted over the wire in plain text. It really is “basic” authentication.

One way of accomplishing this is to use a .htaccess file (assuming AllowOverride AuthConfig is set in the vhost config), but when you have control over the vhost configuration it makes much more sense to put it there. Basically, don’t use htaccess files when you don’t have to as they are both slower and less secure.

First of all we need to setup a htpasswd file. This file contains a username and a password hash to compare against. Create the file with htpasswd file with a new user by typing htpasswd -c /srv/www/htpasswd charlie. Enter a password when prompted, but DON’T use the same password as your shell account. Any machine between you and your server could potentially sniff the password and the last thing you want to be broadcasting is your ssh password!

Next we need to setup the AuthConfig of the vhost, so vim /srv/apache2/sites-available/charlie.example.com. Paste the following in the <Directory /srv/www/charlie.example.com> tag, below the line that says “allow from all”. Make sure it is above the </Directory> line.

AuthName "Restricted" 
AuthType Basic 
AuthUserFile /srv/www/htpasswd 
AuthGroupFile /dev/null 
require valid-user

Restart apache and hit your site again (you may need to refresh).

Voila!

Enable it to start on boot by typing rcupdate-rc.d apache2 enable.

The basic Apache configuration is now done, but of course we haven’t got anything useful on it yet. In a future article I will cover the installation of WordPress, which is quick and easy – especially if you use the subversion method.

MySQL

This is the very basic initial setup, I’m not going to cover creating databases and assigning permissions – the application defines what these should be. What I will cover is the basic installation of MySQL on Ubuntu, and the running of the mysql_secure_installaton script which should give you into a secure platform from which to build on.

Firstly, apt-get install mysql-server. The Ubuntu package prompts you to set a root password, so set it and store it in your password database. This password should be different to any of your system accounts, because it and other mysql passwords could potentially be exposed by an SQL injection attack (as happened in the recent Sony debacle). However it is also a password you will probably need to use now and again so bear that in mind as well.

The mysql secure installation script is installed as part of the Ubuntu/Debian package and is installed in /usr/bin, so run it with /usr/bin/mysql_secure_installation. Basically answer yes to everything except changing root password (because you’ve already set it).

Once the script has completed, login to your database with mysql -u root -p.

As a test, run show databases;. Only information_schema and mysql should be present – these two databases are essential for mysql to function, and you should never have to touch them.

Exit with exit.

As the mysql service has been converted to an upstart job, it is already enabled at boot by default. Upstart is still a bit new to me, but the script which controls the service is /etc/init/mysql.conf. To start, stop and restart the service use service mysql [start/stop/restart].

MySQL Supplementary notes

It’s worth noting that you can login to mysql with mysq -u username -pPassword (no space between -p and your password). However you shouldn’t do this because by doing so your password is recorded in you bash_history file (/home/username/.bash_history). Were this file to be exposed somehow, your password would be there to see in plain text. As a way around this on Ubuntu you can add a space in front of the command to omit the command from the history, however this feature of bash isn’t enabled by default on many distributions, and generally it is not good practice to specify passwords on the command line anyway.

Back in the day it was recommended to disable mysql network access so that only local services could access mysql. Personally I don’t consider this necessary when we have a firewall in place. If other applications (e.g. another web server) need to access your mysql server over the network, open the port to specific IP addresses and not the general internet.

Don’t use PHPMyAdmin…

PHPMyAdmin is notorious for security flaws and is frequently probed for on web servers. If you must install PHPMyAdmin, install it in its own vhost to a non-standard location and restrict access by IP address and/or htpasswd (with https). And keep it up to date!

A much better approach to PHPMyAdmin is to use a local management tool (such as MySQL Administrator) combined with a ssh tunnel. A quick guide I found on the subject is at here – this covers setting up an SSH tunnel with the free Putty SSH client and logging in with MySQL Administrator. And here’s another one with a bit more detail.

PHP

Setting up PHP is straightforward and usually not a lot of configuration is needed:

• apt-get install php5

This automatically symlinks the module config in /etc/apache/mods-enabled and sets up the config files in /etc/php5. Note that php5/apache2/php.ini is the config for the apache module and php5/cli/php.ini is the config for the binary when called from the command line.

After restarting apache, php should already be functioning on your site. To test it, drop a file called test.php in /srv/www/charlie.example.com with the following contents:

<?php
/* I am a comment, should not see me */
die('testing!');

If you see only “testing!” then it’s a success, if you see the text as you entered it then there is something wrong with your configuration.

Postfix

A mail server isn’t essential, but it is nice to have one setup so you can receive backup logs and output from cron jobs. By having postfix and mailutils available, scripts and services running on the machine can notify when things go wrong.

Before installing postfix however, you’ll save yourself some pain if you server is setup with a valid DNS records as its hostname – so make sure you’ve followed that section above.

Start the install with apt-get install postfix mailutils. The ubuntu/debian postfix package prompts you for a configuration template – in this case we want to select internet site. The system mailname should be the FQDN (fully qualified domain name) of your host – e.g. server1.yourdomain.com. This is important, as it affects the headers of any messages sent by the server.

But basically after that you are done. Send a test message to yourself with the command echo "test message" | mail -s "testing" [email protected] and check your email!

Fail2ban

Fail2ban is in a nutshell a daemon that scans your logs for suspicious activity and blocks IP addresses acting suspiciously at the firewall level. You should be aware that organizations such as universities frequently NAT a lot of people behind one IP address, so you could end up unintentionally blocking legitimate users with this approach. But since fail2ban only blocks addresses from the service they were attacking the odds of it impacting legitimate users are rather low unless you enable the apache module.

The main reason to use fail2ban is that it makes dictionary attacks against your server ineffective, as the attacker gets blocked after a predefined number of failed attempts.

Installing it is very straightforward, and the configuration is a little more fiddly than it needs to be but simply installing it will protect your ssh server without any additional configuration. To install: apt-get install fail2ban.

To configure it you first need to cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local. Then open /etc/fail2ban/jail.local in your favourite editor.

The only change you need to make is destemail = root@localhost, change this to the email address you want to recieve notifications at. Make sure you’ve setup an MTA (mail transport agent) such as postfix (as configured above).

You may not want to receive emails every time an IP address gets blocked but I recommend you do for a while, if only to get a feel for how often your server gets probed.

By default only the ssh jail is enabled. If you have password protected apache sites you may want to enable the apache jail as well. Make sure it is pointing at the logs for your vhost.

Once you’re done restart fail2ban with /etc/init.d/fail2ban restart, and you should receive an email with the subject “[Fail2Ban] ssh: started”.

The reboot test

After setting up any server you need to do at least one reboot to ensure everything comes up as designed. If something doesn’t start automatically:

• mysql – check the init script, /etc/init/mysql. Try stating it manually with service mysql start, and if it fails examine /var/log/mysql/error.log for any error messages.
• apache – run update-rc.d apache2 enable, /etc/init.d/apache2 restart, and check /var/log/apache2/error.log

Keeping safe

Always install security updates. While serious exploits are rare for the software installed here, they do come up now and again so you should always keep up with patching. Subscribe to the ubuntu-security-announce mailing list and that way you’ll be notified of any security alerts.

The biggest security hole is probably going to be whatever PHP application you run on Apache, therefore a secure apache configuration is essential. Disabling .htaccess helps with this, but make sure the www-data user can’t do too much on your system – it should only have rights to directories serving web sites. If running WordPress make sure you keep up to date.

Rackspace Cloud

I use Rackspace Cloud Servers (London) for this site, and have so far been happy with the service. It seems to have been born out of Slicehost, as Rackspace acquired them a while back and the features are nearly identical. The console they use for access to your VMs is unfortunately the same – awful. Emergency use only!

The DNS hosting feature is nice but a bit limiting as you can’t create TEXT records (I wanted to add SPF records but I guess I’ll either have to self-host or find a third party DNS provider). On the whole the interface is not as well developed as Amazon EC2, but it seems to be a solid alternative and is well worth considering for your hosting needs.