Highly available Samba on Ubuntu 8.04

This is something I worked on over the last few days. I was confronted with a particular network infrastructure, where a central Samba fileserver stores all important business data. Of course there is a working backup strategy, however – that server being a single point of failure – the situation wasn’t quite optimal. Usual failure recovery times where as long as 2 days over which the critical data was only partly available. Thus the idea was born to move Samba onto a high availability setup. It’s implemented as a two-node cluster using Ubuntu 8.04 Server Edition, DRBD and Hearbeat.

In this article I will guide you through all the necessary steps to implement such sytem.

Introduction

I’ve already mentioned that the two important pieces of software for this project are DRBD and heartbeat. DRBD (Distributed Replicated Block Device) is comparable to RAID 1 in that it replicates write operations on a block device to another block device, thereby introducing redundancy. In contrast to a RAID array however, DRBD works over a TCP/IP network so your drives don’t have to be attached to the same machine.
But that’s only for the storage part. There’s not much use in having a highly available storage device when all running services are down in case of a failure. This is where heartbeat comes into play.
Heartbeat is a high-availability solution that detects server outages and can automatically restart the affected services on another node of the cluster. In fact heartbeat has out of the box support for DRBD devices, so using the two together is a popular choice. Heartbeat uses a so called heartbeat-network to send small special messages (“hearbeats”) to the other cluster nodes which basically is interpreted as “I’m still up and running, so there’s no need for you to take over any of my services”.

For this project we need two physical seperate computers with two NICs each and one free partition that should be about the same size on each of the nodes. Throughout the article I will use the following conventions:

  • master-server – (host)name of the master node, that is the server that should normally run our samba and DRBD services
  • slave-server – (host)name of the slave node that takes over if the other one goes down
  • eth0 – NIC connected to the normal intranet
  • eth1 – connection to the heartbeat network
  • 10.0.0.1 – fixed IP of the master-node in the company intranet (eth0)
  • 10.0.0.2 – same for the slave-node
  • 10.0.0.5 – this is a highly available IP that can switch between the nodes; usual clients (such as SMB clients) should only use this IP to contact the services
  • 10.0.1.1 – master-servers IP in the heartbeat network
  • 10.0.1.1 – same for slave-server

Of course you’ll have to adapt these values in your specific environment.

Basic setup

The following steps are nearly identical for both nodes; there’s only one tiny difference in the network interface configuration.
First of all let’s install the two packages we use as a base – DRBD and Heartbeat.

$ apt-get install heartbeat drbd8-utils

We don’t have to install the DRBD driver as it is already part of the standard Ubuntu 8.04 Server installation. Now we have to check that both nodes have their heartbeat network interfaces configured correctly. As stated earlier, on my setup I use eth1 for the heartbeat interfaces.

As we want a static ip address automatically assigned to the interface on each startup, we have to add the following lines to /etc/network/interfaces

auto eth1
iface eth1 inet static
  address 10.1.1.1
  netmask 255.255.255.0

If there already was a section for eth1, you’ll have to replace that. Note that 10.1.1.1 is the ip address for the master node. Change that value to 10.1.1.2 on the slave. The next step is to add the nodes in /etc/hosts so they can be properly resolved.

10.1.1.1    master-server.domain.tld  master-server
10.1.1.2    slave-server.domain.tld  slave-server

DRBD

Both nodes

/etc/drbd.conf is where all DRBD configuration is done. It’s important to keep this file identical on both nodes, in our example we will use the following content:

resource r0 {
  # protocol to use; C is the the safest variant
  protocol C;

  startup {
    # timeout (in seconds) for the connection on startup
    wfc-timeout       120;
    # timeout (in seconds) for the connection on startup
    # after detection of data inconsistencies ("degraded mode")
    degr-wfc-timeout  120;
  }

  syncer {
    # maximum bandwidth to use for this resource
    rate 100M;
  }

  on master-server {
    ### options for master-server ###
    # name of the allocated blockdevice
    device     /dev/drbd0;
    # underlying blockdevice
    disk       /dev/sdc1;
    # address and port to use for the synchronisation
    # here we use the heartbeat network
    address    10.1.1.1:7788;
    # where to store DRBD metadata; here it's on the underlying device itself
    meta-disk  internal;
  }

  on slave-server {
    ### options for slave-server ###
    device     /dev/drbd0;
    disk       /dev/sdc1;
    address    10.1.1.2:7788;
    meta-disk  internal;
  }
}

Replace master-server and slave-server with the appropriate hostnames of your setup.
Now we’ll load the kernel module and add DRBD to the default runlevels:

modprobe drbd
echo 'drbd' >> /etc/modules
update-rc.d drbd defaults

Setting up a new DRBD resource requires explicit ceation of it, so let’s do that now.

/etc/init.d/drbd restart
drbdadm create-md r0
drbdadm up r0

Ignore errors and warnigs, as we will check the correct operation with

$ cat /proc/drbd

If everything is up and running, that command should print out something like this:

version: 8.0.11 (api:86/proto:86)
GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12 11:56:43
 0: cs:Connected st:Secondary/Secondary ds:Inconsistent/Inconsistent C r---
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
        resync: used:0/31 hits:0 misses:0 starving:0 dirty:0 changed:0
        act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0

Master node

Initially DRBD doesn’t know which of the nodes should be primary, that’s why they are marked as “Inconsistent”. We have to force one of the nodes to be primary – in our case that will be master-server.

$ drbdadm -- -o primary r0

The synchronization should then start immediately, you can check its progress with cat /proc/drbd. After that we are now ready to format the new DRBD block device:

$ mkfs.ext3 /dev/drbd0

Mounting works just like with any other device, do a mount /dev/drbd0 /data if you want to test the setup, but do not add an entry into /etc/fstab, because heartbeat will mount the device later on.

Heartbeat

Both nodes

First of all put the following in /etc/ha.d/authkeys (this file must be chmod 600):

auth 1
1 crc

In /etc/ha.d/haresources we will list all the resources that we want to be highly available (amongst others those are the DRBD device and the Samba service).

master-server 10.0.0.10/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 samba

This line says, that master-server is the preferred node for all of the listed resources

  • the IP address 10.0.0.10 with netask /24 on interface eth0 (that interface has to be brought up by other means, typically by listing it in /etc/network/interfaces)
  • the DRBD device r0
  • a filesystem-mount; namely /dev/dbd0 will be mounted to /data
  • the Samba service

All ressources on one line will be started one after another as it’s assumed that each one depends on the ressources listed before it. When the master node fails, the slave will take over all of those ressources and if we choose to do an automatic failback, master-server will then again take over when it’s recovered later on.

Master node

Into /etc/ha.d/ha.cf put the following content:

auto_failback   on
ucast           eth1 10.1.1.2
node            master-server slave-server
keepalive       2
deadtime        8
  • auto_failback – whether the master node will become master after it becomes available again
  • ucast – on which interface and ip to reach the partner node
  • node – simply lists all nodes of the cluster
  • keepalive – how often to send a heartbeat signal (every 2 seconds in this case)
  • deadtime – time after which the other node will be assumed to be dead

Slave node

/etc/ha.d/ha.cf is quite similar we only have to change the ucast address.

auto_failback   on
ucast           eth1 10.1.1.1
node            master-server slave-server
keepalive       2
deadtime        8

Samba

The following steps apply to both nodes.
To make Samba highly available, we have to recall, that heartbeat should be in charge of starting / stopping the service. Therefor we have to remove Samba from the normal, runlevel-based, state management. Let’s do that first:

$ update-rc.d -f mysql remove

Now we have to put heartbeat in charge of controlling Samba. Open up /etc/ha.d/haresources again and replace the line we put there in the DRBD part with the following one:

master-server 10.0.0.10/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 samba

See that little “samba” at the end? That tells heartbeat to take care of Samba in case of a resource takeover. You could actually add other things like mysql there, too. Heartbeat just takes any command and invokes /etc/init.d/ANY_COMMAND start|stop.
Now we have to make sure that both nodes use the same netbios name so that shares stay available after a resource takeover. As a bonus I like to restrict Sambda to only listen on the highly available ip address. Here is what we have to add to /etc/samba/smb.conf in the [global] section:

[global]
  ...
  netbios name = hv-samba
  interfaces = 10.0.0.10/24
  bind interfaces only = true
  ...

The final step is just an easy example for a Samba share that uses the DRBD device, which is mounted at /data. Add the following lines to /etc/samba/smb.conf:

[myshare]
  path = /data
  guest ok = yes
  read only = no
  browsable = yes

A few notes

It’s certainly useful to be notified somehow if a resource takeover takes place. I will demonstate here how that can be achieved via email.

Email notification

To get an email notification if master-server fails (and slave-server takes over the resources), we will use the simple smtp client “msmtp” which provides a sendmail interface. The version provided in the standard hardy repositories has a bug when used with TLS certificates, thus I installed it from the backport repository.
We’ll also need mailx, a non-interactive command line mailclient.

$ wget http://de.archive.ubuntu.com/ubuntu/pool/universe/m/msmtp/msmtp_1.4.16-1~hardy2_i386.deb
$ wget http://de.archive.ubuntu.com/ubuntu/pool/universe/m/msmtp/msmtp-mta_1.4.16-1~hardy2_all.deb
$ dpkg -i msmtp_1.4.16-1~hardy2_i386.deb msmtp-mta_1.4.16-1~hardy2_all.deb
$ apt-get -f update
$ apt-get install mailx

Now we need to edit /etc/msmtprc. You’ll have to replace all example values with some real smtp account credentials.

defaults
tls on
tls_certcheck off
tls_min_dh_prime_bits 512

account myaccount
host smtp.myprovider.tld
from myaddress@myprovider.tld
auth on
user SMTP_USER
password SMTP_PASSWORD

account default : myaccount

See the msmtp manpage for more details.

At last, append the following line to /etc/ha.d/haresources. If you don’t want to be emailed on a takeover by master-server, do this step only on slave-server.

master-server MailTo::someadmin@domain.tld::Subject

Contact

Alexander Gitter
contact at agitter net