Install DRBD on CentOS 6.9 (Active/Passive)
This is an relatively simple guide about how to install DRBD (Distributed Replicated Block Device) on two CentOS 6.9 hosts in active/passive mode as block-level data failover.
DRBD Introduction
The DRBD is a software-based, shared-nothing, replicated storage solution mirroring the content of block devices (hard disks, partitions, logical volumes etc.) between hosts.
DRBD mirrors data
- in real time. Replication occurs continuously while applications modify the data on the device.
- transparently. Applications need not be aware that the data is stored on multiple hosts.
- synchronously or asynchronously. With synchronous mirroring, applications are notified of write completions after thewrites have been carried out on all hosts. With asynchronous mirroring, applications are notified of write completions when the writes have completed locally, which usually is before they have propagated to the other hosts.
Kernel Module
DRBD’s core functionality is implemented by way of a Linux kernel module. Specifically, DRBD consititutes a driver for a virtual block device, so DRBD is situated right near the bottom of a system’s I/O stack. Because of this, DRBD is extremely flexible and versatile, which makes it a replication solution suitable for adding high availability to just about any application.
DRBD is, by definition and as mandated by the Linux kernel architecture, agnostic of the layers above it. Thus, it is impossible for DRBD to miraculously add features to upper layers that these do not possess. For example, DRBD cannot auto-detect file system corruption or add active-active clustering capability to file systems like ext3 or XFS.
User Space Administration tools
DRBD comes with a set of administration tools which communicate with the kernel module in order to configure and administer DRBD resources.
drbdadm. The high-level administration tool of the DRBD program suite. Obtains
all DRBD configuration parameters from the configuration file /etc/drbd.conf
and acts as a front-end for drbdsetup and drbdmeta. drbdadm has a
dry-run mode, invoked with the -d option, that shows which drbdsetup and
drbdmeta calls drbdadm would issue without actually calling those commands.
drbdsetup. Configures the DRBD module loaded into the kernel. All parameters
to drbdsetup must be passed on the command line. The separation between
drbdadm and drbdsetup allows for maximum flexibility. Most users will rarely
need to use drbdsetup directly, if at all.
drbdmeta. Allows to create, dump, restore, and modify DRBD meta data
structures. Like drbdsetup, most users will only rarely need to use drbdmeta
directly.
Environment Information
In this article, there are two hosts, both are installed with CentOS 6.9:
carlos- IP address: 10.0.1.133
carol- IP address: 10.0.1.233
We’ll make carlos as DRBD primary node and carol as secondary node. Most of
the actions done below should be executed on both hosts, while some are not.
Please be sure what you are doing.
Installation
First we need to upgrade the system and install some build-time dependencies:
[root@carlos ~]# yum -y update
[root@carlos ~]# yum -y install gcc make automake autoconf libxslt libxslt-devel flex rpm-build kernel-devel
Construct the RPM build directory tree. Download DRBD community source code from here. We’ll need to build the code and then package them into RPMs.
[root@carlos ~]# mkdir -p /root/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
[root@carlos ~]# wget http://www.linbit.com/downloads/drbd/8.4/drbd-8.4.7-1.tar.gz http://www.linbit.com/downloads/drbd/utils/drbd-utils-8.9.9.tar.gz
Extract source code from tarballs:
[root@carlos ~]# tar -zxvf drbd-8.4.7-1.tar.gz
[root@carlos ~]# tar -zxvf drbd-utils-8.9.9.tar.gz
[root@carlos ~]# cd drbd-8.4.7-1
[root@carlos drbd-8.4.7-1]# make km-rpm
[root@carlos drbd-8.4.7-1]# cd ../drbd-utils-8.9.9
[root@carlos drbd-utils-8.9.9]# ./configure
[root@carlos drbd-utils-8.9.9]# make rpm
[root@carlos drbd-utils-8.9.9]# cd /root/rpmbuild/RPMS/x86_64
[root@carlos x86_64]# rpm -Uvh drbd-xen-8.9.9-1.el6.x86_64.rpm drbd-udev-8.9.9-1.el6.x86_64.rpm drbd-pacemaker-8.9.9-1.el6.x86_64.rpm drbd-bash-completion-8.9.9-1.el6.x86_64.rpm drbd-utils-8.9.9-1.el6.x86_64.rpm drbd-km-*.rpm drbd-8.9.9-1.el6.x86_64.rpm
Configuration
DRBD requires the same name as its hostname, so please verify that they’re the same:
[root@carlos ~]# uname -n
carlos
Optional: edit /etc/hosts to set hostname alias for convenience:
...
10.0.1.133 carlos
10.0.1.233 carol
We’ll create a new disk partition to act as DRBD disk. Both hosts have a new
disk /dev/sdb. Let’s create a partition dev/sdb1 and use it as DRBD disk.
[root@carlos ~]# fdisk -l
Disk /dev/sda: 17.2 GB, 17179869184 bytes
255 heads, 63 sectors/track, 2088 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006ac64
Device Boot Start End Blocks Id System
/dev/sda1 * 1 64 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 64 2089 16264192 8e Linux LVM
Disk /dev/sdb: 17.2 GB, 17179869184 bytes
255 heads, 63 sectors/track, 2088 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00077967
Device Boot Start End Blocks Id System
Disk /dev/mapper/vg_carlos-lv_root: 14.9 GB, 14935916544 bytes
255 heads, 63 sectors/track, 1815 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/mapper/vg_carlos-lv_swap: 1715 MB, 1715470336 bytes
255 heads, 63 sectors/track, 208 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
[root@carlos ~]# fdisk /dev/sdb
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-2088, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-2088, default 2088):
Using default value 2088
Command (m for help): p
Disk /dev/sdb: 17.2 GB, 17179869184 bytes
255 heads, 63 sectors/track, 2088 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00077967
Device Boot Start End Blocks Id System
/dev/sdb1 1 2088 16771828+ 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Repeat the steps above on the other host.
After creating DRBD disks on both hosts, it’s time to create resource
configuration file of DRBD. Edit /etc/drbd.d/s1.res:
resource s1 {
on carlos {
device /dev/drbd0;
disk /dev/sdb1;
address 10.0.1.133:7799;
meta-disk internal;
}
on carol {
device /dev/drbd0;
disk /dev/sdb1;
address 10.0.1.233:7799;
meta-disk internal;
}
}
In DRBD, resource is the collective term that refers to all aspects of a particular replicated data set. These include:
Resource name. This can be any arbitrary, US-ASCII name not containing whitespace by which the resource is referred to.
Volumes. Any resource is a replication group consisting of one of more
volumes that share a common replication stream. DRBD ensures write fidelity
across all volumes in the resource. Volumes are numbered starting with 0, and
there may be up to 65535 volumes in one resource. A volume contains the
replicated data set, and a set of metadata for DRBD internal use.
At the drbdadm level, a volume within a resource can be addressed by the
resource name and volume number as <resource>/<volume>.
DRBD device. This is a virtual block device managed by DRBD. It has a device
major number of 147, and its minor numbers are numbered from 0 on wards, as is
customary. Each DRBD device corresponds to a volume in a resource. The
associated block device is usually named /dev/drbdX, where X is the device
minor number. DRBD also allows for user-defined block device names which must,
however, start with drbd_.
In this file we define our resource as s1, using /dev/sdb1 on both hosts as
DRBD disk /dev/drbd0. Also, the hosts’ IP addresses are shown, too. This DRBD
configuration file must be identical between two nodes. So we’ll copy it to the
other host.
[root@carlos ~]# scp /etc/drbd.d/s1.res root@carol:/etc/drbd.d/
Up and Running
Load DRBD kernel module and check if it is actually loaded:
[root@carlos ~]# modprobe drbd
[root@carlos ~]# lsmod | grep drbd
drbd 372567 4
libcrc32c 1246 1 drbd
[root@carlos ~]# drbdadm create-md s1
--== Thank you for participating in the global usage survey ==--
The server's response is:
you are the 21721th user to install this version
initializing activity log
NOT initializing bitmap
Writing meta data...
New drbd meta data block successfully created.
success
[root@carlos ~]# service drbd status
drbd driver loaded OK; device status:
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by root@carlos, 2018-03-14 17:33:07
m:res cs ro ds p mounted fstype
[root@carlos ~]# service drbd start
Starting DRBD resources: [
create res: s1
prepare disk: s1
adjust disk: s1
adjust net: s1
]
..........
***************************************************************
DRBD's startup script waits for the peer node(s) to appear.
- If this node was already a degraded cluster before the
reboot, the timeout is 0 seconds. [degr-wfc-timeout]
- If the peer was available before the reboot, the timeout
is 0 seconds. [wfc-timeout]
(These values are for resource 's1'; 0 sec -> wait forever)
To abort waiting enter 'yes' [ 163]:
.
You’ll see that DRBD is actually not ready, it’s waiting for other DRBD nodes to come up. At the same time please login to the other host using another SSH session, then start the DRBD service. And both DRBD service will started successfully.
The following command will initialize the primary server. Please notice that
this should be executed only once on primary server of your choice (e.g. on
carlos).
[root@carlos ~]# drbdadm -- --overwrite-data-of-peer primary s1
Now check DRBD status on the other host:
[root@carol ~]# service drbd status
drbd driver loaded OK; device status:
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by root@carol, 2018-03-14 17:34:22
m:res cs ro ds p mounted fstype
0:s1 SyncTarget Secondary/Primary Inconsistent/UpToDate C
... sync'ed: 1.0% (16228/16376)M
After a while the DRBD status will tell you that it’s fully synced between primary and secondary nodes. We’re done!
[root@carlos ~]# service drbd status
drbd driver loaded OK; device status:
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by root@carlos, 2018-03-14 17:33:07
m:res cs ro ds p mounted fstype
0:s1 Connected Primary/Secondary UpToDate/UpToDate C
Verification
[root@carlos ~]# mkfs.ext4 /dev/drbd0
[root@carlos ~]# mount /dev/drbd0 /mnt
[root@carlos ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_carlos-lv_root
14G 2.2G 11G 17% /
tmpfs 939M 0 939M 0% /dev/shm
/dev/sda1 477M 77M 376M 17% /boot
/dev/drbd0 16G 44M 15G 1% /mnt
[root@carlos ~]# dd if=/dev/zero of=/mnt/testfile bs=1M count=30
[root@carlos ~]# ls -l /mnt
total 30736
drwx------. 2 root root 16384 Mar 14 19:56 lost+found
-rw-r--r--. 1 root root 31457280 Mar 14 19:57 testfile
[root@carlos ~]# umount /mnt
[root@carlos ~]# drbdadm secondary s1
[root@carlos ~]# service drbd status
drbd driver loaded OK; device status:
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by root@carlos, 2018-03-14 17:33:07
m:res cs ro ds p mounted fstype
0:s1 Connected Secondary/Secondary UpToDate/UpToDate C
Now both DRBD nodes are in secondary mode.
And we switch to the other host and make it as primary node, then check its status. You can see that the host is being primary now. It means you can mount the partition and the same data will show up in the end.
[root@carol ~]# drbdadm primary s1
[root@carol ~]# service drbd status
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by root@carol, 2018-03-14 17:34:22
m:res cs ro ds p mounted fstype
0:s1 Connected Primary/Secondary UpToDate/UpToDate C
[root@carol ~]# mount /dev/drbd0 /mnt
[root@carol ~]# ls -l /mnt
total 30736
drwx------. 2 root root 16384 Mar 14 19:56 lost+found
-rw-r--r--. 1 root root 31457280 Mar 14 19:57 testfile