I've been using DRBD for quite some time now. When I started as a Linux system administrator at my first real job, DRBD was this RAID1 high available storage thing that was magic to me. In combination with PiranHA, which retired ten years ago, I built a setup I demonstrated to my colleague as a high availability setup.
Although this was ten years ago, the people at LINBIT haven't been sitting still. DRBD 9 came to life in 2015, but I had never had any experience with its new features other than "Just using it". When I started looking into Kubernetes, I started looking deeper in DRBD.
If you've used DRBD 9, you probably know it can, contrary to DRBD 8, replicate to more than two nodes. Replication means you can use DRBD in a cluster of 3 nodes and not decide which two nodes can run which workload. The downside of this is the increased disk usage, and an even more significant problem is the scalability and flexibility problems when you're reaching ten or even 100 nodes. You're not going to replicate all the data over all the 100 nodes.
Flexibility is where diskless DRBD comes in. Once you've installed DRBD on every node of your cluster, your disks aren't bound to the metal casing they reside in. You can expose a single disk on one node to another without replicating the data. Let's dive into the technical stuff now!
Installing virtual machines is manually boring. Use Hyper-V on your Windows workstation to quickly get some virtual machines up and running. Soon I'll talk about doing the same thing with Terraform on AWS, Azure and GCP!
So I've set up three virtual machines with ubuntu 20.04. I did the first things first. I placed my keys on the server and updated them to the latest packages. Make a snapshot; it saves you time.
Something I wanted to do differently in this blog is compiling DRBD myself. Usually, I would use the LINBIT PPA repo, and of course, you can do so. However, time will pass, this post gets old, versions change, and its accuracy will rot.
Let's prepare the servers for DRBD! The commands used to compile are so simple.
Do this on all three nodes. DRBD should be running, but nothing to be created yet. First, let's create two persistent disks. You can do diskless with even a single node, but let's test the availability if one of them fails in the next blog post.
Create a DRBD resource file on all three nodes.
Prepare the persistent disks on both nodes with disks.
Adjust the DRBD resources on all nodes and check their status.
You'll see that one node says Secondary/Diskless. Let's make this disk primary, create a filesystem and mount it.
Now we can write files to this filesystem without having the disk available here. Let's dive into the reliability of node failures in the next blog post.