👉 This post was initially written in 2007 and referred to specific software versions. When tunning your system, always consider which version you are running. Information below may be outdated. Use it at your own risk.
This post cover a creative solution for a cheap SAN and secure backup architecture solution for small-sized platform (5/10 servers). In this case study we will see how to use Network Block Devices ( nbd) and soft-raid with mdadm.
The initial platform is made of five production servers (Apache, PHP, MySQL, Postfix), one “monitoring server” running with cacti, nagios, POP3, IMAP, Subversion, Trac, Primary DNS, etc. and one “backup server” used as Secondary DNS, MySQL Slave and finally rsnapshot server. Of course I manage all this remotely with my old laptop. In this schema the Backup server can deal only 100 Go of backup.
- Backup server is a single point of failure (SPOF). If i lost my HD, I lost all my backup data
- Don’t use all my disk space on all my server
- When I need more space on my backup server, I have to rent and move to a new and more expensive server
- Backup server represent 15% of all my monthly cost (Really hudge for so small platform !)
There is a large set of options in regard to
distributed filesystem (Coda, AFS, LegionFS, MogileFS, DRDB, etc.). Though, for our current use case the need wasn’t to be able to distribute data to a large number of clients with hudge load. Instead, the requirements are:
- ability to grow backup disk space easily
- store backup data in a redundant maner and secure way (removing SPOF)
- cheap (aka not expensive)
I have lot of space unused on all my production servers. So, I choose to use Network Block Devices (NBD) with Raid 6 implementation. In this schema I currently have 4*100 Go (Raid device) that give me 200 Go of available space for my backups.
Why Raid 6 instead of Raid 5?
As I wrote earlier, my goal isn’t read/write performance, but the reliability of the system. Due to the risk of lost of a network member, I choose to use Raid 6 Implementation that let me the possibility to lose up two node at the same time.
- Distributed backup data between all the array members
- No more SPOF if I crash the backup server (This structure let me the possibility to mount the array on another server or my laptop)
- Use all available space on my servers
- Less expensive platform and easy to upgrade
- If you crash more than two node of your RAID Array you will lost all your data, this mean that you must take care when you choose to reboot a server : RAID reconstruction is not fast !
- When writing to the RAID device that will does many network i/o and probably use some CPU on your nbd-server
How to do it?
first of all, I install
nbd-server on each server with unused disk space. I choose three server from my five production servers. I’m using Ubuntu Server distrib, so I just have to do an
apt-get install nbd-server. Now I have to build an image disk on each server that will be used for the network block device. I use
dd for build a 100 Go file image then I start nbd-server on each server (node).
dd if=/dev/zero of=/home/nbd/backup.img bs=1024 count=100000000 nbd-server 10130 /home/nbd/backup.img
I encourage you to look at the man page of nbd-server for know more about all the available option.
Next, we have to setup the main server for building the RAID Array. I choose RAID 6. We previously setup 3 servers for doing 3 Network Block Devices. I need one more server for implementing my RAID 6 Array. I choose to setup a file image on the local server where I build the RAID Array, I just need to use loop device with
losetup. Then we “mount” the network block devices locally with
nbd-client (look at the man page too).
dd if=/dev/zero of=/home/nbd/backup.img bs=1024 count=100000000 losetup /dev/loop0 /home/nbd/backup.img modprobe nbd nbd-client serv1.example.com 10130 /dev/nbd0 -persist nbd-client serv2.example.com 10130 /dev/nbd1 -persist nbd-client serv3.example.com 10130 /dev/nbd2 -persist
We have all our device up on our main server, we just need to use mdadm to build our array and mount our file system to use it like any other partition.
mdadm -create /dev/md0 -raid-devices=4 -level=6 -spare-devices=0 -chunk=256 /dev/nbd /dev/loop0 mkfs.ext3 /dev/md0 -b 4096 -E stride=64 mount /dev/md0 /.snapshots
That’s all folks! I can now run
rsnapshot using my distributed RAID 6 implementation and stopping to lost my unused spaces. NB : I use a chunk size of 256k because of the RAID 6 implementation and the network constraint, performance looks better. Due to this chunk size, I use a block size of 4k and stride of 64 while building the file system (4k x 64 = 256k).******
I need more space, how can I do?
It’s easy! You just need to install a new
nbd-server on a server having enough space then load a new nbd-client on your server running the RAID Array. When it’s done, just use
mdadm to grow your array.
mdadm -grow /dev/md0 -raid-disks=5****** mdadm -add /dev/md0 /dev/nbd4
The server running the RAID Device crashed. How can I get my backup??
It’s easy! You just need to build a new device on another server or on your laptop.
mdadm -create /dev/md0 -assume-clean -raid-devices=4 -chunk=256 -level=6 /dev/nbd missing
“missing” mean that our last device of our array (/dev/loop0) is missing, the RAID Array will start in degraded mode.
I have a big external HD at home for doing a second site backup. What I have to do for copying and read it?
You will just need to copy all the file image (“backup.img”) to your external disk . Take care to stop your RAID Array before (except if you are sure to don’t have any write on your images while doing your copy). Then You will just have to load each images with
losetup locally and building a RAID Array with
To sum up
I think that this platform design is really useful for small sized (5/10 servers) platform because you get a distributed data backup solution at low cost. You don’t need to have dedicated server to manage your backup. However, you have to take care of the touchy aspect of any raid implementation, a wrong manipulation with mdadm or too many server crash could lead you to the lost of all your data. You have to keep in mind that RAID 6 need lot of time to rebuild large storage. If you plan to reboot lot of your servers, don’t forget to stop your RAID Array.
If you are interested in this kind of platform for build your SAN, I encourage you to look at the