EggXpert

This post is geared towards people who know enough about basic linux commands to get by, and have read the Storage FAQ's RAID post.

Required:

• At least 3 sata drives, of the same size, that you either don't have any data on or don't mind blowing away anything on them.
  
• mdadm installed (hint: "apt-get install mdadm")

Where to start?

Best place to start is to get your drive locations. Start by doing a simple ls command.

ls /dev/sd*

I would recommend NOT using partitions for your arrays. What do I mean by this? I mean use /dev/sdb not /dev/sdb1. Why? Because if you are doing any type of maintenance work on them, it makes things easier to manage. This is because you can now say 'only devices that don't have partitions are in the array', and it helps in situations where you accidently fat finger in adding/removing drives and you forgot to don't specify the partition--especially in adding an existing one to an array.

So if you already have created partitions, get into fdisk utility and clear them out:

ls /dev/sd(letter)

d

w

If you need help, type 'm'. Do this for every disk... Note: the moment you do the d / w commands, ANY DATA ON THOSE DRIVES IS GONE. Granted, you can get them back with some specific tools, but be VERY careful when dealing with fdisk. I'd even go so far as to recommend unplugging all but the drives you want to create the array with and boot from a live CD. One can't be too careful!

Now write down all the drives you'll be using. I'm going to assume you have four drives called /dev/sdb, /dev/sdc, /dev/sdd, and /dev/sde. Note: I would NOT recommend making a RAID5 array any larger than 8 drives.

Creating the array

Here is the easy part. We're going to create a 4 drive RAID5 array called /dev/md0. To do this, you can type in short hand:

mdadm -C -l5 -n4 /dev/md0 /dev/sd[bcde]

Let's disect this real quick. The "-C" part means to "create" or "rebuild", the "-l5" (lowcase L) means "layout 5" or "RAID 5", the "-n4" tells it to expect 4 drives in the array, the next part (/dev/md0) is giving it a device name, and the last part is listing all the devices. The brackets, for those of you familiar with bash commands, makes it "/dev/sdb /dev/sdc /dev/sdd /dev/sde". As you can see, it's much easier to do the brackets.

Watching it build

Assuming there were no issues, it should say 'mdadm: /dev/md0 array started' or something to that affect. To watch the progress, you have two options:

mdadm --detail /dev/md0

or

cat /proc/mdstat

I prefer the latter, and I use the 'watch' program like so:

watch -n 1 cat /proc/mdstat

To get out of the watch, CTRL + C it.

Now what?

Technically, right now, you can throw your new RAID5 device (/dev/md0) into LVM or put a filesystem on it... but I'd highly recommend waiting until it's fully built. I'd also highly recommend that you take a look at the other mdadm commands. Some helpful ones to get you started in the right direction:

mdadm --manage --help  (-arfS are some good ones)

mdadm --assemble --help (-s is only one I really use)

mdadm --misc --help (-QDE and --zero-superblock I use... but DON'T DO THE --zero-superblock unless you want to delete the array information from a drive)

mdadm --grow --help (-lpn)

mdadm --monitor --help (-mcfl)

Example: If you reboot and for whatever reason it doesn't start up automatically you can run:

 mdadm --assemble -s

Note that most commands require that you specify the array first and then the devices that you want to change. I highly recommend consulting the man pages on the command you wish to run before you try anything out (hint: "man mdadm" and type '/' to search for keywords).

 Note #2: superblock is a term you might hear while reading up on the man pages of mdadm. Basically it's like a file that's written to all the drives in your array that says "I'm part of ___ array that's a RAID __ and I'm number __ in line of the array". Superblock allows you to physically move your drives to another linux machine and 'import' them in with "mdadm --assemble -s" command. As long as your superblock is good on all of your drives, you can port them to any machine and start the array at any time. Pretty neat huh?

Tip

After you've created your array, I would HIGHLY recommend writing down the order in which you created your array. In fact, I'd do exactly this (which I do now):

mdadm --detail /dev/md0

hdparm -i /dev/sd[bcdef...n] | grep Serial

You'll probably recognize the first command, but the second command might look new to you. What this tells you is what a particular device's (example: /dev/sdb) make/model/serial is. This is just incase you accidently swap SATA cables for whatever reason (hint: the /dev/sde drive might not be the same physical drive when you swap it's SATA cable with another drive). So having these outputs in a file and stored, say, in your backup hard drive (NOT ON YOUR ARRAY) and--say--as a 'draft' email in your Gmail account (lolz), would make rebuilding your array much easier. I made the mistake of getting one drive out of order when one of my superblocks got corrupted and I had to rebuild the array. Bye bye data.

Ending Notes 

I'll try and create a runbook tomorrow on how to properly align/use LVM and (my favorite filesystem) XFS with a RAID5 array.

Post up if you have any questions/issues.