# [solved] Questions about setting up and maintaining a RAID5

## Jimini

Howdy,

at the moment, my fileserver consists of 2 mirrored 1,5TB-harddisks (RAID1). In the next months, I plan to replace this array with a bigger RAID5 - I'd prefer something about 4 TB of usable diskspace. I also want to be able to resize that array, if I need more space. In comparison to RAID1, which is really easy to set up an maintain, RAID5 ist a bit more complex - so I used Virtualbox to simulate such an array and to play around a bit. 

This is what I did so far:

1) created the partitions (sdb1, sdc1, sdd1, sde1)

2) created the array: mdadm --create /dev/md0 --level=raid5 --raid-devices=4 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1

3) for testing purpose, I removed one device: mdadm -f /dev/md0 /dev/sde1 && mdadm -r /dev/md0 /dev/sde1

4) then I added one device: mdadm -a /dev/md0 /dev/sdf1 and let the array reshape

5) for testing purpose, I added another device and resized the array: mdadm -a /dev/md0 /dev/sde1 && mdadm --grow /dev/md0 --raid-devices=5 (I did this with sdg1 and sdh1, too)

6) at the end, I resized /dev/md0

Is this procedure correct? Or is there anything to improve?

So this is how it looks now:

```
Phobos ~ # cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4]

md0 : active raid5 sdh1[6](S) sdg1[5] sdf1[3] sde1[4] sdd1[2] sdc1[1] sdb1[0]

      10481920 blocks level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]

     

unused devices: <none>
```

```
Phobos ~ # mdadm --detail /dev/md0

/dev/md0:                                                                                                                                                                                                                                   

        Version : 0.90                                                                                                                                                                                                                       

  Creation Time : Mon Aug 16 20:53:47 2010                                                                                                                                                                                                   

     Raid Level : raid5                                                                                                                                                                                                                     

     Array Size : 10481920 (10.00 GiB 10.73 GB)                                                                                                                                                                                             

  Used Dev Size : 2096384 (2047.59 MiB 2146.70 MB)                                                                                                                                                                                           

   Raid Devices : 6                                                                                                                                                                                                                         

  Total Devices : 7                                                                                                                                                                                                                         

Preferred Minor : 0                                                                                                                                                                                                                         

    Persistence : Superblock is persistent                                                                                                                                                                                                   

                                                                                                                                                                                                                                             

    Update Time : Mon Aug 16 22:21:49 2010                                                                                                                                                                                                   

          State : clean                                                                                                                                                                                                                     

 Active Devices : 6                                                                                                                                                                                                                         

Working Devices : 7                                                                                                                                                                                                                         

 Failed Devices : 0                                                                                                                                                                                                                         

  Spare Devices : 1                                                                                                                                                                                                                         

                                                                                                                                                                                                                                             

         Layout : left-symmetric                                                                                                                                                                                                             

     Chunk Size : 64K                                                                                                                                                                                                                       

                                                                                                                                                                                                                                             

           UUID : 21b72be7:5455667f:67dafc41:fc88d1f4                                                                                                                                                                                       

         Events : 0.368                                                                                                                                                                                                                     

                                                                                                                                                                                                                                             

    Number   Major   Minor   RaidDevice State                                                                                                                                                                                               

       0       8       17        0      active sync   /dev/sdb1                                                                                                                                                                             

       1       8       33        1      active sync   /dev/sdc1                                                                                                                                                                             

       2       8       49        2      active sync   /dev/sdd1                                                                                                                                                                             

       3       8       81        3      active sync   /dev/sdf1                                                                                                                                                                             

       4       8       65        4      active sync   /dev/sde1                                                                                                                                                                             

       5       8       97        5      active sync   /dev/sdg1                                                                                                                                                                             

                                                                                                                                                                                                                                             

       6       8      113        -      spare   /dev/sdh1  
```

Everything seems to work fine, but I get a single error message during the boot process:

```
* Starting up RAID devices ...

mdadm: /dev/sde1 has no superblock - assembly aborted                [!!]
```

What does that mean?

Best regards,

Jimini

----------

## BradN

I seem to remember there being an option in mdadm to rewrite the raid superblock on a device.  Otherwise if that doesn't work, you can completely shut down the raid and --create it again (but make sure to get the device order correct as to the order it lists out in --detail, or your data will be scrambled).

In fact one thing that might be good to test now rather than later is whether after --growing a RAID, can it be properly recreated from the devices (or has some kind of formatting changed to make that impossible or strange)?

But your method looks sound - after you resize the underlying raid, you will want to resize the filesystem residing on it (or if you use LVM, expand your physical volume and logical volume, then the filesystem)

----------

## Jimini

 *BradN wrote:*   

> I seem to remember there being an option in mdadm to rewrite the raid superblock on a device.  Otherwise if that doesn't work, you can completely shut down the raid and --create it again (but make sure to get the device order correct as to the order it lists out in --detail, or your data will be scrambled).
> 
> In fact one thing that might be good to test now rather than later is whether after --growing a RAID, can it be properly recreated from the devices (or has some kind of formatting changed to make that impossible or strange)?

 

That's a good idea, I'll check that later.

 *Quote:*   

> But your method looks sound - after you resize the underlying raid, you will want to resize the filesystem residing on it (or if you use LVM, expand your physical volume and logical volume, then the filesystem)

 

Of course, I unmounted /dev/md0, ran fsck -f and then resized it - worked as it was supposed to :)

The thing is, that I'm really confused about that missing superblock on /dev/sde1. I reproduced the problem - everything works fine, until I set one device as faulty and remove it from the array. After readding it, I assemble the array:

```
Phobos ~ # mdadm --stop /dev/md0 

mdadm: stopped /dev/md0

Phobos ~ # mdadm --assemble /dev/md0

mdadm: /dev/md0 has been started with 4 drives.
```

Everything fine so far.

But when I repeat the last command:

```
Phobos ~ # mdadm --assemble /dev/md0

mdadm: /dev/sde1 has no superblock - assembly aborted
```

Is it really necessary to create partitions on the single raid-devices (sdb1, sdc1...)?

Do I have to add ARRAY /dev/md0 devices=/dev/sdb1,/dev/sdc1,/dev/sdd1,/dev/sde1 to /etc/mdadm.conf ?

Best regards,

Jimini

----------

## BradN

I think -a is different than --assemble, but I don't have a machine with mdadm here to test with at the moment.

Is there any reason you're trying to run --assemble twice?  If you only get that error trying to assemble a raid that's already running, I wouldn't worry about it.

As far as I know, I've never used /etc/mdadm.conf and never had a reason to - my raids are being assembled when the kernel starts up before init scripts even begin to run.

As to the partitions, it's necessary to use partitions (with the correct partition ID, FE if I remember) if you want the raids autoassembled by the kernel, in addition all the partitions should have superblocks.

----------

## Jimini

 *BradN wrote:*   

> I think -a is different than --assemble, but I don't have a machine with mdadm here to test with at the moment.

 

Correct, -a = --assemble, -a = --add

 *Quote:*   

> Is there any reason you're trying to run --assemble twice?  If you only get that error trying to assemble a raid that's already running, I wouldn't worry about it.

 

Hehe, of course there Is no reason - I only wanted to reproduce the error (mdadm: /dev/sde1 has no superblock - assembly aborted).

 *Quote:*   

> As far as I know, I've never used /etc/mdadm.conf and never had a reason to - my raids are being assembled when the kernel starts up before init scripts even begin to run.

 

For testing purposes, I commented the mentioned line in /etc/mdadm.conf out, the error still occured.

 *Quote:*   

> As to the partitions, it's necessary to use partitions (with the correct partition ID, FE if I remember) if you want the raids autoassembled by the kernel, in addition all the partitions should have superblocks.

 

The strange thing is, that I created ext3-filesystems on every partition. I'll set it up from scratch tomorrow, I hope then I can find out what the reason for this error is :\

Thanks so far!

Best regards,

Jimini

----------

## Jimini

Ok ok, I could't wait and re-created the whole array now.

1) formatted all partitions and re-created the array, everything was fine

2) Reboot and same error as above occured (no superblock on /dev/sde1)

3) stopped the array

4) assembled the array: mdadm: /dev/md0 has been started with 4 drives.

=> Summary so far: the raid works fine. At boot, I get the error that sde1 has no superblock. If I stop and then assemble the array, I do not get this error, only if I assemble the array again.

Best regards,

Jimini

----------

## BradN

Hmm, you wouldn't have had to format the individual partitions, simply setting the partition type and putting them in a raid with mdadm should suffice.

Maybe the kernel is starting your RAID at boot time (it should do this if the partition types are set), and then your init scripts are trying to start it again (maybe that's causing the error) - is your raid running correctly when you first log in and check with mdadm --detail?  Also you could check dmesg output and see if the kernel was assembling your raid at boot time.

----------

## Jimini

 *BradN wrote:*   

> Hmm, you wouldn't have had to format the individual partitions, simply setting the partition type and putting them in a raid with mdadm should suffice.

 

Then I just misunderstood that - I was also wondering, why I should format these partitions.

 *Quote:*   

> Maybe the kernel is starting your RAID at boot time (it should do this if the partition types are set), and then your init scripts are trying to start it again (maybe that's causing the error) - is your raid running correctly when you first log in and check with mdadm --detail?  Also you could check dmesg output and see if the kernel was assembling your raid at boot time.

 

The error message appeared when the kernel recognized the array during the boot process. But everything worked correctly (as I posted the output of /proc/mdstat and mdadm --detail /dev/md0 above).

I rebuilt my kernel and deactivated CONFIG_MD_AUTODETECT. Now the error disappeared - as I expected. What is the other component, that recognizes the array?

Best regards,

Jimini

----------

## BradN

The kernel itself can assemble raid devices during its startup (this happens before init scripts even run, and requires the partition types to be set, and is useful for root on raid without an initrd), or your boot scripts can do it after the kernel's finished starting (which is where you were seeing the error).

Honestly I'm surprised the boot scripts aren't smart enough to avoid trying to start an already-started raid.

----------

## Jimini

Ah, that makes sense. 

There is one thing I am wondering about:

```
Phobos ~ # df -h

Filesystem            Size  Used Avail Use% Mounted on

[...]

/dev/md0              6.0G  141M  5.5G   3% /home

Phobos ~ # du -sh /home/*

16K     /home/lost+found
```

That doesn't really make sense, does it?

Apart from that, everything works fine so long. I hope, that building such an array on a real system works without bigger problems as well...!

Best regards,

Jimini

----------

## BradN

Hmm, the empty space usage is probably a problem for you to take up with your filesystem - modern ones will allocate a journal of some size even on an empty filesystem, and often other filesystem structures as well.  But, that usage (450MB+) does seem way high for a 6GB partition, at least for the filesystems I've been using (reiserfs, ext3 mostly).

What filesystem are you using on there, and does it behave the same when you format a loopback file with it?

example,

dd if=/dev/zero of=mytest.fs bs=1M count=6000

mk???fs mytest.fs

mount -o loop mytest.fs /mnt/test

df -h

umount /mnt/test

rm mytest.fs

Edit:  Just thought of another possibility, some filesystems reserve a certain percentage of space for root, maybe that's automatically taken off the available space summary.  I suppose to test real space you might want to fill a file until the disk is full.

cat /dev/zero > spacetest

ls -l spacetest

rm spacetest

----------

## Jimini

I use ext3, which is mounted as /home.

As you recommended, I filled up the space on the array:

```
df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/md0              6.0G  141M  5.5G   3% /home

cat /dev/zero > /home/spacetest

cat: write error: No space left on device

du -sh /home/spacetest 

5.8G    /home/spacetest

df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/md0              6.0G  6.0G     0 100% /home
```

Afterwars, I removed the file and grew my array up to 12 GiB (7 devices).

```
df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/md0               12G  144M   12G   2% /home

cat /dev/zero > /home/spacetest

cat: write error: No space left on device

du -sh /home/spacetest 

12G     /home/spacetest

df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/md0               12G   12G     0 100% /home
```

Whatever uses 141M respectively 144M, this should be no problem regarding the size of the array I am planning. What I am still wondering about is the following: 

```
df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/md0              6.0G  141M  5.5G   3% /home
```

For me, 6,0G - 141M makes more than 5,5G ;)

Best regards,

Jimini

----------

## Genone

 *Jimini wrote:*   

> What I am still wondering about is the following: 
> 
> ```
> df -h
> 
> ...

 

The missing ~300 MB are the 5% of space reserved for root. You can change (or even eliminate) it with mke2fs when creating the filesystem or later with tune2fs.

----------

## Jimini

Great, thanks for the explanation. 300M don't hurt at all, I just wanted to clarify that.

Best regards,

Jimini

----------

