# how to recover from faulty raid5 array?

## remix

i had a power outtage, server went down, and hasn't bee nthe same since.

```
# mdadm --assemble --scan

mdadm: /dev/sdd1 has no superblock - assembly aborted

mdadm: /dev/sdd2 has no superblock - assembly aborted

mdadm: /dev/sdd3 has no superblock - assembly aborted

mdadm: /dev/sdd5 has no superblock - assembly aborted

mdadm: /dev/sdd6 has no superblock - assembly aborted

mdadm: /dev/sdd7 has no superblock - assembly aborted

```

```
# mdadm --misc --examine /dev/sdd1

/dev/sdd1:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : 278ea2e2:b89984e2:33bc4bc5:59021a3e

  Creation Time : Sat Mar  6 00:43:43 2010

     Raid Level : raid5

  Used Dev Size : 34186176 (32.60 GiB 35.01 GB)

     Array Size : 102558528 (97.81 GiB 105.02 GB)

   Raid Devices : 4

  Total Devices : 4

Preferred Minor : 1

    Update Time : Mon Feb  4 18:54:39 2013

          State : clean

 Active Devices : 2

Working Devices : 2

 Failed Devices : 2

  Spare Devices : 0

       Checksum : b4dca8e1 - correct

         Events : 963131

         Layout : left-symmetric

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State

this     3       8       49        3      active sync   /dev/sdd1

   0     0       8        1        0      active sync   /dev/sda1

   1     1       0        0        1      faulty removed

   2     2       0        0        2      faulty removed

   3     3       8       49        3      active sync   /dev/sdd1
```

```
# mdadm --misc --examine /dev/sdd2

/dev/sdd2:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : 64205578:cc795cb2:fbf2ea5a:12bfd232 (local to host OptimusPrime)

  Creation Time : Mon Mar  8 22:07:40 2010

     Raid Level : raid5

  Used Dev Size : 258799040 (246.81 GiB 265.01 GB)

     Array Size : 776397120 (740.43 GiB 795.03 GB)

   Raid Devices : 4

  Total Devices : 4

Preferred Minor : 2

    Update Time : Mon Feb  4 18:54:39 2013

          State : clean

 Active Devices : 2

Working Devices : 2

 Failed Devices : 2

  Spare Devices : 0

       Checksum : 948c4a52 - correct

         Events : 386

         Layout : left-symmetric

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State

this     3       8       50        3      active sync   /dev/sdd2

   0     0       8        2        0      active sync   /dev/sda2

   1     1       0        0        1      faulty removed

   2     2       0        0        2      faulty removed

   3     3       8       50        3      active sync   /dev/sdd2
```

sdd3, sdd5, sdd6, and sdd7 are all about the same.

```
# mdadm --misc --examine /dev/sda1

/dev/sda1:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : 278ea2e2:b89984e2:33bc4bc5:59021a3e

  Creation Time : Sat Mar  6 00:43:43 2010

     Raid Level : raid5

  Used Dev Size : 34186176 (32.60 GiB 35.01 GB)

     Array Size : 102558528 (97.81 GiB 105.02 GB)

   Raid Devices : 4

  Total Devices : 4

Preferred Minor : 1

    Update Time : Mon Feb  4 18:54:39 2013

          State : clean

 Active Devices : 2

Working Devices : 2

 Failed Devices : 2

  Spare Devices : 0

       Checksum : b4dca8ab - correct

         Events : 963131

         Layout : left-symmetric

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State

this     0       8        1        0      active sync   /dev/sda1

   0     0       8        1        0      active sync   /dev/sda1

   1     1       0        0        1      faulty removed

   2     2       0        0        2      faulty removed

   3     3       8       49        3      active sync   /dev/sdd1
```

sda2, sda3, sda5, sda6, and sda7 are all about the same.

```
# mdadm --misc --examine /dev/sdb1

/dev/sdb1:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : 278ea2e2:b89984e2:33bc4bc5:59021a3e

  Creation Time : Sat Mar  6 00:43:43 2010

     Raid Level : raid5

  Used Dev Size : 34186176 (32.60 GiB 35.01 GB)

     Array Size : 102558528 (97.81 GiB 105.02 GB)

   Raid Devices : 4

  Total Devices : 4

Preferred Minor : 1

    Update Time : Sat Feb  2 23:55:04 2013

          State : clean

 Active Devices : 3

Working Devices : 3

 Failed Devices : 1

  Spare Devices : 0

       Checksum : b4da4a83 - correct

         Events : 962928

         Layout : left-symmetric

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State

this     1       8       17        1      active sync   /dev/sdb1

   0     0       8        1        0      active sync   /dev/sda1

   1     1       8       17        1      active sync   /dev/sdb1

   2     2       0        0        2      faulty removed

   3     3       8       49        3      active sync   /dev/sdd1
```

sdb2, sdb3, sdb5, sdb6, and sdb7 are all about the same.

```
# mdadm --misc --examine /dev/sdc1

/dev/sdc1:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : 278ea2e2:b89984e2:33bc4bc5:59021a3e

  Creation Time : Sat Mar  6 00:43:43 2010

     Raid Level : raid5

  Used Dev Size : 34186176 (32.60 GiB 35.01 GB)

     Array Size : 102558528 (97.81 GiB 105.02 GB)

   Raid Devices : 4

  Total Devices : 4

Preferred Minor : 1

    Update Time : Wed Jan 16 04:47:26 2013

          State : clean

 Active Devices : 4

Working Devices : 4

 Failed Devices : 0

  Spare Devices : 0

       Checksum : b4a5715a - correct

         Events : 21

         Layout : left-symmetric

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State

this     2       8       33        2      active sync   /dev/sdc1

   0     0       8        1        0      active sync   /dev/sda1

   1     1       8       17        1      active sync   /dev/sdb1

   2     2       8       33        2      active sync   /dev/sdc1

   3     3       8       49        3      active sync   /dev/sdd1
```

sdc2, sdc3, sdc5, sdc6, and sdc7 are all about the same, i'm confused to why sdb and sdc are in state "active sync" here.

is there a way to rebuild my raid partitions?

----------

## eccerr0r

What bothers me is that the array is marked "Clean" which means something was done with the array after the power outage - they should be marked "dirty."

Usually when a disk gets kicked from an array, the kicked disk, if it couldn't be written to at the time of kicking, will still have a superblock on it.  If it comes back on a power cycle - that's why sometimes they show up as a dirty disk with all other array members missing.

You may need to recreate the superblock on all disks again, but I don't know how you created the array to begin with.  Or it's probably possible to hack the missing disks and fake it so that it thinks it's clean.  But since it looks more than one disk was removed from a raid 5, you may be facing data loss even if you could get it to reassemble.

This is one of the reasons why I want to get a UPS for my 4 disk RAID5 :( and RAID is not backup...

----------

## remix

somehow i got mdadm to see the third disk, so then it assembled. i looked and it seems that the fourth is in fact toast. so i've ordered a replacement, and am slowly moving all my backups onto amazon glacier.

i guess i got lucky this time.

the way i got mdadm to see it is to (i had to repeatedly do this) run mdadm -S /dev/md1  to stop it, then reassemble /dev/md1 again.

or stop all /dev/mdX devices, and the reassemble again.

this process took over 8 hours across 2 days cause i'm not very knowledgeable or skilled in this.

----------

