# mdadm --assemble RAID 5 after reboot mid-reshape fails

## djeebus

Hey all, I've been a long time gentoo user, and any help I've needed has always been answered via old forum posts or the incredible gentoo-wiki (may she rest in peace, long live the gentoo-wiki!) but I'm afraid I painted myself into a corner that I can't be helped out of. As a last ditch effort I seek assistance. Help! =)

My gentoo box has a series of RAID 5 devices created from 4x250GB partitions. I recently bought a new 1TB hard drive (/dev/sdd) and wanted to add a single 250GB partition to the rest. After days of rebuilding, my system was growing unresponsive to lvm commands, so (like a retard) I decided to reboot it, forgetting that it was mid rebuild of /dev/md2. Once I finished rebooting, 3 of the 4 raid devices refused to reassemble. I successfully ressurected 2 of them, but the last one is proving difficult.

Here's the error I get when rebuilding the whole thing (/dev/sdd3 is the new device):

```

mediabox ~ # mdadm /dev/md2 --assemble /dev/hdb1 /dev/sda3 /dev/sdc3 /dev/sde2 /dev/sdd3

mdadm: superblock on /dev/sdd3 doesn't match others - assembly aborted

```

Here's /proc/mdstat

```

mediabox ~ # cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4]

md1 : active raid5 sdd2[4] sdf2[3] sde1[2] sdc2[1] sda2[0]

      976590848 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]

      bitmap: 0/233 pages [0KB], 512KB chunk

md3 : active raid5 sdd4[4] sdg1[2] sdc4[1] sdb1[3] sda4[0]

      976590848 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]

      bitmap: 0/233 pages [0KB], 512KB chunk

md0 : active raid5 sdd1[4] sdf1[2] sdc1[1] sdb2[3] sda1[0]

      976558592 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]

      bitmap: 0/233 pages [0KB], 512KB chunk

unused devices: <none>

```

Here's the info on a good partition in /dev/md2 (hdb1, sda3, sdc3 and sde2 all give similar information)

```

mediabox ~ # mdadm -E /dev/sda3

/dev/sda3:

          Magic : a92b4efc

        Version : 0.91.00

           UUID : b671a325:53eadbe2:cac1de09:ac9d17da

  Creation Time : Tue Jul  1 09:15:56 2008

     Raid Level : raid5

  Used Dev Size : 244147712 (232.84 GiB 250.01 GB)

     Array Size : 976590848 (931.35 GiB 1000.03 GB)

   Raid Devices : 5

  Total Devices : 5

Preferred Minor : 2

  Reshape pos'n : 0

  Delta Devices : 1 (4->5)

    Update Time : Sat Jan 24 12:57:08 2009

          State : clean

Internal Bitmap : present

 Active Devices : 5

Working Devices : 5

 Failed Devices : 0

  Spare Devices : 0

       Checksum : cb5be982 - correct

         Events : 10

         Layout : left-symmetric

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State

this     1       8        3        1      active sync   /dev/sda3

   0     0       3       65        0      active sync   /dev/hdb1

   1     1       8        3        1      active sync   /dev/sda3

   2     2       8       35        2      active sync   /dev/sdc3

   3     3       8       66        3      active sync   /dev/sde2

   4     4       8       51        4      active sync   /dev/sdd3

```

Here's the weird stuff - this is what /dev/sdd4 looks like

```

mediabox ~ # mdadm -E /dev/sdd3

/dev/sdd3:

          Magic : a92b4efc

        Version : 0.91.00

           UUID : 02f93cbb:be04cc34:206f9650:56bf46d0

  Creation Time : Wed Jul  2 16:11:04 2008

     Raid Level : raid5

  Used Dev Size : 244147712 (232.84 GiB 250.01 GB)

     Array Size : 976590848 (931.35 GiB 1000.03 GB)

   Raid Devices : 5

  Total Devices : 6

Preferred Minor : 3

  Reshape pos'n : 230850048 (220.16 GiB 236.39 GB)

  Delta Devices : 1 (4->5)

    Update Time : Sat Jan 24 07:40:04 2009

          State : clean

Internal Bitmap : present

 Active Devices : 4

Working Devices : 6

 Failed Devices : 1

  Spare Devices : 2

       Checksum : 9d5cd528 - correct

         Events : 297684

         Layout : left-symmetric

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State

this     5       8       51        5      spare   /dev/sdd3

   0     0       8        4        0      active sync   /dev/sda4        # actually /dev/md3

   1     1       8       36        1      active sync   /dev/sdc4        # actually /dev/md3

   2     2       8       81        2      active sync   /dev/sdf1        # actually /dev/md3

   3     3       8       17        3      active sync   /dev/sdb1        # actually /dev/md3

   4     4       0        0        4      faulty removed

   5     5       8       51        5      spare   /dev/sdd3

   6     6       8       52        6      spare   /dev/sdd4

```

and here's a device in /dev/md3, /dev/sda4

```

mediabox ~ # mdadm -E /dev/sda4

/dev/sda4:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : 02f93cbb:be04cc34:206f9650:56bf46d0

  Creation Time : Wed Jul  2 16:11:04 2008

     Raid Level : raid5

  Used Dev Size : 244147712 (232.84 GiB 250.01 GB)

     Array Size : 976590848 (931.35 GiB 1000.03 GB)

   Raid Devices : 5

  Total Devices : 5

Preferred Minor : 3

    Update Time : Mon Jan 26 15:45:42 2009

          State : clean

Internal Bitmap : present

 Active Devices : 5

Working Devices : 5

 Failed Devices : 0

  Spare Devices : 0

       Checksum : 81ddb60e - correct

         Events : 421664

         Layout : left-symmetric

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State

this     0       8        4        0      active sync   /dev/sda4

   0     0       8        4        0      active sync   /dev/sda4

   1     1       8       36        1      active sync   /dev/sdc4

   2     2       8       97        2      active sync   /dev/sdg1

   3     3       8       17        3      active sync   /dev/sdb1

   4     4       8       52        4      active sync   /dev/sdd4

```

Any ideas how to get /dev/md2 functioning again? All the problems seem to surround /dev/sdd3. I'm 99% positive that I added the partitions to their proper raid arrays and didn't double book any, but the evidence seems to be calling me a lier. 

Thanks in advance for any help!!

----------

## HeissFuss

Can you assemble md2 without specifying the sdd3 device?  If it assembles degraded, you can then remove the /dev/sdd3 device and re-add it.  It should sync back up then.

----------

