# [SOLVED] md RAID 1 partition keeps being added as spare

## c00l.wave

I have a md raid 1 on 2 devices, /dev/sdb2 and /dev/sdc2. Since I wanted to reformat to a different filesystem I decided it would be a good idea to take advantage of my raid, degrade it by removing one device, setting up a new one with a missing drive, rsyncing everything and someday readd the other device to restore raid 1 protection.

I disabled the sdc2 by marking it faulty (it contains the root fs of my running system) and then removed it. I repartitioned sdc etc. but noticed I won't get far as long as I have Windows sitting on sdb thus making it impossible to retain the physical disk layout (more than 3+1 partitions would have needed to be primary). I restored the old layout (with some reboots in between) and now I'm trying to readd sdc2 to my raid. On real disk failures I used to re-add it by simply calling mdadm -a /dev/md0 /dev/sdc2. This doesn't work however - the partition always shows up as spare.

```
# mdadm --detail /dev/md0

/dev/md0:                                  

        Version : 0.90                     

  Creation Time : Mon Apr 14 02:55:34 2008 

     Raid Level : raid1                    

     Array Size : 524286528 (500.00 GiB 536.87 GB)

  Used Dev Size : 524286528 (500.00 GiB 536.87 GB)

   Raid Devices : 2                               

  Total Devices : 1                               

Preferred Minor : 0

    Persistence : Superblock is persistent

    Update Time : Fri Dec 25 18:38:37 2009

          State : clean, degraded

 Active Devices : 1

Working Devices : 1

 Failed Devices : 0

  Spare Devices : 0

           UUID : a4b24559:cd98291a:a5d98e37:1e73f3cc

         Events : 0.314641

    Number   Major   Minor   RaidDevice State

       0       8       18        0      active sync   /dev/sdb2

       1       0        0        1      removed

# mdadm --examine /dev/sdb2

/dev/sdb2:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : a4b24559:cd98291a:a5d98e37:1e73f3cc

  Creation Time : Mon Apr 14 02:55:34 2008

     Raid Level : raid1

  Used Dev Size : 524286528 (500.00 GiB 536.87 GB)

     Array Size : 524286528 (500.00 GiB 536.87 GB)

   Raid Devices : 2

  Total Devices : 1

Preferred Minor : 0

    Update Time : Fri Dec 25 18:38:44 2009

          State : clean

 Active Devices : 1

Working Devices : 1

 Failed Devices : 1

  Spare Devices : 0

       Checksum : 924478c2 - correct

         Events : 314651

      Number   Major   Minor   RaidDevice State

this     0       8       18        0      active sync   /dev/sdb2

   0     0       8       18        0      active sync   /dev/sdb2

   1     1       0        0        1      faulty removed

```

Whether running mdadm --re-add /dev/md0 /dev/sdc2 or mdadm --add /dev/md0 /dev/sdc2, I get this output after it:

```

# mdadm --re-add /dev/md0 /dev/sdc2

mdadm: re-added /dev/sdc2                            

# mdadm --detail /dev/md0

/dev/md0:                                  

        Version : 0.90                     

  Creation Time : Mon Apr 14 02:55:34 2008 

     Raid Level : raid1                    

     Array Size : 524286528 (500.00 GiB 536.87 GB)

  Used Dev Size : 524286528 (500.00 GiB 536.87 GB)

   Raid Devices : 2                               

  Total Devices : 2                               

Preferred Minor : 0                               

    Persistence : Superblock is persistent        

    Update Time : Fri Dec 25 18:40:56 2009

          State : clean, degraded, recovering

 Active Devices : 1

Working Devices : 2

 Failed Devices : 0

  Spare Devices : 1

 Rebuild Status : 0% complete

           UUID : a4b24559:cd98291a:a5d98e37:1e73f3cc

         Events : 0.314798

    Number   Major   Minor   RaidDevice State

       0       8       18        0      active sync   /dev/sdb2

       2       8       34        1      spare rebuilding   /dev/sdc2

# mdadm --examine /dev/sdb2

/dev/sdb2:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : a4b24559:cd98291a:a5d98e37:1e73f3cc

  Creation Time : Mon Apr 14 02:55:34 2008

     Raid Level : raid1

  Used Dev Size : 524286528 (500.00 GiB 536.87 GB)

     Array Size : 524286528 (500.00 GiB 536.87 GB)

   Raid Devices : 2

  Total Devices : 2

Preferred Minor : 0

    Update Time : Fri Dec 25 18:41:08 2009

          State : active

 Active Devices : 1

Working Devices : 2

 Failed Devices : 1

  Spare Devices : 1

       Checksum : 923fad09 - correct

         Events : 314813

      Number   Major   Minor   RaidDevice State

this     0       8       18        0      active sync   /dev/sdb2

   0     0       8       18        0      active sync   /dev/sdb2

   1     1       0        0        1      faulty removed

   2     2       8       34        2      spare   /dev/sdc2

```

How do I get rid of that device 1 displaying as "faulty removed" and replace it with device 2 marked "spare"?

I read there should be a way running mdadm --create over the active RAID but this does not seem right to me... I don't want to create a RAID, I want to remove a removed device from its superblock. Recreating a raid over its own seems risky to me. Isn't a simple working restoration possible?!

----------

## Mad Merlin

I think you have to remove that device.

Say you're back at the beginning, with 2 devices in RAID 1. First mark one device as failed, and then you can remove it. I didn't test this, but something like so:

```

mdadm --fail /dev/sdb2

mdadm --remove /dev/sdb2

```

In this case, you might need to do one of:

```

mdadm --remove failed

#or

mdadm --remove detached

```

These are special modes that remove all failed or detached devices.

----------

## c00l.wave

It seems like mdadm cannot remove a removed device (no way to address it other than "failed" and that doesn't work). I still get only an additional spare:

```
# mdadm /dev/md0 --remove failed --add /dev/sdc2

mdadm: re-added /dev/sdc2

# mdadm --detail /dev/md0

/dev/md0:

        Version : 0.90

  Creation Time : Mon Apr 14 02:55:34 2008

     Raid Level : raid1

     Array Size : 524286528 (500.00 GiB 536.87 GB)

  Used Dev Size : 524286528 (500.00 GiB 536.87 GB)

   Raid Devices : 2

  Total Devices : 2

Preferred Minor : 0

    Persistence : Superblock is persistent

    Update Time : Fri Dec 25 20:07:28 2009

          State : clean, degraded, recovering

 Active Devices : 1

Working Devices : 2

 Failed Devices : 0

  Spare Devices : 1

 Rebuild Status : 0% complete

           UUID : a4b24559:cd98291a:a5d98e37:1e73f3cc

         Events : 0.319105

    Number   Major   Minor   RaidDevice State

       0       8       18        0      active sync   /dev/sdb2

       2       8       34        1      spare rebuilding   /dev/sdc2
```

----------

## Mad Merlin

 *c00l.wave wrote:*   

> It seems like mdadm cannot remove a removed device (no way to address it other than "failed" and that doesn't work). I still get only an additional spare:
> 
> ```
> # mdadm /dev/md0 --remove failed --add /dev/sdc2
> 
> ...

 

No, I think you got it, it's resyncing the array. Give it some time and it should be back to normal again. Also, try checking /proc/mdstat.

----------

## c00l.wave

Thanks, you were right. That was pretty confusing - the spare becomes a normal raid device once the sync has finished and then automatically replaces the old one completely (being enumerated as "1" instead of "2"). Either I didn't notice it previously or there was some change during the last year or so that introduced that behaviour. My last resync has been a while ago...

```
# mdadm --detail /dev/md0                                                          

/dev/md0:                                                                                            

        Version : 0.90                                                                               

  Creation Time : Mon Apr 14 02:55:34 2008                                                           

     Raid Level : raid1                                                                              

     Array Size : 524286528 (500.00 GiB 536.87 GB)                                                   

  Used Dev Size : 524286528 (500.00 GiB 536.87 GB)                                                   

   Raid Devices : 2                                                                                  

  Total Devices : 2                                                                                  

Preferred Minor : 0

    Persistence : Superblock is persistent

    Update Time : Sat Dec 26 01:17:14 2009

          State : active

 Active Devices : 2

Working Devices : 2

 Failed Devices : 0

  Spare Devices : 0

           UUID : a4b24559:cd98291a:a5d98e37:1e73f3cc

         Events : 0.325426

    Number   Major   Minor   RaidDevice State

       0       8       18        0      active sync   /dev/sdb2

       1       8       34        1      active sync   /dev/sdc2

# mdadm --examine /dev/sdb2

/dev/sdb2:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : a4b24559:cd98291a:a5d98e37:1e73f3cc

  Creation Time : Mon Apr 14 02:55:34 2008

     Raid Level : raid1

  Used Dev Size : 524286528 (500.00 GiB 536.87 GB)

     Array Size : 524286528 (500.00 GiB 536.87 GB)

   Raid Devices : 2

  Total Devices : 2

Preferred Minor : 0

    Update Time : Sat Dec 26 01:17:20 2009

          State : clean

 Active Devices : 2

Working Devices : 2

 Failed Devices : 0

  Spare Devices : 0

       Checksum : 92452a85 - correct

         Events : 325426

      Number   Major   Minor   RaidDevice State

this     0       8       18        0      active sync   /dev/sdb2

   0     0       8       18        0      active sync   /dev/sdb2

   1     1       8       34        1      active sync   /dev/sdc2

```

----------

