# How to change RAID superblock format on one of four drives

## Goverp

I've been running a 4 disk software RAID 5 array as my file system for 3 or so years without problem.  The kernel assembles the array in response to boot command line 

```
 root=/dev/md127p3 md=127,/dev/sda3,/dev/sdb3,/dev/sdc3,/dev/sdd3 vga=0x317
```

  This isn't kernel auto-assembly, that's switched off in the kernel and the partitions are type 0xDA just to make sure.

About a fortnight ago, it started up in degraded mode, having lost sdb3.  As the start of the degredation coincided with a power cut, after a bit of research in Gentoo Forums, I decided that re-adding the disk was sufficient 

```
mdadm --add /dev/md127 /dev/sdb3
```

and that got it all going again.

Today, sdb3 dropped again; I added it back again, and ordered a new drive   :Wink:    Then I did some digging, and ran 

```
mdadm --examine /dev/sd{a,b,c,d}3
```

 This showed that sdb3 has a version 0.9 superblock, whereas sda3, sdc3 and sdd3 are all version 1.0 superblocks.  I'm sure they were originally formatted them all as 1.0, so it looks as though adding sdb3 did something odd, and probably mixing superblock versions in the same array is bad.  Also, the event count for sdb3 is massively greater than for the others.  (I'll put the output for sda3 and sdb3 at the end of this message.)

It's clear from the mdadm documentation that it's no problem to fail and remove sdb3.  Is there then a way to change its superblock version before adding it back into the array?  As far as I can tell, superblocks are written when the array is created, and I don't want to recreate the array, just fix this one disk.

I do have a backup.  :Very Happy: 

FWIW, /etc/mdadm.conf contains the original setup, but of course mdadm can't see it when the kernel assembles the array, as /etc is in /dev/md127p3.  But perhaps this superblock issue is why the array doesn't appear as /dev/md/gentoo, as it did when I created it.  Here's /etc/mdadm.conf - the ARRAY line is commented out for some reason lost in the mists of time, but as I say, I think it's now irrelevant.  Its UUID matches that for the version 1.0 superblocks.

```
DEVICE /dev/sd[a-d]3

#ARRAY /dev/md/gentoo metadata=1.00 name=acer:gentoo UUID=c80d66f1:9bc95138:78a82083:d3cc09dc

HOMEHOST acer
```

All advice gratefully received.

```
acer ~ # mdadm --examine /dev/sd{a,b}3

/dev/sda3:

          Magic : a92b4efc

        Version : 1.0

    Feature Map : 0x1

     Array UUID : c80d66f1:9bc95138:78a82083:d3cc09dc

           Name : acer:gentoo  (local to host acer)

  Creation Time : Sun Mar 21 10:51:13 2010

     Raid Level : raid5

   Raid Devices : 4

 Avail Dev Size : 524281000 (250.00 GiB 268.43 GB)

     Array Size : 1572842880 (749.99 GiB 805.30 GB)

  Used Dev Size : 524280960 (250.00 GiB 268.43 GB)

   Super Offset : 524281256 sectors

          State : clean

    Device UUID : fc377d41:9dd67ea4:01b65109:0e56d006

Internal Bitmap : 2 sectors from superblock

    Update Time : Fri Mar 26 10:53:28 2010

       Checksum : fe762ec7 - correct

         Events : 306

         Layout : left-symmetric

     Chunk Size : 64K

   Device Role : Active device 0

   Array State : AAAA ('A' == active, '.' == missing)

/dev/sdb3:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : 50d6983f:79bdb0a2:47d7a9c0:0d34b7b1 (local to host acer)

  Creation Time : Sat Mar 20 20:41:06 2010

     Raid Level : raid5

  Used Dev Size : 262140544 (250.00 GiB 268.43 GB)

     Array Size : 786421632 (749.99 GiB 805.30 GB)

   Raid Devices : 4

  Total Devices : 4

Preferred Minor : 127

    Update Time : Sat Dec 15 19:38:50 2012

          State : clean

 Active Devices : 4

Working Devices : 4

 Failed Devices : 0

  Spare Devices : 0

       Checksum : 74e22df8 - correct

         Events : 105948

         Layout : left-symmetric

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State

this     1       8       19        1      active sync   /dev/sdb3

   0     0       8        3        0      active sync   /dev/sda3

   1     1       8       19        1      active sync   /dev/sdb3

   2     2       8       35        2      active sync   /dev/sdc3

   3     3       8       51        3      active sync   /dev/sdd3
```

----------

## NeddySeagoon

Goverp,

Thats all very odd, since when you create the array, you tell mdam what superblock version to use for all elements of the array.

The different versions of the raid superblock are located in different places on the block devices being donated to the raid.  Some version, not 0.9, have several copies of the superblock.

Google will tell you more about how the various raid superblock versions differ.

Is it possible that the version 0.9 superblock you are seeing is a leftover from an old raid set?

You can drop /dev/sdb3 out of the raid, zero the superblock and add  it back. The resync will write a new superblock.

You might try 

```
echo check > /sys/block/md127/md/sync_action
```

This will read the raid and check the parity data.  I'm not sure if it covers the superblock content or not.

----------

## Goverp

Thanks Neddy.

I thought I'd replied to this yesterday, but looks like I forgot to press "Submit", but that reply was a red herring anyway.  What I've since learned is beware of looking under stones...   :Sad: 

There's only one array on this machine:

```
acer ~ # mdadm --detail --scan -v

ARRAY /dev/md127 level=raid5 num-devices=4 metadata=0.90 UUID=50d6983f:79bdb0a2:47d7a9c0:0d34b7b1

   devices=/dev/sda3,/dev/sdb3,/dev/sdc3,/dev/sdd3
```

But mdadm --examine thinks there are two:

```
acer ~ # mdadm --examine --scan

ARRAY /dev/md127 UUID=50d6983f:79bdb0a2:47d7a9c0:0d34b7b1

ARRAY /dev/md/gentoo metadata=1.0 UUID=c80d66f1:9bc95138:78a82083:d3cc09dc name=acer:gentoo
```

The individual partitions claim sdb3 in the first, and the other three in the second - see below if you want the gory details for sda3 and sdb3.

As a result of this, and following your advice, I removed sdb3 from the array, zeroed the superblock, and tried adding it back to the array while explicitly specifying version 1.0.  mdadm under Gentoo wouldn't allow that, so I booted up Knoppix from a USB key, and manually started the array with an mdadm.conf reading

```
ARRAY /dev/md127 metadata=1.0 UUID=c80d66f1:9bc95138:78a82083:d3cc09dc name=acer:gentoo
```

which was what I'd got when I created the array three years ago, apart from changing the device from /dev/md/gentoo.  That worked.

So, now under Knoppix the array built using V 1.0 metadata.  Reboot.  And sdb3 drops out with message: 

```
 Dec 16 22:40:31 kernel: md: invalid raid superblock magic on sdb3

 Dec 16 22:40:31 kernel: md: sdb3 does not have a valid v0.0 superblock, not importing!

 Dec 16 22:40:31 kernel: md: md_import_device returned -22
```

Whoops!  Adding /dev/sdb3 into the array under Gentoo "cured" the problem and now it all looks hunky dory, with /dev/md127 being built using V0.9 metadata.

What I think this shows is that the in-kernel array assembly uses a twisted version of mdadm that only recognizes V0.9 metadata.  The md.txt documentation says so for auto-assembly, but this is specified using kernel boot parameters, not auto-assembly.  I'd hoped doing it this way would get the extra security of V1.0 metadata to avoid the documented dangers of the auto-assembly (which IIUC will include components from "foreign" arrays 'cos it doesn't check properly).  It looks like kernel boot parameter assembly is just as bad.  I probably ought to convert to using an initramfs to build it properly.  Gosh, then I could also stop worrying about separate /var and start using systemd (just a joke   :Wink:  ).

Meanwhile it's a bit worrying that a partition can be shown as having V1.0 superblock but still be built into a V0.9 array:

```
acer ~ # mdadm --examine /dev/sd[a-b]3

/dev/sda3:

          Magic : a92b4efc

        Version : 1.0

    Feature Map : 0x1

     Array UUID : c80d66f1:9bc95138:78a82083:d3cc09dc                                                                                                    

           Name : acer:gentoo  (local to host acer)                                                                                                      

  Creation Time : Sun Mar 21 10:51:13 2010

     Raid Level : raid5

   Raid Devices : 4

 Avail Dev Size : 524281000 (250.00 GiB 268.43 GB)

     Array Size : 1572842880 (749.99 GiB 805.30 GB)

  Used Dev Size : 524280960 (250.00 GiB 268.43 GB)

   Super Offset : 524281256 sectors

          State : clean

    Device UUID : fc377d41:9dd67ea4:01b65109:0e56d006

Internal Bitmap : 2 sectors from superblock

    Update Time : Sun Dec 16 16:49:41 2012

       Checksum : 2d99a3b - correct

         Events : 997

         Layout : left-symmetric

     Chunk Size : 64K

   Device Role : Active device 0

   Array State : AAAA ('A' == active, '.' == missing)

/dev/sdb3:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : 50d6983f:79bdb0a2:47d7a9c0:0d34b7b1 (local to host acer)

  Creation Time : Sat Mar 20 20:41:06 2010

     Raid Level : raid5

  Used Dev Size : 262140544 (250.00 GiB 268.43 GB)

     Array Size : 786421632 (749.99 GiB 805.30 GB)

   Raid Devices : 4

  Total Devices : 4

Preferred Minor : 127

    Update Time : Mon Dec 17 10:33:03 2012

          State : clean

 Active Devices : 4

Working Devices : 4

 Failed Devices : 0

  Spare Devices : 0

       Checksum : 74e46269 - correct

         Events : 108170

         Layout : left-symmetric

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State

this     1       8       19        1      active sync   /dev/sdb3

   0     0       8        3        0      active sync   /dev/sda3

   1     1       8       19        1      active sync   /dev/sdb3

   2     2       8       35        2      active sync   /dev/sdc3

   3     3       8       51        3      active sync   /dev/sdd3
```

----------

## Goverp

Arrgh!

I thought I'd look to see whether the array has working bit maps or not.  sda3, sdc3 and sdd3 say yes:

```
acer ~ # mdadm -X /dev/sda3

        Filename : /dev/sda3

           Magic : 6d746962

         Version : 4

            UUID : c80d66f1:9bc95138:78a82083:d3cc09dc

          Events : 997

  Events Cleared : 306

           State : OK

       Chunksize : 16 MB

          Daemon : 5s flush period

      Write Mode : Normal

       Sync Size : 262140480 (250.00 GiB 268.43 GB)

          Bitmap : 16000 bits (chunks), 0 dirty (0.0%)
```

but sdb3 looks nasty:

```
acer ~ # mdadm -X /dev/sdb3

        Filename : /dev/sdb3

           Magic : 00000000

mdadm: invalid bitmap magic 0x0, the bitmap file appears to be corrupted

         Version : 0

mdadm: unknown bitmap version 0, either the bitmap file is corrupted or you need to upgrade your tools
```

initramfs here I come.

----------

