# RAID1 confusion when a disc goes bad

## graysky

I have one of the most simple RAID1 arrays I think: 2x 4TB discs (/dev/sdb1 and /dev/sdc1).  Everything is fine now, but let's assume that one of the discs starts going bad some day.  

1) If I scrub by running `echo check > /sys/block/md0/md/sync_action` regularly, how can I understand which files have differing checksums if I do actually have corruption on the array?

2) When I do find out that /mnt/md0/foo has a different checksum from its copy, how do I know which of the two is the "good" copy?

EDIT: I think the error in my thinking is explained by the fact that mdadm is not file-level RAID, it is block-level RAID and when I scrub, I am not looking at the checksums of the files, but rather am looking for corrupt blocks on the device.  Do I have this right?

----------

## frostschutz

If you had three disks in the RAID 1 instead of just two, you could probably tell one bad disk from the other two good disks. With just two disks it's not possible, if they differ, either one may be correct.

This kind of corruption does not usually occur. HDDs do check the integrity of their stored data, they are able to detect bad blocks, and yield an I/O error message instead of silently returning bad data.

In case of the I/O error, the RAID layer reads the data from the other disk instead. It may try to rewrite it on the bad disk (scrub) and if that fails, kick the disk out of the RAID entirely.

That's the same thing the check sync action should be doing.

EDIT: This is described in the manpage (man md) in more detail. In case of a mismatch as you describe, check only reports it (md/mismatch_cnt); repair would cause it to be fixed by writing one of the other blocks (or in case of RAID5/6, the parity blocks). So one side of the information is simply lost/ignored in such a case, as it can't tell which one should be correct. The manpage also lists some cases where mismatches in a RAID 1 can occur naturally without causing any real harm.

----------

## eccerr0r

I have a 4-disk RAID1 (for boot purposes of a RAID5) and I think the contents of the four are desynched.  I'm not sure what what the kernel would do in this case  - what if all the disks' metadata was up to date, and all four had different information?  Would it make a note where the differences were?

(how I could tell?  When I tried booting off of them, they all produced different results as they loaded in grub.  Some had loaded in an old menu.lst, some wouldn't even boot.)

Right now I tried telling to resync as I've done in the past but not sure if they are all actually synchronized.  Perhaps I should fail all of them except for the disk I know is up to date and re-add each of them...

I'm not exactly sure I how I got it into this state.

----------

## graysky

@frost - Thank you for the info and the links to the man page.

@eec - Why do you have 4 discs in a RAID1?  Why use a higher RAID level like 5 or 10?

----------

## eccerr0r

I actually do have a 4 hard drive MDRAID5.  But... BIOS/EFI doesn't know how to boot off of MDRAID5.  But it can boot off of MDRAID1.

So... I partitioned a small RAID1 with the beginnings of each disk that can be boot from, and that bootstraps the RAID5 (rootfs on LVM on RAID5).  Now I can use the 4 disk RAID5 as a boot "disk" and have a bit of redundancy on that too.  Except for this apparently annoying sync issue.  I suppose not many people do more than 2-disk RAID1s, might be a bug, dunno...

----------

## NeddySeagoon

eccerr0r.

What does mdadm -E say for the components of your raid1 ?

What about /proc/mdstat  ?

I have two systems as you describe, one with 4 spindles and one with 5.   I've not noticed any issues.

```
$ cat /proc/mdstat 

Personalities : [raid1] [raid6] [raid5] [raid4]       

md125 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]

      40064 blocks [4/4] [UUUU]
```

----------

## eccerr0r

Apologies for threadjacking (sort of) but I thought it was relevant to the initial query.

This disk array is 4 disk plus 1 hot spare.  I suffered a machine (motherboard) failure recently and hence /dev/sdb turned to be the spare disk as I rushedly reconnected the SATA cables to the new motherboard.  I had this apparent inconsistency before this move, as can be witnessed as I had boot problems trying to boot this machine when the sata cables are switched around.  Theoretically swapping cables should be fine with mdraid... but if the contents of the disks in the raid1 don't match, it can cause boot issues.

The only disk that I really shouldn't be able to boot from is the hot spare - I need to make sure that /dev/sdb is not part of the boot routine or at least make sure bios won't hang on that disk or try that disk last.  But of the four main disks of the array, I ended up with 1 disk with old data, one disk that failed stage 1.5, and one that worked (and the last one I didn't check as I finally got the machine to boot, elated)...  Hmm.

Worth to test this again and again to make sure it actually works... 

/proc/mdstat for the raid1 looks like

```
md0 : active raid1 sde1[0] sdb1[4](S) sdd1[3] sdc1[2] sda1[1]

      136448 blocks [4/4] [UUUU]

```

I dumped the mdstat -E of each member to files and diffing them:

- The checksum differs for the superblock for each member.  Kind of expected as they are different members

- The "this" line is different, which is expected.

----------

## NeddySeagoon

eccerr0r,

The Event count was identical on all drives ?

That the filesystem in /boot is consistent across drives is not sufficient to get booted.

You also need to maintain grub stage1 and stage1.5 across drives. These parts of grub are outside the filesystem  and therefore not covered by the raid.

Did you forget that when you got grub updates?

Grub ignores any raid that may be present on /boot at boot time.

----------

## eccerr0r

Yes, the event counts are all equal.

The drive that completely failed to boot probably had a bad stage1 installed on it (or never installed).  But the failed stage 1.5 (and the one with old data - which should have been read after stage2 which belongs on the raid) I don't know.

I'll have to see what to do, now that grub2 is stabilized...

----------

## NeddySeagoon

eccerr0r,

 *eccerr0r wrote:*   

> I'll have to see what to do, now that grub2 is stabilized...

 

Just mask grub2.  If you have a (U)EFI BIOS, you don't need a boot loader at all.  If you don't have a (U)EFI BIOS then grub is fine.

I run grub-static from 2009 as running grub to install it needs 32 bit kernel support and I don't have that.

----------

