# Raid5 really bad day help

## redbeardmcg

It's definitely been a while since I've been on this forum, but if there is any group of people on the internet capable of making my life better tonight it is you guys. Me and my bottle of scotch certainly hopes so...

So the story... I'm at work and I noticed my imapd was down... I try to ssh into my fileserver through my Linux router and nothing. I called home, had the girlfriend reset the file server, and bang, dead....

Booted up a Gentoo livecd and here is what mdadm seems to think....

```
livecd ~ # mdadm -E /dev/sda3

mdadm: No md superblock detected on /dev/sda3

livecd ~ # mdadm -E /dev/sdb3

/dev/sdb3:

          Magic : a92b4efc

        Version : 00.90.00

           UUID : 8c394ca9:8c59f11a:fd614d7c:a0cc1838

  Creation Time : Mon Aug 11 22:04:07 2008

     Raid Level : raid5

  Used Dev Size : 388644416 (370.64 GiB 397.97 GB)

     Array Size : 1165933248 (1111.92 GiB 1193.92 GB)

   Raid Devices : 4

  Total Devices : 3

Preferred Minor : 2

    Update Time : Mon Aug 11 22:04:14 2008

          State : clean

 Active Devices : 3

Working Devices : 3

 Failed Devices : 0

  Spare Devices : 0

       Checksum : 858a0af - correct

         Events : 0.4

         Layout : left-symmetric

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State

this     1       8       19        1      active sync   /dev/sdb3

   0     0       0        0        0      removed

   1     1       8       19        1      active sync   /dev/sdb3

   2     2       8       35        2      active sync   /dev/sdc3

   3     3       8       51        3      active sync   /dev/sdd3

```

Looks OK right, 3 out of 4 drives?

```

livecd ~ # mdadm --create /dev/md2 --assume-clean --level=5 --verbose --raid-devices=4 missing /dev/sdb3 /dev/sdc3 /dev/sdd3

mdadm: layout defaults to left-symmetric

mdadm: chunk size defaults to 64K

mdadm: /dev/sdb3 appears to be part of a raid array:

    level=raid5 devices=4 ctime=Mon Aug 11 22:04:07 2008

mdadm: /dev/sdc3 appears to contain an ext2fs file system

    size=1165933248K  mtime=Mon Jul 14 12:26:34 2008

mdadm: /dev/sdc3 appears to be part of a raid array:

    level=raid5 devices=4 ctime=Mon Aug 11 22:04:07 2008

mdadm: /dev/sdd3 appears to be part of a raid array:

    level=raid5 devices=4 ctime=Mon Aug 11 22:04:07 2008

mdadm: size set to 388644416K

Continue creating array? y

mdadm: array /dev/md2 started.

(from dmesg)

raid5: device sdd3 operational as raid disk 3

raid5: device sdc3 operational as raid disk 2

raid5: device sdb3 operational as raid disk 1

raid5: allocated 4211kB for md2

raid5: raid level 5 set md2 active with 3 out of 4 devices, algorithm 2

RAID5 conf printout:

 --- rd:4 wd:3

 disk 1, o:1, dev:sdb3

 disk 2, o:1, dev:sdc3

 disk 3, o:1, dev:sdd3

livecd ~ # mkdir /mnt/recover

livecd ~ # mount /dev/md2 /mnt/recover/

mount: unknown filesystem type 'ext4'
```

EXTENSION 4??????? WTF???????? This was formatted EXT3???

```
livecd ~ # fsck.ext3 /dev/md2

e2fsck 1.40.8 (13-Mar-2008)

fsck.ext3: Filesystem revision too high while trying to open /dev/md2

The filesystem revision is apparently too high for this version of e2fsck.

(Or the filesystem superblock is corrupt)
```

SO....

Have I lost it all? I'd probably snap if I did, EVERYTHING is on there.

I found a post from someone that never got an answer and seems to have the same problem:

http://lkml.org/lkml/2007/7/12/156

----------

## Hu

On the theory that your superblock is slightly corrupt and managed to somehow obtain a version number that flags it as ext4, try directing fsck to use an alternate superblock.  The exact location will vary based on how the filesystem was formatted.  Try at 8193 or 32768.

Also, what does file -s /dev/md2 show?  Does head -c8192 /dev/md2 | od -tx1z given any indication that some other data has replaced your filesystem?

----------

## redbeardmcg

Thanks for the reply!

Unfortunately I don't have a console on the box and I am at work so I will have to try this when I get home unless I can get the girlfriend to boot to the livecd and fire up sshd.

Any further ideas are all welcome!

-Ryan

----------

## redbeardmcg

It looks like file isn't on the livecd

```
livecd ~ # find / -name file -print

livecd ~ # 
```

Here is the only readable text in the first 8192 bytes of md2 that seems to have anything to do with the filesystem / superblock:

```

0002140 06 00 00 04 03 00 00 04 31 4a 1a ae 02 59 43 1e  >........1J...YC.<

0002160 9a 75 9c 4d 69 e9 5f 50 52 61 69 64 35 20 72 6f  >.u.Mi._PRaid5 ro<

0002200 6f 74 20 2c 4b 65 67 2d 00 00 00 04 fe 7d 00 00  >ot ,Keg-.....}..<
```

The rest is distorted fragments of txt from what was once a filesystem:

```

11201040 00 00 00 00 00 00 00 00 2f 68 6f 6d 65 2f 72 65  >......../home/re<

11201060 64 62 65 61 72 64 6d 63 67 2f 2e 6d 6f 74 64 00  >dbeardmcg/.motd.<

...

11201240 00 00 00 00 00 00 00 00 2e 2e 2f 2e 2e 2f 73 62  >........../../sb<

11201260 69 6e 2f 66 75 6e 63 74 69 6f 6e 73 2e 73 68 00  >in/functions.sh.<

...

11204040 00 00 00 00 00 00 00 00 2f 65 74 63 2f 69 6e 69  >......../etc/ini<

11204060 74 2e 64 2f 63 6f 75 72 69 65 72 2d 69 6d 61 70  >t.d/courier-imap<

11204100 64 2d 73 73 6c 00 00 00 00 00 00 00 00 00 00 00  >d-ssl...........<
```

Does this seem strange to you?

```

livecd ~ # dumpe2fs -h /dev/sda3

dumpe2fs 1.40.8 (13-Mar-2008)

dumpe2fs: Filesystem revision too high while trying to open /dev/sda3

Couldn't find valid filesystem superblock.

livecd ~ # dumpe2fs -h /dev/sdb3

dumpe2fs 1.40.8 (13-Mar-2008)

dumpe2fs: Bad magic number in super-block while trying to open /dev/sdb3

Couldn't find valid filesystem superblock.

livecd ~ # dumpe2fs -h /dev/sdc3 | head -n 10

dumpe2fs 1.40.8 (13-Mar-2008)

Filesystem volume name:   Raid5 root (Keg)

Last mounted on:          <not available>

Filesystem UUID:          314a1aaa-fc24-431e-9a75-9c4d69e95f50

Filesystem magic number:  0xEF53

Filesystem revision #:    1 (dynamic)

Filesystem features:      has_journal ext_attr filetype needs_recovery sparse_super large_file

Default mount options:    (none)

Filesystem state:         clean with errors

Errors behavior:          Continue

Filesystem OS type:       Linux

livecd ~ # dumpe2fs -h /dev/sdd3

dumpe2fs 1.40.8 (13-Mar-2008)

dumpe2fs: Bad magic number in super-block while trying to open /dev/sdd3

Couldn't find valid filesystem superblock.
```

It looks like SDC is the only disk with valid info on it, is it possible to copy the superblock on sdc3 to sd[abd]3 ? 

Many thanks,

-Ryan

----------

## Hu

 *redbeardmcg wrote:*   

> 
> 
> Does this seem strange to you?
> 
> ```
> ...

 

No, it does not seem strange.  Since RAID5 is a striping design, it seems perfectly reasonable that a filesystem created on the md container would have its header appear on only one of the underlying disks.  I would expect to get more useful output from running dumpe2fs on the md container device, but the output so far proves that at least some data has survived.

 *redbeardmcg wrote:*   

> It looks like SDC is the only disk with valid info on it, is it possible to copy the superblock on sdc3 to sd[abd]3 ? 
> 
> 

 

It is possible, but my understanding of your disk usage is that such copying would be equivalent to copying the superblock into the middle of some other piece of the disk, likely damaging a file or corrupting metadata.

----------

## jcat

You should probably clone your disks if you have the room, that was you can be sure you aren't going to loose any more during recovery attempts.

Once yo have backups, why not try and mount the container as Ext4 if that's what the FS thinks it is   :Wink:   It's worth a try, I'm no disk or FS guru, but I'm not sure how else you might be able to recover this.  Needless to say you've been _very_ unlucky.

Cheers,

jcat

----------

