# Help with failing disk

## remix

I have a 4 disk Raid5 array. One disk has completely died, and before I found time to replace that disk, (while running with 3 or 4 disks) one of my other hard drives started to die. first one partition, then another, while the others on that same disk continue to work.

I really need to recover some files from one of the partitions in the newly failing disk, and i just bought a couple new hard drives and replaced the one completely destroyed disk.

My question is, is there any way to recover one of the partitions in that 'fourth' disk so that i can assemble and mount it (using 3 of 4 partitions)?

```
mdadm --assemble  --force /dev/md5 /dev/sda6 /dev/sdc6 /dev/sdd6

mdadm: cannot open device /dev/sdd6: No such file or directory

mdadm: /dev/sdd6 has no superblock - assembly aborted
```

/dev/sdd is the failing drive

/dev/sdb is the completely failed drive that has been replaced

here is some snippets from smartctl -a /dev/sdd

```
=== START OF READ SMART DATA SECTION ===

SMART overall-health self-assessment test result: FAILED!

Drive failure expected in less than 24 hours. SAVE ALL DATA.

See vendor-specific Attribute list for failed Attributes.

Error 16129 occurred at disk power-on lifetime: 35980 hours (1499 days

+ 4 hours)

  When the command that caused the error occurred, the device was activ

e or idle.

  After command completion occurred, registers were:

  ER ST SC SN CL CH DH

  -- -- -- -- -- -- --

  40 51 01 5f bb 7e 00  Error: UNC 1 sectors at LBA = 0x007ebb5f = 8305

503

  Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name

  -- -- -- -- -- -- -- --  ----------------  --------------------

  25 00 02 5e bb 7e e0 0a   3d+16:18:43.635  READ DMA EXT

  25 00 08 76 ba 7e e0 0a   3d+16:18:43.275  READ DMA EXT

  ca 00 08 3f 00 00 e0 0a   3d+16:18:43.141  WRITE DMA

  ca 00 08 6f 30 00 e0 0a   3d+16:18:43.091  WRITE DMA

  ca 00 08 67 30 00 e0 0a   3d+16:18:43.038  WRITE DMA

Error 16128 occurred at disk power-on lifetime: 35980 hours (1499 days

+ 4 hours)

  When the command that caused the error occurred, the device was activ

e or idle.

After command completion occurred, registers were:

  ER ST SC SN CL CH DH

  -- -- -- -- -- -- --

  40 51 05 73 bb 7e 00  Error: UNC 5 sectors at LBA = 0x007ebb73 = 8305

523

Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name

  -- -- -- -- -- -- -- --  ----------------  --------------------

  25 00 08 70 bb 7e e0 0a   3d+14:28:37.227  READ DMA EXT

  27 00 00 00 00 00 e0 0a   3d+14:28:37.225  READ NATIVE MAX ADDRESS EX

T

  ec 00 00 00 00 00 a0 0a   3d+14:28:37.102  IDENTIFY DEVICE

  ef 03 46 00 00 00 a0 0a   3d+14:28:36.982  SET FEATURES [Set transfer

 mode]

  27 00 00 00 00 00 e0 0a   3d+14:28:36.981  READ NATIVE MAX ADDRESS EX

T

Error 16127 occurred at disk power-on lifetime: 35980 hours (1499 days

+ 4 hours)

  When the command that caused the error occurred, the device was activ

e or idle.

  After command completion occurred, registers were:

  ER ST SC SN CL CH DH

  -- -- -- -- -- -- --

  40 51 01 77 bb 7e 00  Error: UNC 1 sectors at LBA = 0x007ebb77 = 8305

527

  Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name

  -- -- -- -- -- -- -- --  ----------------  --------------------

  25 00 08 70 bb 7e e0 0a   3d+14:27:52.480  READ DMA EXT

  27 00 00 00 00 00 e0 0a   3d+14:27:52.479  READ NATIVE MAX ADDRESS EX

T

  ec 00 00 00 00 00 a0 0a   3d+14:27:52.356  IDENTIFY DEVICE

  ef 03 46 00 00 00 a0 0a   3d+14:27:52.236  SET FEATURES [Set transfer

 mode]

  27 00 00 00 00 00 e0 0a   3d+14:27:52.234  READ NATIVE MAX ADDRESS EX

T

  Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name

  -- -- -- -- -- -- -- --  ----------------  --------------------

  25 00 08 70 bb 7e e0 0a   3d+14:27:52.480  READ DMA EXT

  27 00 00 00 00 00 e0 0a   3d+14:27:52.479  READ NATIVE MAX ADDRESS EX

T

  ec 00 00 00 00 00 a0 0a   3d+14:27:52.356  IDENTIFY DEVICE

  ef 03 46 00 00 00 a0 0a   3d+14:27:52.236  SET FEATURES [Set transfer

 mode]

  27 00 00 00 00 00 e0 0a   3d+14:27:52.234  READ NATIVE MAX ADDRESS EX

T

Error 16126 occurred at disk power-on lifetime: 35980 hours (1499 days

+ 4 hours)

  When the command that caused the error occurred, the device was activ

e or idle.

  After command completion occurred, registers were:

  ER ST SC SN CL CH DH

  -- -- -- -- -- -- --

  40 51 05 73 bb 7e 00  Error: UNC 5 sectors at LBA = 0x007ebb73 = 8305

523

  Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name

  -- -- -- -- -- -- -- --  ----------------  --------------------

  25 00 08 70 bb 7e e0 0a   3d+14:27:00.958  READ DMA EXT

  27 00 00 00 00 00 e0 0a   3d+14:27:00.956  READ NATIVE MAX ADDRESS EX

T

  ec 00 00 00 00 00 a0 0a   3d+14:27:00.834  IDENTIFY DEVICE

  ef 03 46 00 00 00 a0 0a   3d+14:27:00.713  SET FEATURES [Set transfer

 mode]

  27 00 00 00 00 00 e0 0a   3d+14:27:00.712  READ NATIVE MAX ADDRESS EX

T

Error 16125 occurred at disk power-on lifetime: 35980 hours (1499 days

+ 4 hours)

  When the command that caused the error occurred, the device was activ

e or idle.

  After command completion occurred, registers were:

  ER ST SC SN CL CH DH

  -- -- -- -- -- -- --

  40 51 08 70 bb 7e 00  Error: UNC 8 sectors at LBA = 0x007ebb70 = 8305

520

  Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name

  -- -- -- -- -- -- -- --  ----------------  --------------------

  25 00 08 70 bb 7e e0 0a   3d+14:26:40.956  READ DMA EXT

  25 00 08 30 02 8a e0 0a   3d+14:26:40.854  READ DMA EXT

  c8 00 40 20 00 00 e0 0a   3d+14:26:40.853  READ DMA

  25 00 08 a8 6d 70 e0 0a   3d+14:26:40.725  READ DMA EXT

  c8 00 20 00 00 00 e0 0a   3d+14:26:40.633  READ DMA

```

I'm hoping someone who understands this stuff can help me point me in the right direction of possibly preserving even a little of the unreadable partition.

----------

## NeddySeagoon

remix,

Install ddrescue and use that to make an image of the most recently failed drive onto the new drive

Make sure to put the ddrescue log on a third drive.  

```
# Rescue Logfile. Created by GNU ddrescue version 1.15

# Command line: ddrescue -b 4096 -r 8 -f /dev/sde3 /dev/null /root/rescue_log.txt

# current_pos  current_status

0x18D786D0000     ?

#      pos        size  status

0x00000000  0x16E4BE9E000  +

0x16E4BE9E000  0x00002000  *

0x16E4BEA0000  0xFD4F9D000  +

0x17E20E3D000  0x00003000  *

0x17E20E40000  0x8FBF8000  +

0x17EB0A38000  0x00008000  *

0x17EB0A40000  0x358CC2000  +

0x18209702000  0x0000E000  *

0x18209710000  0x2DE00000  +

0x18237510000  0x00010000  *

0x18237520000  0x01AC0000  +

0x18238FE0000  0x00010000  *

0x18238FF0000  0x012C1000  +

0x1823A2B1000  0x0000F000  *

0x1823A2C0000  0x2D752000  +

0x18267A12000  0x0000E000  *

0x18267A20000  0x11EDD4000  +

0x183867F4000  0x0000C000  *

0x18386800000  0x9F1ED0000  +

0x18D786D0000  0x4260530000  ?
```

is one I did earlier.

DON'T DO THIS YET Notice the output file here is /dev/null ... all I was trying to do was get the drive to do one last read and relocate the data so I could grab it later.

You need the best image you can get first. 

You need the -b 4096 for advanced format drives. There is no point in trying to recover 512 bytes if the drive has a4k block size.

-r 8 eight retries is a good place to start.

The log allows ddrescue to resume recovery, even with a different command.  It will only work on areas not yet recovered.

ddrescuse can work much harder and you can help it too but thats for another post.

----------

## remix

I got confused when you said "Don't do this yet"

If I am understanding you correctly, i should not perform a Disk to Disk ddrescue yet, 

first, I should just copy to /dev/null and output the errors to rescue_log.txt

my setup,

/dev/sda good

/dev/sdb newly installed, partitioned to match raid ( -b 4096)

/dev/sdc good

/dev/sdd failing ( -b 512)

and another brand new hd ( -b 4096) that is waiting to replace /dev/sdd once i can recover anything i can from my 5th raid partition.

/dev/sdd1, working

/dev/sdd2, working

/dev/sdd3, starting to fail

/dev/sdd4, extended

/dev/sdd5, failed  (most important)

/dev/sdd6, failed  (don't really care)

```
ddrescue -b 512 -r 8 -f /dev/sdd5 /dev/null /root/rescue_log.txt 
```

then i'll inspect /root/rescue_log.txt knowing nothing of what those hex addresses mean

then actually perform the copy

```
ddrescue -b 4096 -f -n /dev/sdd5 /dev/sdb5 /root/rescue_copy_log.txt
```

or should i be copying over the entire disk?

```
ddrescue -b 4096 -f -n /dev/sdd /dev/sdb /root/rescue_copy_log.txt
```

----------

## NeddySeagoon

remix,

The disk may fail completely at any time.

Do the disc to disc rescue first.  The disk to /dev/null is a final desperate attempt to get more data recovered.

You may as well copy the entire disk.  If you only copy a partition, how will you recover your raid sets?

You will need to get the good partitions onto the new drive at some time.

Of course, if you are still using the raid sets in degraded mode, the data will change and whatever ddrescue copies now from the degraded arrays will be useless.

You should not use ddrescue at all until yon know what its telling you. Read its man and/or info pages.

The hex numbers are block numbers.  The symbols at the end of each line tell what the block numbers mean.

A log showing perfect data recovery will have exactly one line of data.

Being able to do arithmetic is hex is useful, since you can work out where and how many blocks you have lost, or still have to recover.

With a bit more poking about the filesystem, you can get a rough idea of whats there and determine its importance.

That enables an informed decision about giving up or trying harder.

Only use -b 4096 on drives with 4k physical sectors. Use -b 512 on other drives.

----------

## remix

Good point.

Sounds like it would be safer to boot into a livedvd and not mount any of the degraded raid partitions.

I'll install the new blank disk, and perform the ddrescue

```
ddrescue -b 4096 -r 8 -n /dev/sdd /dev/sdb /sshfs_mounted_volume/rescue_copy_log.txt
```

I've read this guide, http://wiki.gentoo.org/wiki/Ddrescue

I'll check out the man page as well

thanks!

----------

## remix

i just read the man page and -b is the block size of the input device, which in my case is 512

output device is 409

so it is

```
ddrescue -b 512 -r 8 -n /dev/sdd /dev/sdb /sshfs_mounted_volume/rescue_copy_log.txt
```

----------

## NeddySeagoon

remix,

Give it a go.  Post the log when it stops.

You will find that gravity can assist the data recovery,

rerun the same command using the same log file a total of six times.

Make a copy of the log each time the command completes, so you can look at the differences later.

Between each invocation of the command, move the drive so you try it with each edge and both faces 'down'.

If you have a bearing failure, gravity and the odd orientations can get you one last read and thats all you need.

----------

## remix

awesome tip! should i be flipping it during the retries? (without stopping or rebooting)

i just finished the first go through, i did set it to retry 8 times.

```
GNU ddrescue 1.16

Press Ctrl-C to interrupt

rescued:     1000 GB,  errsize:    543 kB,  current rate:        0 B/s

   ipos:   717936 MB,   errors:      40,    average rate:    8546 kB/s

   opos:   717936 MB,     time since last successful read:     4.3 m

Retrying bad sectors... Retry 1
```

----------

## NeddySeagoon

remix,

That looks fairly good so far.

```
rescued:     1000 GB,  errsize:    543 kB,    errors:      40,
```

You have 543 kB still to recover in 40 regions of the drive.

Its time to tell ddrescue to try harder, now that most of the data has been read.

By using the same input device, output device and log file, ddrescue will try to fill in the holes in your image an ignore data already recovered.

Look back at the copies of the logs and determine which drive orientation produced the best resuts.

You will still run all four edges and two faces but treat each drive spin up as it it were the last, so start with that orientation.    

--retries= can be increased.  I tend to try 8, 16, 32, 64 and 128 

--direct  may be useful, it has no effect on some operating systems

--try-again can help when you have a group of contiguous  blocks that cant be read.  

--retrim will help too.

After you have tried the above, on all 6 faces, (just with --retries=8) its time to look at what is still missing.

If its unallocated space, it doesn't matter.

If its a file or two, they are gone - you need to decide if you need these files and how much time you want to spend on data recovery.

If its a directory then the files in that directory and its child directories cannot be accessed normally but they may be perfectly recovered.

If its filesystem metadata, it depends what aspects of the metadata are damaged.

Please post the log - like my sample above, next time and we can begin to take into account whats damaged.

----------

## remix

i'm ok with some of the files being completely inaccessible, well let's see how many that would be.

```
# Rescue Logfile. Created by GNU ddrescue version 1.16

# Command line: ddrescue -b 512 -r 8 -f /dev/sdd /dev/sdb ddrescue.log

# current_pos  current_status

0xA728A5FE00     +

#      pos        size  status

0x00000000  0xA2FBE8F000  +

0xA2FBE8F000  0x00003000  -

0xA2FBE92000  0x008E0000  +

0xA2FC772000  0x00006000  -

0xA2FC778000  0x0001A000  +

0xA2FC792000  0x00001000  -

0xA2FC793000  0x00199000  +

0xA2FC92C000  0x00003000  -

0xA2FC92F000  0x000C6000  +

0xA2FC9F5000  0x00001000  -

0xA2FC9F6000  0x00488000  +

0xA2FCE7E000  0x00001000  -

0xA2FCE7F000  0x00008000  +

0xA2FCE87000  0x00001000  -

0xA2FCE88000  0x000E6000  +

0xA2FCF6E000  0x00005000  -

0xA2FCF73000  0x0060A000  +

0xA2FD57D000  0x00004000  -

0xA2FD581000  0x0000B000  +

0xA2FD58C000  0x00001000  -

0xA2FD58D000  0x00002000  +

0xA2FD58F000  0x0000F000  -

0xA2FD59E000  0x000D0000  +

0xA2FD66E000  0x00002000  -

0xA2FD670000  0x00042000  +

0xA2FD6B2000  0x00006000  -

0xA2FD6B8000  0x000A1000  +

0xA2FD759000  0x00001000  -

0xA2FD75A000  0x00016000  +

0xA2FD770000  0x00002000  -

0xA2FD772000  0x00ADB000  +

0xA2FE24D000  0x00002000  -

0xA2FE24F000  0x00005000  +

0xA2FE254000  0x00001000  -

0xA2FE255000  0x00001000  +

0xA2FE256000  0x0000B000  -

0xA2FE261000  0x00172000  +

0xA2FE3D3000  0x00005000  -

0xA2FE3D8000  0x002C8000  +

0xA2FE6A0000  0x00006000  -

0xA2FE6A6000  0x00006000  +

0xA2FE6AC000  0x00001000  -

0xA2FE6AD000  0x00002000  +

0xA2FE6AF000  0x00001000  -

0xA2FE6B0000  0x1C121000  +

0xA31A7D1000  0x00002000  -

0xA31A7D3000  0x00137000  +

0xA31A90A000  0x00001000  -

0xA31A90B000  0x2AD8C000  +

0xA345697000  0x00002000  -

0xA345699000  0x001BE000  +

0xA345857000  0x00002000  -

0xA345859000  0x3E2D43000  +

0xA72859C000  0x00001000  -

0xA72859D000  0x00005000  +

0xA7285A2000  0x00007000  -

0xA7285A9000  0x003E4000  +

0xA72898D000  0x00010000  -

0xA72899D000  0x000B4000  +

0xA728A51000  0x0000F000  -

0xA728A60000  0x41B8356000  +
```

would it be safe then to just replace that 'fourth' drive with this new copied drive? and it will function normally except for those few files or directories?

----------

## NeddySeagoon

remix,

```
#      pos        size  status

0x00000000  0xA2FBE8F000  + 
```

Says that from the start of the drive, to block 0xA2FBE8F000' all the data has been recovered.

At 

```
#      pos        size  status

0xA2FBE8F000  0x00003000  - 
```

is the first bad area.  A 512b block is 0x200 byles, so 0x1000 is eight blocks. (Its hex)

So this area is 24 (decimal) blocks.

Each Status + is recovered data.  Each status - is data yet to be recovered. 

You can do better than 

```
ddrescue -b 512 -r 8 -f /dev/sdd /dev/sdb ddrescue.log 
```

```
ddrescue -b 512 -r 16 --direct  --try-again --retrim -f /dev/sdd /dev/sdb ddrescue.log
```

 may get back more data.

Don't forget to do all six faces/edges.

The idea is to restart the raid with this drive in place of the failing drive but not yet.

What metadata version is the raid set. You need to know that the metadata is recovered.

mdadm -E /dev/.... will tell.

What filesystem is on the raid set?

Once you add the recovered drive into the raid set, you have decided that further data recovery is not worthwhile.

The raid can't tell the data is corrupt due to the unrecovered data. It will just operate in degraded mode and assume all is well.

When you add another drive, it will regenerate the redundent data based on whatever is on the other drives at that time

----------

## remix

thanks for the info, it makes sense to me.

the filesystem on the raid partitions is reiserfs 

the new log is long so i pastie'd it here http://pastie.org/9118746

----------

## remix

i don't think i recovered enough, not sure what i did wrong (other than not having backups)

```
OptimusPrime / # mdadm --assemble /dev/md4 --scan --force

mdadm: /dev/sdd5 has no superblock - assembly aborted

OptimusPrime / # mdadm --assemble /dev/md5 --scan --force

mdadm: /dev/sdb6 has no superblock - assembly aborted

OptimusPrime / # mdadm --assemble /dev/md6 --scan --force

mdadm: /dev/sdb7 has no superblock - assembly aborted

```

I have 2 partitions that seem to be ok

```
OptimusPrime / # cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4]

md4 : inactive sda5[0](S) sdd5[3](S) sdc5[2](S)

      527373312 blocks

md1 : active raid5 sda1[0] sdd1[3] sdc1[2] sdb1[1]

      102558528 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md2 : active raid5 sda2[0] sdd2[3] sdc2[2] sdb2[1]

      776397120 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md3 : inactive sda3[0](S) sdb3[4](S) sdd3[3](S) sdc3[2](S)

      859412736 blocks

unused devices: <none>
```

should i just forfeit all the data in those 3 partitions and reformat?

----------

## NeddySeagoon

remix,

We are not done yet.

Can you remember the parameters to --create that you used when you created md3 and md4?

What does 

```
mdadm -E /dev/sd[abcd]3
```

Also 

```
mdadm -E dev/sd[abcd]5
```

How many elements are in each raid set?

How many have at least some data in now?

Read and understand RAID Recovery.  In a nutshell, you can recreate the metadata.

You need to do it on a degraded array.  Its best if you don't get the raid metadata version wrong as metadata version 0.9 is at the end of eacd partition and the filesystem starts in the normal place, as if raid was not in use. With metadata version >=1, the metadata is at the start of the volume, where the filesystem superblock would be.

You can recover from getting it wrong but its best that you don't need to

The basic idea is to run a mdadm --create to rewrite the raid metadata in exactly the way you did when you first made the raid set but in degraded mode with the known clean option, so the raid is not resynced.  That leaves your original data in place.

Were you able to recover any more data?

----------

## remix

i created the md devices using

```
mdadm --create --verbose --level=5 --raid-devices=4 /dev/md4 /dev/sdb5 /dev/sdc5 /dev/sdd5 /dev/sde5

mdadm --create --verbose --level=5 --raid-devices=4 /dev/md5 /dev/sdb6 /dev/sdc6 /dev/sdd6 /dev/sde6

mdadm --create --verbose --level=5 --raid-devices=4 /dev/md6 /dev/sdb7 /dev/sdc7 /dev/sdd7 /dev/sde7

...
```

the output of mdadm -E /dev/sd[abcd]5

looks like i'll need to restore the superblocks in /dev/sdb

/dev/sdb is the drive that i restored the old failed drive into

when you wrote

 *Quote:*   

> The basic idea is to run a mdadm --create to rewrite the raid metadata in exactly the way you did when you first made the raid set but in degraded mode with the known clean option, so the raid is not resynced. That leaves your original data in place. 

 

do you mean 'in degraded mode' by adding only 3 of the 4 disks?

i read the RAID Recovery guide and i didn't get how to perform what you asked.

----------

## NeddySeagoon

remix,

There is no need to use --create on the raid sets that are now working. There is nothing to do to them if /proc/mdstat shows that they are up to strength.  

If its /dev/md4 thats the problem, I need the output of 

```
mdadm -E /dev/sd[abcd]5
```

and I need to know what you think is in each /dev/sd?5.

Your command 

```
mdadm --create --verbose --level=5 --raid-devices=4 /dev/md4 /dev/sdb5 /dev/sdc5 /dev/sdd5 /dev/sde5 
```

makes use of a few mdadm defaults.

Like

--chunk= ... its now 512k, it used to be 64k

--metadata= its now 1.2, it used to be 0.90

The raid metadata that you need to create is a data structure that points to your data.  Getting --chunk= incorrect, is harmless, you can have an many goes as you want but it must be correct to allow the kernel to read the filesystem on the raid.  It tells how big the individual data elements are on the drive, so reading 512k at a time, when its actually 64k doesn't work.

The --metadata= is rather more important.  It tells where the raid metadata is on the underlying block device. If its wrong, it will either overwrite the end of your filesystem or the filesystem (not raid)

I was considering --create in degraded mode possibly with --assume-clean ... depending what 

```
mdadm -E /dev/sd[abcd]5
```

 shows and what you believe is on each partition also with the explicit --chunk= and --metadata= values that 

```
mdadm -E /dev/sd[abcd]5
```

will show.

Its also important to choose the 'best' 3 of the four elements from the raid set. 

```
$ sudo mdadm -E /dev/sda5

Password: 

/dev/sda5:

          Magic : a92b4efc

        Version : 0.90.00       <----

           UUID : 5e3cadd4:cfd2665d:96901ac7:6d8f5a5d

  Creation Time : Sat Apr 11 20:30:16 2009

     Raid Level : raid5

  Used Dev Size : 5253120 (5.01 GiB 5.38 GB)

     Array Size : 15759360 (15.03 GiB 16.14 GB)

   Raid Devices : 4

  Total Devices : 4

Preferred Minor : 126

    Update Time : Sun Mar 16 11:02:16 2014

          State : clean

 Active Devices : 4

Working Devices : 4

 Failed Devices : 0

  Spare Devices : 0

       Checksum : 78b17729 - correct

         Events : 77

         Layout : left-symmetric

     Chunk Size : 64K       <-----

```

I've highlighted my --chunk= and --metadata= above.

----------

## remix

Chunk Size : 64K

Metadata : 0.90.00

```

# mdadm -E /dev/sd[abcd]5

/dev/sda5:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : bfb60f39:66b601a2:fbf2ea5a:12bfd232 (local to host OptimusPrime)

  Creation Time : Mon Mar  8 22:09:01 2010

     Raid Level : raid5

  Used Dev Size : 175791104 (167.65 GiB 180.01 GB)

     Array Size : 527373312 (502.94 GiB 540.03 GB)

   Raid Devices : 4

  Total Devices : 4

Preferred Minor : 4

    Update Time : Fri Apr 18 18:23:26 2014

          State : clean

 Active Devices : 2

Working Devices : 2

 Failed Devices : 2

  Spare Devices : 0

       Checksum : 87adfde0 - correct

         Events : 17379

         Layout : left-symmetric

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State

this     0       8        5        0      active sync   /dev/sda5

   0     0       8        5        0      active sync   /dev/sda5

   1     1       0        0        1      faulty removed

   2     2       8       37        2      active sync   /dev/sdc5

   3     3       0        0        3      faulty removed

mdadm: No md superblock detected on /dev/sdb5.

/dev/sdc5:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : bfb60f39:66b601a2:fbf2ea5a:12bfd232 (local to host OptimusPrime)

  Creation Time : Mon Mar  8 22:09:01 2010

     Raid Level : raid5

  Used Dev Size : 175791104 (167.65 GiB 180.01 GB)

     Array Size : 527373312 (502.94 GiB 540.03 GB)

   Raid Devices : 4

  Total Devices : 4

Preferred Minor : 4

    Update Time : Fri Apr 18 18:23:26 2014

          State : clean

 Active Devices : 2

Working Devices : 2

 Failed Devices : 2

  Spare Devices : 0

       Checksum : 87adfe04 - correct

         Events : 17379

         Layout : left-symmetric

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State

this     2       8       37        2      active sync   /dev/sdc5

   0     0       8        5        0      active sync   /dev/sda5

   1     1       0        0        1      faulty removed

   2     2       8       37        2      active sync   /dev/sdc5

   3     3       0        0        3      faulty removed

/dev/sdd5:

          Magic : a92b4efc

        Version : 0.90.00

           UUID : bfb60f39:66b601a2:fbf2ea5a:12bfd232 (local to host OptimusPrime)

  Creation Time : Mon Mar  8 22:09:01 2010

     Raid Level : raid5

  Used Dev Size : 175791104 (167.65 GiB 180.01 GB)

     Array Size : 527373312 (502.94 GiB 540.03 GB)

   Raid Devices : 4

  Total Devices : 4

Preferred Minor : 4

    Update Time : Fri Apr 18 18:22:24 2014

          State : active

 Active Devices : 3

Working Devices : 3

 Failed Devices : 1

  Spare Devices : 0

       Checksum : 87adb9e3 - correct

         Events : 17375

         Layout : left-symmetric

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State

this     3       8       53        3      active sync   /dev/sdd5

   0     0       8        5        0      active sync   /dev/sda5

   1     1       0        0        1      faulty removed

   2     2       8       37        2      active sync   /dev/sdc5

   3     3       8       53        3      active sync   /dev/sdd5
```

----------

## NeddySeagoon

remix,

First of all, understand that something will be corrupt but we have no idea what.

As it stands, that raid set should assemble and run with the --force option, if not, we need te rewrite the raid meta data, which does nothing to the user data on the raid.

Its just like rewriting a partition table.

We must be sure to pass the chunk size and metadata versions to mdadm --create as 64k and 0.90 are no longer the defaults and we want to recreate the raid metadata as was, so your data reappears.

```
/dev/sda5: 

    Update Time : Fri Apr 18 18:23:26 2014 

         Events : 17379 

/dev/sdc5:

    Update Time : Fri Apr 18 18:23:26 2014

          Events : 17379 

/dev/sdd5: 

      Update Time : Fri Apr 18 18:22:24 2014 

         Events : 17375 
```

Notice the update times and Event Counts /dev/sdd5 is a few writes behind.  They may be anthing.  Also /dev/sdb5 is missing.

Before you go any further, understand that assembling the raid and mounting any filesystem it may contain are separate operations.

Getting the raid assembled is a prerequsite to reading the filesystem but depending on whats damaged, there may be further steps to get at your data.

If all else fails ...

```
mdadm --create /dev/md4 --metadata=0.90 --raid-devices=4 --chunk=64 --level=raid5 --assume-clean /dev/sda5 missing  /dev/sdc5 /dev/sdd5
```

Before you do that, make sure you understand what it is trying to do.

After it completes 

```
mdadm -E /dev/sd[abcd]5 
```

should show

```
      Number   Major   Minor   RaidDevice State

   0     0       8        5        0      active sync   /dev/sda5

   1     1       0        0        1      missing

   2     2       8       37        2      active sync   /dev/sdc5

   3     3       8       53        3      active sync   /dev/sdd5

```

Its important that the partitions are in the same slots.

The Event counts will all be zero and the raid should be assembled and running.  Look in /proc/mdstat. 

So far so good.  The next step is to try to mount /dev/md4 read only and look around.

```
mount -o ro /dev/md4 /mnt/someplace
```

There are lots of reason that can fail and a few things to try to fix it.

Do not be tempted to run fsck.  That makes guesses about what to do and often does the wrong thing.

You are not ready to allow any writes to the filesystem yet, even if it mounts.

----------

