# EVMS recovery problem [SOLVED]

## hardhead

I have a RAID5 managed by EVMS using LVM2.

Earlier today the system went down unexpectedly and when the box came up one of the disks had a bad block so the raid is running in degraded mode.  Unfortunately, when I try to add a spare disk to the array--I get a failure message:

```
_0_ Engine: engine_user_message: Message is: Engine: Plug-in MDRaid5RegMgr returned error 19 when committing changes for region md/md1 during phase 3.
```

I began to research this error message--and I *think* the problem is the MD object isn't activating.  Without that activation, I can not manage the RAID (so I can't add a spare).  When running a simple activate command in the CLI, the following appears in my evms-engine.log file:

```
_5_ Engine: is_object_change_pending: Change pending: Object lvm2/media/home needs to be activated.

_5_ Engine: is_object_change_pending: Change pending: Object lvm2/media/media needs to be activated.

_5_ Engine: is_object_change_pending: Change pending: Object md/md1 needs to be activated.

_5_ Engine: is_volume_change_pending: Change pending: Volume /dev/evms/home needs to be activated.

_5_ Engine: is_volume_change_pending: Change pending: Volume /dev/evms/media needs to be activated.

_3_ Engine: engine_ioctl_object: ioctl to object md/md1 failed with error code 5: Input/output error

_2_ MDRaid5RegMgr: md_ioctl_run_array: md_ioctl_run_array: IOCTL failed, region(md/md1) rc(5)

_3_ Engine: activate_object: Error code 5 activating object md/md1: Input/output error

_3_ Engine: engine_ioctl_object: ioctl to object md/md1 failed with error code 16: Device or resource busy
```

In /var/log/messages I see lots of:

```
md: array md1 already has disks!
```

When I run mdadm --detail /dev/md1 I get:

```
/dev/md1:

        Version : 00.90.03

  Creation Time : Mon May 15 20:55:42 2006

     Raid Level : raid5

    Device Size : 244195840 (232.88 GiB 250.06 GB)

   Raid Devices : 7

  Total Devices : 6

Preferred Minor : 1

    Persistence : Superblock is persistent

    Update Time : Sat Oct  6 22:26:37 2007

          State : active, degraded

 Active Devices : 6

Working Devices : 6

 Failed Devices : 0

  Spare Devices : 0

         Layout : left-asymmetric

     Chunk Size : 256K

           UUID : 66774afe:e3d47cc5:57d24caa:8eb4f32b

         Events : 0.30322321

    Number   Major   Minor   RaidDevice State

       0     253        0        0      active sync   /dev/evms/.nodes/hdc1

       1       0        0        -      removed

       2     253        1        2      active sync   /dev/evms/.nodes/sdb1

       3     253        2        3      active sync   /dev/evms/.nodes/sdc1

       4     253        3        4      active sync   /dev/evms/.nodes/sdd1

       5     253        4        5      active sync   /dev/evms/.nodes/sde1

       6     253        5        6      active sync   /dev/evms/.nodes/sdf1
```

I have tried mdadm --stop /dev/md1 before running evms (or evmsn) and I keep running into the same problem.  Any help on this issue would be much appreciated.  I know I *could* potentially rebuild the raid with mdadm -a /dev/sda1, but I'm not sure how EVMS would react to that, and I'm not sure my two volumes stored on the raid (/dev/evms/media and /dev/evms/home) would remain intact. 

Also--when I run ls /dev/evms/.nodes I see:

```
hdc  hdc1  sda  sdb  sdb1  sdc  sdc1  sdd  sdd1  sde  sde1  sdf  sdf1
```

I notice sda1 doesn't yet exist, so I can't manually use mdadm to add sda1.

Please let me know if any more information would be helpful--and if there is any way to solve this problem.

Thanks,

-Michael BeamLast edited by hardhead on Sun Oct 07, 2007 5:32 am; edited 2 times in total

----------

## drescherjm

 *Quote:*   

> I notice sda1 doesn't yet exist, so I can't manually use mdadm to add sda1. 

 

If sda1 does not exist then you need to partition the sda drive so that the sda1 partition is the exact same size as the other raid members. I know you will need to do this if you were using mdadm maybe evms requires this as well.

You can do this with fdisk or sfdisk.

If sdb and sda are the same size this should do the trick:

```
sfdisk -d /dev/sdb | sfdisk /dev/sda 
```

----------

## hardhead

Great, I got /dev/evms/.nodes/sda1 to appear.  According to mdadm things look a little better:

```
    Update Time : Sat Oct  6 22:26:37 2007

          State : active, degraded

 Active Devices : 6

Working Devices : 7

 Failed Devices : 0

  Spare Devices : 1

         Layout : left-asymmetric

     Chunk Size : 256K

           UUID : 66774afe:e3d47cc5:57d24caa:8eb4f32b

         Events : 0.30322321

    Number   Major   Minor   RaidDevice State

       0     253        0        0      active sync   /dev/evms/.nodes/hdc1

       1     253        6        1      spare rebuilding   /dev/evms/.nodes/sda1

       2     253        1        2      active sync   /dev/evms/.nodes/sdb1

       3     253        2        3      active sync   /dev/evms/.nodes/sdc1

       4     253        3        4      active sync   /dev/evms/.nodes/sdd1

       5     253        4        5      active sync   /dev/evms/.nodes/sde1

       6     253        5        6      active sync   /dev/evms/.nodes/sdf1

```

BUT---the rebuild doesn't appear to be automatically started.  Does anyone know how to force the rebuild--or to make the sda1 disk "active" rather than "spare"

/proc/mdstat yeilds:

```
Personalities : [raid5] 

md1 : inactive dm-6[1] dm-5[6] dm-4[5] dm-3[4] dm-2[3] dm-1[2] dm-0[0]

      1709370880 blocks

       

unused devices: <none>
```

I think that's supposed to be active now?  I am not seeing any percentage progress in the mdadm --detail display

----------

## drescherjm

The problem is the raid is inactive.

How about:

```
mdadm --run /dev/md1
```

Or there may be an evms command to do that but I do not use evms.

Also a simple reboot will probably do this as well since you now have sda1 the degraded array should fix itself.

----------

## hardhead

 *drescherjm wrote:*   

> The problem is the raid is inactive.
> 
> How about:
> 
> ```
> ...

 

I get

```
mdadm: failed to run array /dev/md1: Input/output error

```

In /var/log/messages I get:

```
raid5: device dm-5 operational as raid disk 6

raid5: device dm-4 operational as raid disk 5

raid5: device dm-3 operational as raid disk 4

raid5: device dm-2 operational as raid disk 3

raid5: device dm-1 operational as raid disk 2

raid5: device dm-0 operational as raid disk 0

raid5: cannot start dirty degraded array for md1

RAID5 conf printout:

 --- rd:7 wd:6 fd:1

 disk 0, o:1, dev:dm-0

 disk 1, o:1, dev:dm-6

 disk 2, o:1, dev:dm-1

 disk 3, o:1, dev:dm-2

 disk 4, o:1, dev:dm-3

 disk 5, o:1, dev:dm-4

 disk 6, o:1, dev:dm-5

raid5: failed to run raid set md1

md: pers->run() failed ...
```

----------

## drescherjm

Has the /proc/mdstat changed?

----------

## drescherjm

This may help:

http://www.linuxquestions.org/questions/linux-general-1/raid5-with-mdadm-does-not-ron-or-rebuild-505361/

From this I am most interested if your events are all the same for all but the new spare disk.

----------

## hardhead

Unfortunately I can't run a mdadm --assemble because the config isn't setup in /etc/mdadm.conf, rather EVMS manages the volume.  I'm a little wary of setting everything up in mdadm.conf and running the assemble--because even in md1 comes up--will EVMS still be able to manage the two volumes that md has been divided into? (/dev/evms/media and /dev/evms/home)....

It seems like the crash/reassemble one disk situation is somewhat common with software RAID users--so I'm hoping to find someone with EVMS experience with this.

----------

## hardhead

Using some search terms from the link you sent I stumbled upon: 

http://evms.sourceforge.net/irc/2005-Jul/%23evms-2005-Jul-06

Which had my solution:

```
mdadm --stop /dev/md1

mdadm -A /dev/md1 -f /dev/evms/.nodes/hdc1 /dev/evms/.nodes/sdb1 /dev/evms/.nodes/sdc1 /dev/evms/.nodes/sdd1 /dev/evms/.nodes/sde1 /dev/evms/.nodes/sdf1 

```

This activated the RAID and then I was able to add the spare disc using EVMS.

The RAID is back up and running and rebuilding.

Thanks so much for all your help tonight!

----------

## drescherjm

I was very sure that would work. Thanks for posting back. At times I have wanted to use evms however it does not support raid6 (patch did not work for me) which I use almost exclusively at work. I may try it again when raid6 support is officially added.

----------

