# Raid1 problem

## frising

Hi!

I'm running a gentoo server with 2 x Maxtor 6Y160P0, ATA DISK drive (160 GB each). The disks are attached to the ide0 and ide1 controllers on the moterboard. A cd-burner that is only used when booting the system with a Live CD is also attached to ide1. I use software raid1 on the disk as follows:

cat /proc/mdstat

 *Quote:*   

> 
> 
> Personalities : [raid1]
> 
> md1 : active raid1 hdc1[1] hda1[0]
> ...

 

When I checked my email this morning I got a message from mdadm that said "A Fail event had been detected on md device /dev/md6.". So /proc/mdstat looked like:

 *Quote:*   

> 
> 
> Personalities : [raid1]
> 
> md1 : active raid1 hdc1[1] hda1[0]
> ...

 

Yesterday morning I had a failure on md5:

 *Quote:*   

> 
> 
> Personalities : [raid1]
> 
> md1 : active raid1 hdc1[1] hda1[0]
> ...

 

I have also had the same problem 2 times earlier, in December and April 2005. Note that it's always hdc that's failing. I have solved the problem temporarly by:

 *Quote:*   

> 
> 
> 1. Reboot
> 
> 2. Format hdcX:
> ...

 

The disk performance looks as follows:

tux ~ # for ((i=1 ; i<9 ; i++)) ; do hdparm -Tt /dev/hda$i && hdparm -Tt /dev/hdc$i && hdparm -Tt /dev/md$i ; done

 *Quote:*   

> 
> 
> /dev/hda1:
> 
>  Timing cached reads:   1064 MB in  2.00 seconds = 530.91 MB/sec
> ...

 

What shall I do? Please help!

Regards

Philip

----------

## NeddySeagoon

frising,

I don't see any problems in your post. Your drives are zoned. That means there are more sectors per track at the outside of the platter than at the inside. This accounts for the higher data rates on partitions near the outside of the platter than near the inside.

Your first step should be to 

```
emerge smartmontools
```

 and read the drives internal status log.

This will help provide an early warning of failures.

Next time it happens, look in dmesg. That will show you what the kernel saw.

----------

## frising

 *NeddySeagoon wrote:*   

> 
> 
> frising,
> 
> I don't see any problems in your post. Your drives are zoned. That means there are more sectors per track at the outside of the platter than at the inside. This accounts for the higher data rates on partitions near the outside of the platter than near the inside.
> ...

 

Thanks for your reply! 

Yeah, I can see why the drives speed differs on different parts of the disk. But I think it's strange that hda5 is much faster than hdc5. I bought both of the drives at the same time and they were as fast when they were new.

 *NeddySeagoon wrote:*   

> 
> 
> Your first step should be to 
> 
> ```
> ...

 

smartctl -a /dev/hdc6 outputs following information:

 *Quote:*   

> 
> 
> smartctl version 5.33 [i686-pc-linux-gnu] Copyright (C) 2002-4 Bruce Allen
> 
> Home page is http://smartmontools.sourceforge.net/
> ...

 

If I run the same command with /dev/hda6 as arg, I get no errors. I don't realy understand the output. It says "Commands leading to the command that caused the error were:" "READ DMA EXT". Does this means that the disk is not working properly? Or is it the ide controller?

 *NeddySeagoon wrote:*   

> 
> 
> Next time it happens, look in dmesg. That will show you what the kernel saw.
> 
> 

 

According to my logs following happended when hdc5 failed the day before yesterday:

 *Quote:*   

> 
> 
> Feb  3 03:13:07 tux hdc: dma_intr: status=0x51 { DriveReady SeekComplete Error }
> 
> Feb  3 03:13:07 tux hdc: dma_intr: error=0x40 { UncorrectableError }, LBAsect=176722936, high=10, low=8950776, sector=176722936
> ...

 

hdc6 failed at the same time but yesterday. I run emerge sync as a cron job around 03:00 and use logcheck and logrotate to send the latest log file to my email.

If I run hdparm -Tt /dev/hdcX when there's a raid failure I get loads errors (for ex seek errors) in the log files. The computer freezes util I kill the hdparm process. If I then format the paritition and add it to the raid array hdparm works as expected and there's no seek errors.

Please explain what I should do in order to "fix" this.

----------

## NeddySeagoon

frising,

```
Feb 3 03:13:07 tux hdc: dma_intr: status=0x51 { DriveReady SeekComplete Error }

Feb 3 03:13:07 tux hdc: dma_intr: error=0x40 { UncorrectableError }, LBAsect=176722936, high=10, low=8950776, sector=176722936 
```

Tells that a DMA error occured.

Please use code tags rather than quote tags when you post output. It preserves the formatting by using a fixed space font, so columns line up. The lower speeds returned from hdc6 (compared to hda6) can be due to several reasons.

1. hdc6 will have different surface defects to hda6. Both will appear to have no bad blocks to the operating system because the drive hides bad blocks present at manufacture and grown through life by mapping in spare sectors. Accessing these out of order sectors takes extra time (due to head movements) and reduces the data rate. 

2. hdc6 may be doing retries

Please post the output of 

```
hdparm /dev/hdc
```

so I can see the options you have set.

At the moment, it appears that a few sectors on /dev/hdc6 cannot be read. Making a new filesystem on /dev/hdc6 does not fix that becasue the partition is not tested. I suspect you can provoke the error with

```
dd if=/dev/hdc6 of=/dev/null
```

This will read the entire partition and discard the data that was read.

If it fails, take /dev/hdc6 out of the raid set, write to it all, which will force the drive to remap and really bad sectors, then try the dd again. If it works, put it back in the raid set. If not, you probably have another problem.

----------

## frising

 *NeddySeagoon wrote:*   

> 
> 
> At the moment, it appears that a few sectors on /dev/hdc6 cannot be read. Making a new filesystem on /dev/hdc6 does not fix that becasue the partition is not tested. I suspect you can provoke the error with
> 
> ```
> ...

 

Thanks for your reply!

I have tested running

```
dd if=/dev/hdc5 of=/dev/null
```

and

```
dd if=/dev/hdc6 of=/dev/null
```

Both works. There's no error in logs and smartctl -a /dev/hdc looks as before.

Output from hdparm, hdc:

```

/dev/hdc:

 multcount    = 16 (on)

 IO_support   =  1 (32-bit)

 unmaskirq    =  1 (on)

 using_dma    =  1 (on)

 keepsettings =  0 (off)

 readonly     =  0 (off)

 readahead    = 256 (on)

 geometry     = 19929/255/63, sectors = 163928604672, start = 0

```

hda:

```

/dev/hda:

 multcount    = 16 (on)

 IO_support   =  1 (32-bit)

 unmaskirq    =  1 (on)

 using_dma    =  1 (on)

 keepsettings =  0 (off)

 readonly     =  0 (off)

 readahead    = 256 (on)

 geometry     = 19929/255/63, sectors = 163928604672, start = 0

```

 *NeddySeagoon wrote:*   

> 
> 
> If it fails, take /dev/hdc6 out of the raid set, write to it all, which will force the drive to remap and really bad sectors, then try the dd again. If it works, put it back in the raid set. If not, you probably have another problem.
> 
> 

 

How should I have done "write to it all"? Is there any smart program that can fill the partition (including the space that's reserved for the filesystem?)?

I ran

```

smartctl -t long /dev/hdc 

```

but smartctl found no errors.

Do you have any other idea what I can test?

----------

## NeddySeagoon

frising,

You have 

```
unmaskirq    =  1 (on) 
```

This is a bad idea. It allows other lower priority IRQs to interrupt

disk IRQs that are already in progress. In extreme cases, the disk IRQ can time out and create errors like the ones you see.

You can turn it off with hdparm or by rebuilding your kernel. Please post your /proc/interrups so I can see how your IRQs are used.

You can write to a partition, including all the space outside of the filesystem with

```
dd if=/dev/zero of=/dev/hdc6 
```

be very very careful with that command.

It can write the whole drive just as easily.

/dev/zero ia an endless supply of zero bytes

/dev/urandom is an endless supply of psudo random bytes

----------

## frising

 *NeddySeagoon wrote:*   

> frising,
> 
> You have 
> 
> ```
> ...

 

Thanks for you reply!

Ok, I wasn't aware of that. Wouldn't it be best to keep it in kernel and then disable the option on hda and hdc and then keep it enabled on hdd (cdburner)? My /proc/interrupts:

```

           CPU0

  0:   10825395    IO-APIC-edge  timer

  1:        152    IO-APIC-edge  i8042

  8:          2    IO-APIC-edge  rtc

  9:          1   IO-APIC-level  acpi

 14:     183055    IO-APIC-edge  ide0

 15:    1193472    IO-APIC-edge  ide1

 16:    1222477   IO-APIC-level  eth0

 17:    3135680   IO-APIC-level  eth1

 18:     183014   IO-APIC-level  eth2

 20:     296732   IO-APIC-level  ehci_hcd:usb1, uhci_hcd:usb2, uhci_hcd:usb3, uhci_hcd:usb4

NMI:          0

LOC:   10824798

ERR:          0

MIS:          0

```

 *NeddySeagoon wrote:*   

> 
> 
> You can write to a partition, including all the space outside of the filesystem with
> 
> ```
> ...

 

Okey, I see. Thanks for the info!

----------

## NeddySeagoon

frising,

Hmm thats intersting. On the AT, that uses two cascaded PICs the priority is

```

0   Timer

1   Keyboard

8   Real Time Clock

9

10

11

12  Mouse

13  Floating Point Exception

14  IDE0

15  IDE1

3   Serial Port

4   Serial Port

5   Parallel Port

6

7

```

With the APIC, I'm not sure what the priority order is but I guess that all your ethernet and USB IRQs can interrupt

your disk IRQs at the moment.

By all means turn off unmaskirq with hdparm if you wish. Its up to you how you configure your system.

Both achieve the same effect. 

The original intent of the option was to allow the serial port IRQs to be processed during disk I/O to avoid missed characters in the days when the serial port did not have a FIFO. You clearly don't need that.

----------

## frising

 *NeddySeagoon wrote:*   

> 
> 
> By all means turn off unmaskirq with hdparm if you wish. Its up to you how you configure your system.
> 
> Both achieve the same effect. 
> ...

 

Thanks for the help! 

I have modified my hdparm config. I also removed hdc from the raid array, wrote 0:s to the whole disk, fixed the partition table so that it matched hda, formated the hdc partitions and inserted it into the raid array again. I hope this will help. As I wrote before, the time between the first and the second time the problem occured happened was about 6 months. I'll post if another failure occurs.

----------

