# raid1: Disk failure - smartctl can't find anything, fault?

## mijenix

Hi

Today my raid1 had again the following problem (the error occurred 2 times until now):

 *Quote:*   

> 
> 
> Nov 19 22:03:57 reese kernel: raid1: sdc1: rescheduling sector 1953187152
> 
> Nov 19 22:03:57 reese kernel: raid1: sdc1: rescheduling sector 1953187400
> ...

 

now it only runs with one device. 

I ran smartctl -t short and -t long but it can't find any error, following the smartctl output:

```

smartctl --all /dev/sdc

smartctl 5.39.1 2010-01-28 r3054 [x86_64-pc-linux-gnu] (local build)

Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===

Model Family:     Western Digital Caviar Green family

Device Model:     WDC WD10EADS-00L5B1

Serial Number:    WD-WCAU46928032

Firmware Version: 01.01A01

User Capacity:    1,000,204,886,016 bytes

Device is:        In smartctl database [for details use: -P show]

ATA Version is:   8

ATA Standard is:  Exact ATA specification draft version not indicated

Local Time is:    Fri Nov 26 23:45:45 2010 CET

SMART support is: Available - device has SMART capability.

SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===

SMART overall-health self-assessment test result: PASSED

General SMART Values:

Offline data collection status:  (0x82)   Offline data collection activity

               was completed without error.

               Auto Offline Data Collection: Enabled.

Self-test execution status:      (   0)   The previous self-test routine completed

               without error or no self-test has ever 

               been run.

Total time to complete Offline 

data collection:        (22200) seconds.

Offline data collection

capabilities:           (0x7b) SMART execute Offline immediate.

               Auto Offline data collection on/off support.

               Suspend Offline collection upon new

               command.

               Offline surface scan supported.

               Self-test supported.

               Conveyance Self-test supported.

               Selective Self-test supported.

SMART capabilities:            (0x0003)   Saves SMART data before entering

               power-saving mode.

               Supports SMART auto save timer.

Error logging capability:        (0x01)   Error logging supported.

               General Purpose Logging supported.

Short self-test routine 

recommended polling time:     (   2) minutes.

Extended self-test routine

recommended polling time:     ( 255) minutes.

Conveyance self-test routine

recommended polling time:     (   5) minutes.

SCT capabilities:           (0x303f)   SCT Status supported.

               SCT Feature Control supported.

               SCT Data Table supported.

SMART Attributes Data Structure revision number: 16

Vendor Specific SMART Attributes with Thresholds:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE

  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0

  3 Spin_Up_Time            0x0027   163   162   021    Pre-fail  Always       -       6816

  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       304

  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0

  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0

  9 Power_On_Hours          0x0032   095   095   000    Old_age   Always       -       4358

 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0

 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0

 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       302

192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       4

193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       304

194 Temperature_Celsius     0x0022   115   108   000    Old_age   Always       -       35

196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0

197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0

198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0

199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1

No Errors Logged

SMART Self-test log structure revision number 1

Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error

# 1  Extended offline    Completed without error       00%      4358         -

# 2  Short offline       Completed without error       00%      4354         -

SMART Selective self-test log data structure revision number 1

 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS

    1        0        0  Not_testing

    2        0        0  Not_testing

    3        0        0  Not_testing

    4        0        0  Not_testing

    5        0        0  Not_testing

Selective self-test flags (0x0):

  After scanning selected spans, do NOT read-scan remainder of disk.

If Selective self-test is pending on power-up, resume after 0 minute delay.

```

The first time the error occured I re-added the disk to the raid and it worked for 2 months.

But I'm a little bit worried, any ideas what the problem can be, what do you think?

Thanks in advice.

----------

## frostschutz

Smart is not everything... try a long self test, also try accessing these sectors directly using dd seek, see if it works or not. If it does work, maybe you have an erratic cable or controller. Otherwise, the disk is just plain bad and smart is not smart enough to know about it.

----------

## mijenix

I re-added the device to the raid and got this

```

ov 27 10:24:25 reese kernel: md: bind<sdc1>

Nov 27 10:24:25 reese kernel: RAID1 conf printout:

Nov 27 10:24:25 reese kernel: --- wd:1 rd:2

Nov 27 10:24:25 reese kernel: disk 0, wo:0, o:1, dev:sdb1

Nov 27 10:24:25 reese kernel: disk 1, wo:1, o:1, dev:sdc1

Nov 27 10:24:25 reese kernel: md: recovery of RAID array md1

Nov 27 10:24:25 reese kernel: md: minimum _guaranteed_  speed: 1000 KB/sec/disk.

Nov 27 10:24:25 reese kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.

Nov 27 10:24:25 reese kernel: md: using 128k window, over a total of 976759936 blocks.

Nov 27 10:24:25 reese mdadm[1988]: RebuildStarted event detected on md device /dev/md1

Nov 27 10:29:48 reese kernel: ata6.00: exception Emask 0x50 SAct 0x7fffffff SErr 0xc80900 action 0x6 frozen

Nov 27 10:29:48 reese kernel: ata6.00: irq_stat 0x08000000, interface fatal error

Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:00:bf:e2:3e/00:00:03:00:00/40 tag 0 ncq 65536 out

Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:08:3f:e3:3e/00:00:03:00:00/40 tag 1 ncq 65536 out

Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:10:bf:e4:3e/00:00:03:00:00/40 tag 2 ncq 65536 out

Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:18:3f:e5:3e/00:00:03:00:00/40 tag 3 ncq 65536 out

Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:20:bf:e5:3e/00:00:03:00:00/40 tag 4 ncq 65536 out

Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:28:3f:e6:3e/00:00:03:00:00/40 tag 5 ncq 65536 out

Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:30:bf:e6:3e/00:00:03:00:00/40 tag 6 ncq 65536 out

Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:38:3f:e7:3e/00:00:03:00:00/40 tag 7 ncq 65536 out

Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:40:bf:e7:3e/00:00:03:00:00/40 tag 8 ncq 65536 out

Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:48:3f:e8:3e/00:00:03:00:00/40 tag 9 ncq 65536 out

Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:50:bf:e8:3e/00:00:03:00:00/40 tag 10 ncq 65536 out

Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:58:3f:e9:3e/00:00:03:00:00/40 tag 11 ncq 65536 out

Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:60:bf:e9:3e/00:00:03:00:00/40 tag 12 ncq 65536 out

Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:68:3f:ea:3e/00:00:03:00:00/40 tag 13 ncq 65536 out

Nov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:29:48 reese kernel: ata6.00: cmd 61/80:70:bf:ea:3e/00:00:03:00:00/40 tag 14 ncq 65536 out

...

ov 27 10:29:48 reese kernel: res 40/00:8c:3f:ec:3e/00:00:03:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:29:48 reese kernel: ata6: hard resetting link

Nov 27 10:29:49 reese kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

Nov 27 10:29:49 reese kernel: ata6.00: configured for UDMA/133

Nov 27 10:29:49 reese kernel: ata6: EH complete

Nov 27 10:30:01 reese cron[3898]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )

Nov 27 10:35:46 reese kernel: ata6.00: exception Emask 0x50 SAct 0x7ffeffff SErr 0xc80900 action 0x6 frozen

Nov 27 10:35:46 reese kernel: ata6.00: irq_stat 0x08000000, interface fatal error

Nov 27 10:35:46 reese kernel: ata6.00: cmd 61/80:00:3f:62:dd/00:00:06:00:00/40 tag 0 ncq 65536 out

Nov 27 10:35:46 reese kernel: res 40/00:7c:bf:63:dd/00:00:06:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:35:46 reese kernel: ata6.00: cmd 61/80:08:bf:62:dd/00:00:06:00:00/40 tag 1 ncq 65536 out

...

Nov 27 10:35:46 reese kernel: res 40/00:7c:bf:63:dd/00:00:06:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:35:46 reese kernel: ata6: hard resetting link

Nov 27 10:35:47 reese kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

Nov 27 10:35:47 reese kernel: ata6.00: configured for UDMA/133

Nov 27 10:35:47 reese kernel: ata6: EH complete

Nov 27 10:36:14 reese kernel: ata6.00: exception Emask 0x50 SAct 0x1ffff SErr 0xc80900 action 0x6 frozen

Nov 27 10:36:14 reese kernel: ata6.00: irq_stat 0x08000000, interface fatal error

...

Nov 27 10:36:14 reese kernel: res 40/00:24:3f:1f:25/00:00:07:00:00/40 Emask 0x50 (ATA bus error)

Nov 27 10:36:14 reese kernel: ata6: hard resetting link

Nov 27 10:36:15 reese kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

Nov 27 10:36:15 reese kernel: ata6.00: configured for UDMA/133

Nov 27 10:36:15 reese kernel: ata6: EH complete

Nov 27 10:37:52 reese kernel: ata6: limiting SATA link speed to 1.5 Gbps

Nov 27 10:37:52 reese kernel: ata6.00: exception Emask 0x10 SAct 0x7fffbfff SErr 0x480100 action 0x6 frozen

Nov 27 10:37:52 reese kernel: ata6.00: irq_stat 0x08000000, interface fatal error

Nov 27 10:37:52 reese kernel: ata6.00: cmd 61/80:00:3f:ef:1f/00:00:08:00:00/40 tag 0 ncq 65536 out

...

Nov 27 10:37:54 reese kernel: res 40/00:3c:bf:36:20/00:00:08:00:00/40 Emask 0x52 (ATA bus error)

Nov 27 10:37:54 reese kernel: ata6: hard resetting link

Nov 27 10:37:54 reese kernel: ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 310)

Nov 27 10:37:54 reese kernel: ata6.00: failed to IDENTIFY (I/O error, err_mask=0x100)

Nov 27 10:37:54 reese kernel: ata6.00: revalidation failed (errno=-5)

Nov 27 10:37:59 reese kernel: ata6: hard resetting link

Nov 27 10:38:00 reese kernel: ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 310)

Nov 27 10:38:00 reese kernel: ata6.00: configured for UDMA/133

Nov 27 10:38:00 reese kernel: ata6: EH complete

Nov 27 10:38:00 reese kernel: ata6.00: exception Emask 0x50 SAct 0x7ffffff SErr 0xc80900 action 0x6 frozen

Nov 27 10:38:00 reese kernel: ata6.00: irq_stat 0x08000000, interface fatal error

Nov 27 10:38:00 reese kernel: ata6.00: cmd 61/80:00:3f:4d:20/00:00:08:00:00/40 tag 0 ncq 65536 out

...

Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Unhandled error code

Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Result: hostbyte=0x04 driverbyte=0x00

Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] CDB: cdb[0]=0x2a: 2a 00 08 21 72 3f 00 01 00 00

Nov 27 10:38:28 reese kernel: end_request: I/O error, dev sdc, sector 136409663

Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Unhandled error code

Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Result: hostbyte=0x04 driverbyte=0x00

Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] CDB: cdb[0]=0x2a: 2a 00 08 21 71 3f 00 00 80 00

Nov 27 10:38:28 reese kernel: end_request: I/O error, dev sdc, sector 136409407

...

Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] READ CAPACITY(16) failed

Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Result: hostbyte=0x04 driverbyte=0x00

Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Sense not available.

Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] READ CAPACITY failed

Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Result: hostbyte=0x04 driverbyte=0x00

Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Sense not available.

Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Asking for cache data failed

Nov 27 10:38:28 reese kernel: sd 5:0:0:0: [sdc] Assuming drive cache: write through

Nov 27 10:38:28 reese kernel: sdc: detected capacity change from 1000204886016 to 0

Nov 27 10:38:29 reese kernel: RAID1 conf printout:

Nov 27 10:38:29 reese kernel: --- wd:1 rd:2

Nov 27 10:38:29 reese kernel: disk 0, wo:0, o:1, dev:sdb1

Nov 27 10:38:29 reese kernel: disk 1, wo:1, o:0, dev:sdc1

Nov 27 10:38:29 reese kernel: RAID1 conf printout:

Nov 27 10:38:29 reese kernel: --- wd:1 rd:2

Nov 27 10:38:29 reese kernel: disk 0, wo:0, o:1, dev:sdb1

Nov 27 10:38:29 reese mdadm[1988]: RebuildFinished event detected on md device /dev/md1

Nov 27 10:38:29 reese mdadm[1988]: FailSpare event detected on md device /dev/md1, component device /dev/sdc1

```

I plugged the sata-cable out and in and started the server, now the raid is rebuilding. 

When the error occurs again I'll change the faulty disk.

Do you think the controller can be faulty? 

On that controller the system disk (sda) and the non faulty raid disk (sdb) is attached. Those disks had not any problems until now.

----------

## NeddySeagoon

mijenix,

If the fault has cleared, its probably a problem with poor quality connectors on the data cable. It will be OK for about a year, then it will happen again - due to the poor quality connectors.

----------

## mijenix

so after the resync the raid1 seems fine.

I know the cable problem from my eSATA backupdrive. 

The cable hasn't any plug fixation but the external disk is more affected due to agitation then the internal. 

Are there any better sata cables then the mobo provided ones?

Thanks again for all your help.

----------

## NeddySeagoon

mijenix,

Almost any SATA cables are better than motherboard provided ones. 

SATA cables are a commodity item. Look at cheap ones, look at expensive ones, then buy in the middle of the price range, or just a little above.

----------

