# mdadm RAID5 with 4x2TB drives

## cwall64

running into a strange scenario with RAID5 and a 8TB array.  mdadm seems to always fail on /dev/sdd1, same machine but with 4x1TB drives works fine.  Is there a limit on a RAID5 mdadm built array?  Forgot to mention that all drives do work independently, and pass SMART test, drives were partitioned with one 1.82 TB raid partition.

```

mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1

```

I have used all the available version of mdadm...

```

gentoonas ~ # eix mdadm

[I] sys-fs/mdadm

     Available versions:  2.6.8 (~)2.6.9-r1 3.0 (~)3.0.2 (~)3.0.2-r1 (~)3.0.3 (~)3.1.1 (~)3.1.1-r1 {static}

     Installed versions:  3.1.1-r1(19:17:03 02/17/10)(-static)

     Homepage:            http://neil.brown.name/blog/mdadm

     Description:         A useful tool for running RAID systems - it can be used as a replacement for the raidtools

```

```

gentoonas ~ # uname -ar

Linux gentoonas 2.6.31-gentoo-r6 #1 SMP PREEMPT Wed Feb 17 18:58:17 CST 2010 x86_64 Intel(R) Core(TM)2 Duo CPU E7200 @ 2.53GHz GenuineIntel GNU/Linux

```

System has 4GB of memory.

```

gentoonas ~ # emerge --info

Portage 2.1.7.16 (default/linux/amd64/10.0/desktop, gcc-4.3.4, glibc-2.10.1-r1, 2.6.31-gentoo-r6 x86_64)

=================================================================

System uname: Linux-2.6.31-gentoo-r6-x86_64-Intel-R-_Core-TM-2_Duo_CPU_E7200_@_2.53GHz-with-gentoo-1.12.13

Timestamp of tree: Thu, 18 Feb 2010 12:45:01 +0000

app-shells/bash:     4.0_p35

dev-java/java-config: 2.1.10

dev-lang/python:     2.6.4

dev-util/cmake:      2.6.4-r3

sys-apps/baselayout: 1.12.13

sys-apps/sandbox:    1.6-r2

sys-devel/autoconf:  2.13, 2.63-r1

sys-devel/automake:  1.9.6-r2, 1.10.3, 1.11.1

sys-devel/binutils:  2.18-r3

sys-devel/gcc:       4.3.4

sys-devel/gcc-config: 1.4.1

sys-devel/libtool:   2.2.6b

virtual/os-headers:  2.6.27-r2

ACCEPT_KEYWORDS="amd64"

ACCEPT_LICENSE="* -@EULA"

CBUILD="x86_64-pc-linux-gnu"

CFLAGS="-march=core2 -msse4.1 -O2 -pipe"

CHOST="x86_64-pc-linux-gnu"

CONFIG_PROTECT="/etc /usr/share/X11/xkb"

CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/udev/rules.d"

CXXFLAGS="-march=core2 -msse4.1 -O2 -pipe"

DISTDIR="/usr/portage/distfiles"

FEATURES="assume-digests distlocks fixpackages news parallel-fetch protect-owned sandbox sfperms strict unmerge-logs unmerge-orphans userfetch"

GENTOO_MIRRORS="ftp://gentoo.mirrors.tds.net/gentoo ftp://mirror.iawnet.sandia.gov/pub/gentoo/ ftp://ftp.gtlib.gatech.edu/pub/gentoo "

LDFLAGS="-Wl,-O1"

LINGUAS="en"

MAKEOPTS="-j3"

PKGDIR="/usr/portage/packages"

PORTAGE_CONFIGROOT="/"

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"

PORTAGE_TMPDIR="/var/tmp"

PORTDIR="/usr/portage"

SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"

USE="X a52 aac acl acpi alsa amd64 automount avahi banshee berkdb bluetooth branding bzip2 cairo cdr cli consolekit cracklib crypt cxx daap dbus dri dts dvd dvdr eds emboss encode evo extras fam ffmpeg firefox flac fortran fuse gdbm gif gnome gnomecd gpm gstreamer gtk hal iconv ipv6 java jpeg kdrive ldap libnotify libsamplerate mad mikmod mmx mng modules monomusicbrainz mp3 mp4 mpeg mudflap multilib nautilus ncurses nls nptl nptlonly nsplugin ogg opengl openmp pam pcre pdf perl png policykit ppds pppd pulseaudio python qt4 quicktime readline reflection samba sasl sdl session slp spell spl sse sse2 ssl startup-notification svg sysfs tcl tcpd thunar tiff truetype unicode usb vorbis x264 xcomposite xml xorg xulrunner xv xvid zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="nvidia"

Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY

```

----------

## NeddySeagoon

cwall64,

What error message do you get?

----------

## cwall64

Neddy,

It would just go to about 13.9% recovery then stop during the "watch cat /proc/mdstat" about 30 minutes into it.  And when I would do a mdadm -D /dev/md0 it would report that /dev/sdd1 failed and /dev/sde1 was spare (which i did not setup that way).  Unfortunately I threw a new drive in to the machine (/dev/sda to replace the Gentoo sda drive), and loaded openSuSE on it to see if any difference.  If/once that fails I will throw Gentoo drive back in to test anything i find in the forum...

----------

## cwall64

looks like SuSE fails too:

```

freenas:~ # mdadm -D /dev/md0

/dev/md0:

        Version : 1.00

  Creation Time : Thu Feb 18 04:39:01 2010

     Raid Level : raid5

     Array Size : 5860535424 (5589.04 GiB 6001.19 GB)

  Used Dev Size : 1953511808 (1863.01 GiB 2000.40 GB)

   Raid Devices : 4

  Total Devices : 4

    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Thu Feb 18 06:01:04 2010

          State : active, degraded

 Active Devices : 2

Working Devices : 3

 Failed Devices : 1

  Spare Devices : 1

         Layout : left-asymmetric

     Chunk Size : 128K

           Name : freenas:0  (local to host freenas)

           UUID : 017d26b7:7b440f81:5fd4fd8d:fbc734e3

         Events : 796

    Number   Major   Minor   RaidDevice State

       0       8       17        0      active sync   /dev/sdb1

       1       8       33        1      active sync   /dev/sdc1

       2       0        0        2      removed

       3       0        0        3      removed

       2       8       49        -      faulty spare   /dev/sdd1

       4       8       65        -      spare   /dev/sde1

```

So, both are giving same error, but if I create a partition and format /dev/sdd1 with ext4, i can share it out with samba/afp and read/write files to it all day?

Spoke too soon, looks like /var/log/messages does have some ATA errors for sdd.

----------

## cwall64

```

Feb 18 20:35:58 gentoonas smartd[15785]: Device: /dev/sda, SMART Usage Attribute: 194 Temperature_Celsius changed from 124 to 121

Feb 18 20:35:58 gentoonas smartd[15785]: Device: /dev/sdb, SMART Usage Attribute: 194 Temperature_Celsius changed from 240 to 206

Feb 18 20:35:59 gentoonas smartd[15785]: Device: /dev/sdc, SMART Usage Attribute: 194 Temperature_Celsius changed from 240 to 206

Feb 18 20:36:00 gentoonas smartd[15785]: Device: /dev/sdd, 13 Currently unreadable (pending) sectors

Feb 18 20:36:00 gentoonas smartd[15785]: Device: /dev/sdd, SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 97 to 100

Feb 18 20:36:00 gentoonas smartd[15785]: Device: /dev/sdd, SMART Usage Attribute: 194 Temperature_Celsius changed from 250 to 214

Feb 18 20:36:01 gentoonas smartd[15785]: Device: /dev/sde, SMART Usage Attribute: 194 Temperature_Celsius changed from 250 to 214

Feb 18 20:36:02 gentoonas smartd[15785]: Device: /dev/sdf, SMART Usage Attribute: 194 Temperature_Celsius changed from 121 to 114

Feb 18 20:36:02 gentoonas smartd[15785]: Device: /dev/sdg, SMART Usage Attribute: 194 Temperature_Celsius changed from 122 to 117

Feb 18 20:40:01 gentoonas cron[17478]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )

Feb 18 20:50:01 gentoonas cron[17802]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )

Feb 18 20:56:09 gentoonas kernel: ata4.00: exception Emask 0x0 SAct 0x7e SErr 0x0 action 0x0

Feb 18 20:56:09 gentoonas kernel: ata4.00: irq_stat 0x40000008

Feb 18 20:56:09 gentoonas kernel: ata4.00: cmd 60/00:08:47:ed:33/01:00:20:00:00/40 tag 1 ncq 131072 in

Feb 18 20:56:09 gentoonas kernel: res 51/40:3a:0d:ee:33/00:00:20:00:00/00 Emask 0x409 (media error) <F>

Feb 18 20:56:09 gentoonas kernel: ata4.00: status: { DRDY ERR }

Feb 18 20:56:09 gentoonas kernel: ata4.00: error: { UNC }

Feb 18 20:56:09 gentoonas kernel: ata4.00: configured for UDMA/133

Feb 18 20:56:09 gentoonas kernel: ata4: EH complete

Feb 18 20:56:12 gentoonas kernel: ata4.00: exception Emask 0x0 SAct 0xff SErr 0x0 action 0x0

Feb 18 20:56:12 gentoonas kernel: ata4.00: irq_stat 0x40000008

Feb 18 20:56:12 gentoonas kernel: ata4.00: cmd 60/00:28:47:ed:33/01:00:20:00:00/40 tag 5 ncq 131072 in

Feb 18 20:56:12 gentoonas kernel: res 51/40:3a:0d:ee:33/93:00:20:00:00/00 Emask 0x409 (media error) <F>

Feb 18 20:56:12 gentoonas kernel: ata4.00: status: { DRDY ERR }

Feb 18 20:56:12 gentoonas kernel: ata4.00: error: { UNC }

Feb 18 20:56:12 gentoonas kernel: ata4.00: configured for UDMA/133

Feb 18 20:56:12 gentoonas kernel: ata4: EH complete

Feb 18 20:56:15 gentoonas kernel: ata4.00: exception Emask 0x0 SAct 0xff SErr 0x0 action 0x0

Feb 18 20:56:15 gentoonas kernel: ata4.00: irq_stat 0x40000008

Feb 18 20:56:15 gentoonas kernel: ata4.00: cmd 60/00:10:47:ed:33/01:00:20:00:00/40 tag 2 ncq 131072 in

Feb 18 20:56:15 gentoonas kernel: res 51/40:3a:0d:ee:33/93:00:20:00:00/00 Emask 0x409 (media error) <F>

Feb 18 20:56:15 gentoonas kernel: ata4.00: status: { DRDY ERR }

Feb 18 20:56:15 gentoonas kernel: ata4.00: error: { UNC }

Feb 18 20:56:15 gentoonas kernel: ata4.00: configured for UDMA/133

Feb 18 20:56:15 gentoonas kernel: ata4: EH complete

Feb 18 20:56:17 gentoonas kernel: ata4.00: exception Emask 0x0 SAct 0xfe SErr 0x0 action 0x0

Feb 18 20:56:17 gentoonas kernel: ata4.00: irq_stat 0x40000008

Feb 18 20:56:17 gentoonas kernel: ata4.00: cmd 60/00:28:47:ed:33/01:00:20:00:00/40 tag 5 ncq 131072 in

Feb 18 20:56:17 gentoonas kernel: res 51/40:3a:0d:ee:33/93:00:20:00:00/00 Emask 0x409 (media error) <F>

Feb 18 20:56:17 gentoonas kernel: ata4.00: status: { DRDY ERR }

Feb 18 20:56:17 gentoonas kernel: ata4.00: error: { UNC }

Feb 18 20:56:17 gentoonas kernel: ata4.00: configured for UDMA/133

Feb 18 20:56:17 gentoonas kernel: ata4: EH complete

Feb 18 20:56:20 gentoonas kernel: ata4.00: exception Emask 0x0 SAct 0xff SErr 0x0 action 0x0

Feb 18 20:56:20 gentoonas kernel: ata4.00: irq_stat 0x40000008

Feb 18 20:56:20 gentoonas kernel: ata4.00: cmd 60/00:10:47:ed:33/01:00:20:00:00/40 tag 2 ncq 131072 in

Feb 18 20:56:20 gentoonas kernel: res 51/40:3a:0d:ee:33/93:00:20:00:00/00 Emask 0x409 (media error) <F>

Feb 18 20:56:20 gentoonas kernel: ata4.00: status: { DRDY ERR }

Feb 18 20:56:20 gentoonas kernel: ata4.00: error: { UNC }

Feb 18 20:56:20 gentoonas kernel: ata4.00: configured for UDMA/133

Feb 18 20:56:20 gentoonas kernel: ata4: EH complete

Feb 18 20:56:23 gentoonas kernel: ata4.00: exception Emask 0x0 SAct 0xfe SErr 0x0 action 0x0

Feb 18 20:56:23 gentoonas kernel: ata4.00: irq_stat 0x40000008

Feb 18 20:56:23 gentoonas kernel: ata4.00: cmd 60/00:28:47:ed:33/01:00:20:00:00/40 tag 5 ncq 131072 in

Feb 18 20:56:23 gentoonas kernel: res 51/40:3a:0d:ee:33/93:00:20:00:00/00 Emask 0x409 (media error) <F>

Feb 18 20:56:23 gentoonas kernel: ata4.00: status: { DRDY ERR }

Feb 18 20:56:23 gentoonas kernel: ata4.00: error: { UNC }

Feb 18 20:56:23 gentoonas kernel: ata4.00: configured for UDMA/133

Feb 18 20:56:23 gentoonas kernel: sd 3:0:0:0: [sdd] Unhandled sense code

Feb 18 20:56:23 gentoonas kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

Feb 18 20:56:23 gentoonas kernel: sd 3:0:0:0: [sdd] Sense Key : Medium Error [current] [descriptor]

Feb 18 20:56:23 gentoonas kernel: Descriptor sense data with sense descriptors (in hex):

Feb 18 20:56:23 gentoonas kernel: 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 

Feb 18 20:56:23 gentoonas kernel: 20 33 ee 0d 

Feb 18 20:56:23 gentoonas kernel: sd 3:0:0:0: [sdd] Add. Sense: Unrecovered read error - auto reallocate failed

Feb 18 20:56:23 gentoonas kernel: end_request: I/O error, dev sdd, sector 540274189

Feb 18 20:56:23 gentoonas kernel: raid5:md0: read error not correctable (sector 540274120 on sdd1).

Feb 18 20:56:23 gentoonas kernel: raid5: Disk failure on sdd1, disabling device.

Feb 18 20:56:23 gentoonas kernel: raid5: Operation continuing on 2 devices.

Feb 18 20:56:23 gentoonas kernel: raid5:md0: read error not correctable (sector 540274128 on sdd1).

Feb 18 20:56:23 gentoonas kernel: raid5:md0: read error not correctable (sector 540274136 on sdd1).

Feb 18 20:56:23 gentoonas kernel: raid5:md0: read error not correctable (sector 540274144 on sdd1).

Feb 18 20:56:23 gentoonas kernel: raid5:md0: read error not correctable (sector 540274152 on sdd1).

Feb 18 20:56:23 gentoonas kernel: raid5:md0: read error not correctable (sector 540274160 on sdd1).

Feb 18 20:56:23 gentoonas kernel: raid5:md0: read error not correctable (sector 540274168 on sdd1).

Feb 18 20:56:23 gentoonas kernel: raid5:md0: read error not correctable (sector 540274176 on sdd1).

Feb 18 20:56:23 gentoonas kernel: ata4: EH complete

Feb 18 20:56:23 gentoonas kernel: md: md0: recovery done.

Feb 18 20:56:23 gentoonas kernel: RAID5 conf printout:

Feb 18 20:56:23 gentoonas kernel: --- rd:4 wd:2

Feb 18 20:56:23 gentoonas kernel: disk 0, o:1, dev:sdb1

Feb 18 20:56:23 gentoonas kernel: disk 1, o:1, dev:sdc1

Feb 18 20:56:23 gentoonas kernel: disk 2, o:0, dev:sdd1

Feb 18 20:56:23 gentoonas kernel: disk 3, o:1, dev:sde1

Feb 18 20:56:23 gentoonas kernel: RAID5 conf printout:

Feb 18 20:56:23 gentoonas kernel: --- rd:4 wd:2

Feb 18 20:56:23 gentoonas kernel: disk 0, o:1, dev:sdb1

Feb 18 20:56:23 gentoonas kernel: disk 1, o:1, dev:sdc1

Feb 18 20:56:23 gentoonas kernel: disk 2, o:0, dev:sdd1

Feb 18 20:56:23 gentoonas kernel: RAID5 conf printout:

Feb 18 20:56:23 gentoonas kernel: --- rd:4 wd:2

Feb 18 20:56:23 gentoonas kernel: disk 0, o:1, dev:sdb1

Feb 18 20:56:23 gentoonas kernel: disk 1, o:1, dev:sdc1

Feb 18 20:56:23 gentoonas kernel: disk 2, o:0, dev:sdd1

Feb 18 20:56:23 gentoonas kernel: RAID5 conf printout:

Feb 18 20:56:23 gentoonas kernel: --- rd:4 wd:2

Feb 18 20:56:23 gentoonas kernel: disk 0, o:1, dev:sdb1

Feb 18 20:56:23 gentoonas kernel: disk 1, o:1, dev:sdc1

```

So I will go exchange the drive today and  see if this solves the issue...

----------

