# HORRIBLE Disk to Disk copy/move performance

## zakl

My hardware setup is as follows...

Athlon XP 1700+

EpoX 8KHA+ (via KT266a chipset)

768MB PC2100 (kingston)

IDE Channel 1 (using ATA133 cable):

Master - IBM 40GB ATA100

Slave - Plextor 40/12/40A Burner

IDE Channel 2 (using ATA100 cable): 

Master - Western Digital WD1200BB 120GB ATA100

Slave - Sony 16x DVD

The WD 120 is formatted at max capacity as FAT32, while the IBM is split in half... W2k Pro takes first half of drive, followed by /boot par, root par, and swap par.  Gentoo's half gives swap 1GB and the rest goes to root par.

This is the problem, when I copy/move from drive to drive, I average 2-3MB/s!  I started a 4GB move, and it started at 27MB/s for about 6 seconds then dropped to 3MB/s.  Every once in a while it will bounce up to 6MB/s, but that's it.  It takes me almost a minute and a half to move a 200MB file.  I know it's not my hardware setup, because in W2K, I can move a 200MB file between the drives in 10-15 seconds, which is how it should be.

hdparm -Tt reports:

root@Logical yoyo # hdparm -Tt /dev/hdc

/dev/hdc:

 Timing buffer-cache reads:   128 MB in  0.44 seconds =290.91 MB/sec

 Timing buffered disk reads:  64 MB in  1.38 seconds = 46.38 MB/sec

root@Logical yoyo # hdparm -Tt /dev/hda

/dev/hda:

 Timing buffer-cache reads:   128 MB in  0.44 seconds =290.91 MB/sec

 Timing buffered disk reads:  64 MB in  1.83 seconds = 34.97 MB/sec

hdparm -d shows same thing on both drives:

 using_dma    =  1 (on)

I can post my hdparm -d results if requested. 

Thanks, 

Zak

----------

## taskara

hmm.. as a test, try disconnecting your dvd and burner and have JUST the hard drives connected.

let me know if it improves at all.

----------

## amne

which kernel are you using? i have the same motherboard and there seemed to be some problems with the chipset, i think. in 2.4.18 copying large files sometimes even froze the system for a few seconds and the copying didn't go on in that time as well, 2.4.19 was better and 2.4.20 (all vanilla) works pretty fine for me.

----------

## zakl

I'm using ck-sources kernel 2.4.20-ck6.  I haven't got a chance to disconnect my drives, but I will ASAP.  

Thanks for the help, 

Zak

----------

## zakl

Could it be possible that my round ATA133 cable is causing this slowdown?  I have a 24" ATA133 cable on channel 0.  

I've always read in the past that IDE cables have a maximum length of 18", and anything longer causes attenuation.  Also, my IBM is at the very end of the cable, which means it's 24" from the mother board.  This is something I haven't thought about till now.  

Is it possible that the length of that cable is causeing my problems?

Zak

----------

## flokno

i have had the same or a similar problem.

have a look here: https://forums.gentoo.org/viewtopic.php?p=239841#239841

after disabling this validation bit setting some weeks ago (now i don't think that it is necessary) and carefully disabling all not necessary ide support in ide config things are working without a problem now. have a look at every help-text available there and in case of doubt follow recommendations.

i can post my kernel config if necessary though i only use vanilla-sources-2.4.21.

-- flo

----------

## zakl

Thanks for the tip.  It was not enabled, so I enabled it and recompiled my kernel.  I will reboot to test it as soon as OpenOffice 1.1 finishes compiling, and post the results here. 

And man does OpenOffice take a long time to compile!  It's been going for over 12 hours now on an Athlon XP 1700+, and used up 2GB of disk space.

----------

## zakl

Nope, that kernel option made no difference. The thing that gets me is it's only when I try to copy between drives in the machine.  I have two other machines on the LAN, and I can copy from them over the network @ 10MB/s to either drive.  But when if I copy between the two drives it's stuck around 3-4MB.  I'm going to try what taskara requested, as well as taking out the 24" IDE cable.

I'll post my results here.

Zak

----------

## zakl

Ok, I disconnected both of my CD drives, and performed a copy test.  Now drive to drive copy is between 10-30MB/s.  Why would my CD drives be causing conflict?  I did notice however, that hdparm gives me an error whenever I try to use it with my Plextor burner.  Says something about not being compatible, or hdparm does not work with drive.  I'll get the exact error when I hook my drives back up.  Could that be the problem?

I should have known based on the fact that windows didn't have the same problem, that it was not my IDE cable length.

What can I do?  I move LOTS of files between my hard drives, and 3MB/s is just not gonna cut it.  And operating my system with no CD drives isn't gonna cut it either.  Please help me out.

----------

## zakl

Ok, this is tripped out, but after reconnecting my CD drives, it's still copying at 10-30MB/s now.  No complaining here.

The error hdparm was giving me for my plextor is as follows.

root@logical banji # hdparm -d /dev/cdroms/cdrom1

/dev/cdroms/cdrom1 not supported by hdparm

But if performance stays the same, it's all good.  

Taskara, thanks for the test, I have no clue what was wrong, but reconnecting the drives seems to have fixed it.

Zak

----------

## taskara

groovy  :Smile: 

----------

## MasterX

Sorry to bring this up, but I did not quite understand how you fixed your problem. 

My problem is that when I am emerging files or when I am copying/moving the system almost freezes. The pointer does not move, I can not change desktops etc.

The hdparm -d gave 

using_dma    =  0 (off)

how can I change it to 4 (This is what I have in the Bios)

Can I do the same thing for the CD driver/Dvd

----------

## taskara

you won't change it for a burner or dvd if you're using scsi emulation, and for your hdd you don't want to change  *Quote:*   

> using_dma = 0 (off)

  to 4, just to 1 but you want to change the access mode to 4 (ata66)

if so then this if what you want to run on your device

```
hdparm -c1 -d1 -m16 -X68 -k1 /dev/hda
```

then check the output again and test with 

```
hdparm -tT /dev/hda
```

----------

## taskara

btw if you're using ata100, then subsitute -X69

----------

## MasterX

Taskara, thanks for the info.

This is what I got

/dev/hdb:

 setting 32-bit IO_support flag to 1

 setting multcount to 16

 setting using_dma to 1 (on)

 HDIO_SET_DMA failed: Operation not permitted

 setting keep_settings to 1 (on)

 setting xfermode to 69 (UltraDMA mode5)

 HDIO_DRIVE_CMD(setxfermode) failed: Input/output error

 multcount    = 16 (on)

 IO_support   =  1 (32-bit)

 using_dma    =  0 (off)

 keepsettings =  1 (on)

So, I was not able to change it

----------

## MasterX

If I need to umount /dev/hdb in order hdprm to  work, how can I do that since I am using Linux and /dev/hdb has to be mounted

Thanks

----------

## taskara

if you can't enable dma, it sounds like the driver for your ide controller is not compiled into your kernel, or you don't have dma for hard disks enabled.

check your kernel config again.

what is your ide controller?

----------

## MasterX

I looked at the config file for the kernel

In the section IDE,ATA and Atari support, the following options are checked

Generic PCI bus-master DMA support

Use PCI-DMA by deafult when available

Is this what you were talking about?

How can I find the Ide-controler?

Thanks

----------

## taskara

what is the model and brand of your mainboard?

----------

## MasterX

Asus P4Pe with intel 845 chipset

----------

## taskara

if you plugged your hdd into the main ide channel, do you have 

```
IntelPIIXn chipsets support
```

support compiled in to your kernel? and if so, as a module or directly in?

----------

## MasterX

No, it is included, I will built in and let you know

----------

## MasterX

I can not find the PIIXn Tuning support. Is that a problem?

----------

## MasterX

Thanks a lot Taskara. Now using_dma is on

----------

## taskara

you shoudl not need tuning  :Smile:  just the other built directly in

----------

## taskara

you should not need tuning  :Smile:  just the other built directly in

edit: you're welcome  :Smile: 

what's your output of [code]hdparm -tT /dev/hdb[./code] ?

----------

## zakl

I LIED!!! It was a complete fluke.  The problem is not fixed! Maybe I just need to buy a new motherboard with a better chipset.  This is pissing me off.  I copy an average of 3-5GB at a time between drives, and these crappy 2-3MB speeds I'm getting in Linux are rediculous.  It's getting frustrating now, I can copy from my FTP server across the network faster than I can copy between local ATA 100 hard drives.  This is weak.  Maybe the KT266a chipset just doesn't have good IDE support for Linux.

Ex. 

5GB copy from FTP to local machine: 9-11MB/s solid.

5GB copy from hda to hdc: 4MB/s MAX, an average of 2-3MB/s.  

Sometimes it spikes on different files, and I get 20MB/s for a couple seconds then it flops back down.

This is making me sick.  It shouldn't take me 40 minutes to move 3GB of files to my other hard drive.

----------

## taskara

maybe one of the drives is faulty?  :Wink: 

what if u ftp a file down to ur other drive, do u get the same speeds?

have u tried different hdd's?

kt266 shoudl ahve good linux support.

even if it doesn't (which it does) the generic ide chipset should have better performance!

is dma ON? I can't remember..

----------

## alec

I had a similar problem a while back - grabbing from the LAN was quicker than grabbing from the hard drive - but just including every motherboard in the IDE section of my kernel and using hdparm did the trick - guess that won't help you.  I have heard that 2.4.21 boasts signifigant IDE performance increases - perhaps using a 2.4.21 kernel instead of a 2.4.20 kernel would help?  I'm running -ac right now, as it is the only kernel aside from the vanilla I saw that is 2.4.21.

Few more questions - have you used other linuxes or Windows with which you measured the performance you're looking for?  If you boot up a LiveCD and do some copying, do you get better performance?  Did the `/etc/init.d/hdparm` script not help?

----------

## zakl

taskara: 

dma is on

32bit transfer on

active mode is udma5

And it can't be the drives because speeds in Win2K are on par... 20MB+/s drive to drive copy/move.

alec:

I will try a newer kernel.  Probably the ac-sources or the pfeifer-sources. 

Yes, I do/have used windows and other linux distro's, I don't have any tests from the linux's, but see above for the windows results.

hdparm is installed, and didn't improve performance at all.

I might try the live CD after a bit, but currently I have stuff compiling via ssh in Konsole, so I can't reboot right now.

----------

## zakl

This may sound a bit off to some of you, but I have been playing around and I just tried something new.  I honestly think it is the filesystem on my 120GB drive.  I formatted the whole drive as FAT32, which technically FAT32 is suppose to be able to handle much larger partitions, but I think there are bugs and flaws, being the reason that Microsoft no longer suggests it be used in todays windows installations.  

Anyway, this is what I did.  I started some rather large bit torrent downloads, each torrent in excess of 1.5GB.  I decided to save them directly to my storage drive(the 120GB) this time, and when I saved them, it took forever in the space allocation.  This leads me to believe that it is the file system giving me problems.  Whenever I start large torrents like this in NTFS or reiserfs, the space is allocated almost instantly, as where it took almost 2 minutes to allocate on my FAT32 drive.  Also, every time I boot to windows, windows wants to run a check on the partition.  Although this could mean there are errors and they simply need to be fixed, it's not that simple, because even after fixing them it happens again.  So, as soon as I get money to buy 80GB worth of CD's, I will be burning all my anime and formatting it as reiserfs.  

I'll update this post when that happens to let yall know if it was in fact my problem.

----------

## MasterX

There was a discussion in another post about enabling the dma on a scsi. The solution was to look at the /proc/ide/hdX/settings. This is what I get for my hard disk

```

name                    value           min             max             mode

----                    -----           ---             ---             ----

acoustic                0               0               254             rw

address                 0               0               2               rw

bios_cyl                1867            0               65535           rw

bios_head               255             0               255             rw

bios_sect               63              0               63              rw

breada_readahead        8               0               255             rw

bswap                   0               0               1               r

current_speed           69              0               70              rw

failures                0               0               65535           rw

file_readahead          124             0               16384           rw

init_speed              69              0               70              rw

io_32bit                0               0               3               rw

keepsettings            0               0               1               rw

lun                     0               0               7               rw

max_failures            1               0               65535           rw

max_kb_per_request      128             1               255             rw

multcount               16              0               16              rw

nice1                   1               0               1               rw

nowerr                  0               0               1               rw

number                  1               0               3               rw

pio_mode                write-only      0               255             w

slow                    0               0               1               rw

unmaskirq               0               0               1               rw

using_dma               1               0               1               rw

wcache                  0               0               1               rw

```

It seems to me that I can change the current speed to 70 and max_kb_per_request to 255 do you know if this will improve the performance? Moreover how can I change these with hdparm  :Rolling Eyes: 

----------

## zakl

I now am 100% sure it is a problem with the FAT32 file system.  I downloaded from my ftp server to that partition, and it is downloading at 4MB/s.  As to where when I download from the ftp server to any other partitions(W2K NTFS, or linux reiserfs) it goes at 9-11MB/s.

----------

## darkangael

I have the same problem. 

I am using 

Athlon 2400XP+

Epox 8k9a2+ motherboard (KT400)

512 meg DDR333

Pioneer DVD-ROM 

Pioneer DVR-106 DVD Burner

Fujitsu MPE 3136AT 13 gig HDD(Slave)

MAXTOR 6L080J4 80 gig HDD(Master)

Copy between /home (ext3) and /mnt/winstor (fat32) is abominable. It is really slow, maxing at 20 meg/sec and averaging 3-8 meg. It also slows down the whole system while it does it (with no added CPU usage I should add).

I have DMA turned on, and the xfermode is whatever the default is (supposedly it's detected properly by default, and I am pretty sure at some earlier stage last time I tried to solve this I checked it out).

 I agree totally that there is something wrong with the fat32 driver, or the filesystem itself. I have no troubles in windows with the same sort of copying (using explore2fs from my /home partition to fat32) EDIT: I just realised that it could also not be, I am no sure but I think compiling does similar things to performance, but not as bad). 

By the way: The /homes and /mnt/winstor are on the same hard disk. I have the same problems copying from another partition on another disk which is also fat32.

I am using gentoo-sources 2.4.2(1?). I will test the theory on the fat32 when I boot back to linux after using my windows only programs.

----------

## pilla

 *MasterX wrote:*   

> I can not find the PIIXn Tuning support. Is that a problem?

 

Not all kernel versions have it. You should be OK without it too.

----------

## darkangael

Definitely not FAT32 problem for me. It also sucks when copying /usr/src/linux to /home/darkangael

hdparm -tT gives

```

/dev/hda:

 Timing buffer-cache reads:   128 MB in  0.52 seconds =246.15 MB/sec

 Timing buffered disk reads:  64 MB in  1.69 seconds = 37.87 MB/sec

```

hdparm -I gives

```

/dev/hda:

ATA device, with non-removable media

        Model Number:       MAXTOR 6L080J4

        Serial Number:      664216412353

        Firmware Revision:  A93.0500

Standards:

        Used: ATA/ATAPI-5 T13 1321D revision 1

        Supported: 5 4 3 2 & some of 6

Configuration:

        Logical         max     current

        cylinders       16383   16383

        heads           16      16

        sectors/track   63      63

        --

        CHS current addressable sectors:   16514064

        LBA    user addressable sectors:  156355584

        device size with M = 1024*1024:       76345 MBytes

        device size with M = 1000*1000:       80054 MBytes (80 GB)

Capabilities:

        LBA, IORDY(can be disabled)

        bytes avail on r/w long: 4      Queue depth: 1

        Standby timer values: spec'd by Vendor, no device specific minimum

        R/W multiple sector transfer: Max = 16  Current = 16

        Recommended acoustic management value: 128, current value: 254

        DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6

             Cycle time: min=120ns recommended=120ns

        PIO: pio0 pio1 pio2 pio3 pio4

             Cycle time: no flow control=120ns  IORDY flow control=120ns

Commands/features:

        Enabled Supported:

           *    READ BUFFER cmd

           *    WRITE BUFFER cmd

           *    Host Protected Area feature set

           *    Look-ahead

           *    Write cache

           *    Power Management feature set

                Security Mode feature set

           *    SMART feature set

           *    Mandatory FLUSH CACHE command

           *    Device Configuration Overlay feature set

           *    Automatic Acoustic Management feature set

                SET MAX security extension

           *    DOWNLOAD MICROCODE cmd

           *    SMART self-test

           *    SMART error logging

Security:

        Master password revision code = 65534

                supported

        not     enabled

        not     locked

        not     frozen

        not     expired: security count

        not     supported: enhanced erase

        40min for SECURITY ERASE UNIT.

HW reset results:

        CBLID- above Vih

        Device num = 0 determined by the jumper

Checksum: correct

```

It turns out I am using 2.4.20 kernel. Gonna test it out with the 2.6 kernel I built a while ago as my opengl performance is better with it (although nwn still crashes, and somewhat harder I think  :Sad: )

----------

## darkangael

using 2.6.0-test6-mm1 I get really strange results. Everything appears to be quicker, but konqueror says it's still 3.6 meg/sec and gkrellm says 0. Gkrellm says that the CPU goes up to 100% as well, which is just wrong. top says that the cpu usage is "iowait". Anyone know what this is? It never used to use CPU. It's still using udma6 mode and DMA is on like before.

hdparm -tT gives

```

/dev/hda:

 Timing buffer-cache reads:   128 MB in  0.45 seconds =285.76 MB/sec

 Timing buffered disk reads:  64 MB in  1.79 seconds = 35.74 MB/sec

```

----------

## ww9rivers

I would also like to share my bit of expierence regarding disk performance issue.

I have 3 disks in a Dell 400SC box, using gs-sources-2.4.23_pre8. I got very bad disk performances. I tweaked a few "hdparm" settings but things went only slightly better. After reading this thread and went into my kernel settings, I found out that I had the PIIx support compiled as a module, which caused problems when I tried to turn on "using_dma" with hdparm.

After "modprobe piix", I was able to turn on "using_dma" which improved "hdparm -Tt" output of "buffered disk reads" from less than 5MB/sec to above 56MB/sec.  :Smile: 

----------

