# High performance in benchmarks, HORRIBLE in real world

## Flintz

Hi there, I have a really odd problem here:

I have a server with a single IDE drive and a Raid5-drvie.

Now I noticed a horrible performance when copying a file from the ide to the raid drive. So I tracked down the problem: The ide drive has an horrible performance (5MB/s). BUT only in real world scenarios, when using hdparm -tT /dev/hda oder piozone or bonnie I get very acceptable results. Thats not very logical for me. DMA is of course activated, here are a few outputs:

```
hdparm /dev/hda

/dev/hda:

 multcount    = 16 (on)

 IO_support   =  1 (32-bit)

 unmaskirq    =  1 (on)

 using_dma    =  1 (on)

 keepsettings =  0 (off)

 readonly     =  0 (off)

 readahead    = 256 (on)

 geometry     = 24321/255/63, sectors = 390721968, start = 0

```

```
piozone /dev/hda

[PIOZONE, version 1.0 - Copyright (c) 2002 Peter Eriksson <pen@lysator.liu.se>]

Linear read transfer rates:

48.0 MB/s at offset   0 GB using  64 KiB reads

46.4 MB/s at offset  68 GB using  64 KiB reads

35.6 MB/s at offset 136 GB using  64 KiB reads

32.7 MB/s at offset 204 GB using  64 KiB reads

35.4 MB/s at offset 272 GB using  64 KiB reads

40.1 MB/s at offset 340 GB using  64 KiB reads

Testing... /

```

```
hdparm -tT /dev/hda

/dev/hda:

 Timing cached reads:   2828 MB in  2.00 seconds = 1413.91 MB/sec

 Timing buffered disk reads:  156 MB in  3.00 seconds =  52.00 MB/sec

```

```
dd count=200000 if=/file_on_hda of=/dev/null

200000+0 Datensätze ein

200000+0 Datensätze aus

102400000 Bytes (102 MB) kopiert, 24,4871 Sekunden, 4,2 MB/s

```

"cp -g" also gives around 4,2 to 4,7 MB/s, completely inacceptable!! But I have absolutely no idea whats causing this, I played around a bit with the accoustic management setting in hdparm, but its set again to 254 (most speed) and I don't think that this should cause such a big drop in performance.

----------

## NeddySeagoon

Flintz,

Tell us about the raid and show us your /proc/interrups.  

```
unmaskirq    =  1 (on) 
```

is often a bad thing to set.

Also look in dmesg for evidence of DMA timeouts and the kernel reverting to PIO for the IDE drive.

----------

## Flintz

Heya, first thanks for the quick response!

Well I didn't set unmaskingirq on purpose, actually I didn't ever mess around the with settings on /dev/hda (with exception for the AAM setting).

The raid drive is a Raid5 of 5 disks on a Promise EX8350 hardware controller. I thought that this might be causing the problem, but it isn't (at least I think), thats why I think this:

1.) if I rund the dd command I wrote in my first post, I bypass the raid device and still get bad numbers

2.) when I fetch something from my two smb-shares (one on the IDE drive, one on the raid-drive) I also encounter this speed issues, from the one I get the well known 4-5 MB/s, from the other 15-20 MB/s

This is the output of my interrupts, should be everything fine, every device has its own IRQ:

```
cat /proc/interrupts

           CPU0

  0:     274886    IO-APIC-edge  timer

  8:          0    IO-APIC-edge  rtc

  9:          0   IO-APIC-level  acpi

 14:      28109    IO-APIC-edge  ide0

 50:        837   IO-APIC-level  shasta

 58:      37900   IO-APIC-level  NVidia CK804

 66:      79674   IO-APIC-level  fglrx

217:          2   IO-APIC-level  ehci_hcd:usb1

225:      17638   IO-APIC-level  ohci_hcd:usb2

233:     162369   IO-APIC-level  eth0

NMI:        298

LOC:     274857

ERR:          0

MIS:          0

```

A few searches in dmesg:

```
dmesg | grep PIO

ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx

```

```
dmesg | grep DMA

  DMA zone: 2690 pages, LIFO batch:0

  DMA32 zone: 254505 pages, LIFO batch:31

PCI-DMA: Disabling IOMMU.

NFORCE-CK804: 0000:00:06.0 (rev f2) UDMA133 controller

    ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:DMA, hdb:DMA

    ide1: BM-DMA at 0xf008-0xf00f, BIOS settings: hdc:DMA, hdd:DMA

hda: 390721968 sectors (200049 MB) w/8192KiB Cache, CHS=24321/255/63, UDMA(100)

hdb: ATAPI 52X DVD-ROM drive, 256kB Cache, UDMA(33)

```

```
dmesg | grep hda

Bootdata ok (command line is root=/dev/hda3 vga=0x31B)

Kernel command line: root=/dev/hda3 vga=0x31B

    ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:DMA, hdb:DMA

hda: SAMSUNG SP2014N, ATA DISK drive

hda: max request size: 1024KiB

hda: 390721968 sectors (200049 MB) w/8192KiB Cache, CHS=24321/255/63, UDMA(100)

hda: cache flushes supported

 hda: hda1 hda2 hda3

Adding 1004052k swap on /dev/hda2.  Priority:-1 extents:1 across:1004052k

EXT3 FS on hda3, internal journal

```

I changed the unmaskirq setting with hdparm, but that didn't change anything.

----------

## Flintz

I've tracked down the error a little more: Its not the drive in general, but some files. To be more precisely the only files that seem to encounter this weird speed issue are files downloaded by MLDONKEY  :Confused: 

I have absolutely no clue why, but I did two things:

1. Made me a file only with zeros: "dd if=/dev/zero of=test.zero count=1024000" and did some "benchmarking" with it, absolutely no problems, speed is ok

2. I copied one the the files with speed issues over to the same harddrive (with the well known low speed) and then did again some benchmarking with the NEW (duplicate) file, and see no speed issues there!

That leads to 2 possible conclusion in my eyes:

1. The drive is damaged in some sectors where accidentely the downloadfiles from mldonkey are stored and when copying them to another intact position the problem goes away.

2. Somehow mldonkey messes up the downloaded files, probably some thing of weird fragmentation? Or I have an filesystem-error here.

Whats your thoughts on that?

----------

## NeddySeagoon

Flintz,

I can see from your /proc/interrups that irqunmask is not your problem. Its purpose was to allow lower priority IRQs to interrupt disk transfer interrups,   which take a long time to process. Specificaly, it was to allow serial port IRQs to be serviced so that data was not lost.

Today, with some arrangements of interrups, it causes hard drive IRQs to time out and PIO mode to be used.

It sounds like your mldonky files are in fragments scattered all over the disk, so head movement time reduces the data rate.

The drive will remap bad sectors so the OS never sees them until it runs out of spares. To see if this is an issue, emerge smartmontools and look at the drives error log,

----------

## Flintz

Seems like its a mldonkey related problem. Look here: http://mldonkey.sourceforge.net/forums/viewtopic.php?t=2836&start=0

I read in the posts that mldonkey had an anti-fragmentation-code but it was removed. Someone suggested to set these settings:

```
set buffer_writes true 

set buffer_writes_delay 480 

set buffer_writes_threshold 16384
```

I'll give it a try and see if it works

----------

## HisN

Flintz

Would you like to explain how you get the driver compiled? (am besten wenn Du nen AMD64 und nen Gentoo64 benutzen würdest *g*)

----------

## Flintz

Yes, I used an AMD64 and Gentoo64, the makefile does the check for 64bit architecture somehow wrong. I solved it by openen the Makefile with a text-editor scrolled down to the "############# architecher configuration #############" section removed all if commands and set ARCH_TYPE=x86_64.

This is how it looks now here:

```
############# architecher configuration #############

CONFIG_AMD64=$(shell cat $(KERNEL_BLD_DIR)/.config | grep CONFIG_X86_64 | awk -F= '{print $$2}')

        ARCH_TYPE=x86_64

ifeq ($(ARCH_TYPE),x86_64)

        ARCH_FLAGS=-D_AMD64B -D_64BPLATFORM

else

        ARCH_FLAGS=-D_X8632B -D_32BPLATFORM

endif

############# end global configuation ################

```

Worked for me, not the nice way but works  :Wink: 

----------

## HisN

Thx for your help,

seems like not solving the problem to me. I get the same error (maybe the Architecture-Check performs well on my system) but the compiler need an include ioctl32.h in /usr/src/linux/include/asm and this file is missing on my system.

I tried using linux/ioctr32.h (this file is present) but the compiler will not work with this. He produces a nonfunctional modules shasta.ko *sigh*

Maybe you have clue or can post the content of ioctl32.h in your asm directory  :Smile: 

----------

## Flintz

I have a ioctl32.h present, look here:

```

shasta # locate ioctl32.h

/usr/src/linux-2.6.15-gentoo-r5/fs/xfs/linux-2.6/xfs_ioctl32.h

/usr/src/linux-2.6.15-gentoo-r5/include/linux/ioctl32.h

/usr/src/linux-2.6.15-gentoo-r7/fs/xfs/linux-2.6/xfs_ioctl32.h

/usr/src/linux-2.6.15-gentoo-r7/include/linux/ioctl32.h

/usr/src/linux-2.6.16-gentoo-r7/fs/xfs/linux-2.6/xfs_ioctl32.h

/usr/src/linux-2.6.16-gentoo-r7/include/linux/ioctl32.h

/usr/src/linux-2.6.16-gentoo-r9/fs/xfs/linux-2.6/xfs_ioctl32.h

/usr/src/linux-2.6.16-gentoo-r9/include/linux/ioctl32.h

/usr/include/linux/ioctl32.h

/usr/include/asm-x86_64/ioctl32.h

/usr/include/asm/ioctl32.h

```

Probably you should try to point the shasta.c and shasta24.c to the right location, in my shasta.c:

```
#ifdef _64BPLATFORM

#include <linux/ioctl32.h>

#endif

```

and shasta24.c:

```
#ifdef __x86_64__

#include <linux/ioctl32.h>

#endif

```

----------

## HisN

Maybe I use the wrong source-version. There are no shasta.c and shasta24.c in the driver-directory. I downloaded the open-source driver.

----------

## spiralvoice

 *Flintz wrote:*   

> I read in the posts that mldonkey had an anti-fragmentation-code but it was removed.

 

Current MLDonkey CVS, newer than 2.7.7, has again anti-fragmentation-code.

I can acknowledge that MLDonkey downloaded files can be heavily fragmented,

especially if downloaded from BT due to smaller block sizes in that network.

 *Quote:*   

> 2006/07/15
> 
> 5203: Swarmer: Anti-fragmentation (pango, antifrag_v7)
> 
> * Each file is divided into blocks saved in new option
> ...

 

----------

## Flintz

you have the 4_shasta_linux_src-2.9.0.10.zip? If I decompress it I get 4 files, Readme, Makefile and shasta.c and shasta24.c, what files do you have?

----------

## schrippe

```
set buffer_writes true 

set buffer_writes_delay 480 

set buffer_writes_threshold 16384
```

Sorry, but I can't say it working for me. The test written above are true for me. 

Is anybody here that solved this problem?

THX

----------

## spiralvoice

buffer_writes were only flushed to disk using EDK module. If you use

MLDonkey with deactivated EDK module buffer_writes do not work.

This is fixed in current CVS

 *Quote:*   

> 2006/09/01
> 
> 5355: Move buffer_writes_delay timer from Donkey to Global module

 

Current CVS also has anti-fragmentation-code mentioned above.

----------

## schrippe

Can you say me, how to build my own ebuild from the cvs source? It this possible?

THX

----------

## spiralvoice

 *schrippe wrote:*   

> Can you say me, how to build my own ebuild from the cvs source?

 

```
cvs -d:pserver:anonymous@cvs.sv.gnu.org:/sources/mldonkey co -P mldonkey

cd mldonkey

./configure

make
```

Overwrite the old MLDonkey binary (in /usr/bin/mlnet ?) with ./mlnet

----------

## schrippe

yes, but is there no gentoo-ebuild especially for emerge?

I know how to build my own mldonk per cvs source.

I thought that would be much easier as cvs.

----------

## PaulBredbury

 *schrippe wrote:*   

> how to build my own ebuild from the cvs source? It this possible?

 

Here's 300 examples:

```
find /usr/portage/ -name \*.ebuild | xargs grep ECVS
```

They reference /usr/portage/eclass/cvs.eclass

----------

