# oddest performance problem.... 2005.1-r1 [SOLVED]

## heinzg

Hi there!

I have the oddest performance problem.... 

It started out that Samba performance was real poor & and still is( from windos all traffic to and from is slow, but from other system traffic speeds are normal)... But I started to look a bit deeper and found the system itself is not a responsive as it should be!

I have 2 simmilar file servers in the cellar. Both are Gentoo: but the one shages the bollox off of the other, and the slow one has far better hardware specs. 

Why could this be ??

Anybody have any ideas on this? 

I am at my end, I even pluged the 3ware card in other PCI slots to see if that would change this, it did not. Reinstalled nothing... it makes me sad, it does   :Crying or Very sad: 

Just checked the BIOS settings everything that is not needed is off.

E.g  I made a file 2.2GB big and did the following: 

Just as a simples check on both systems.... and one is almost 2 time the speed of the other!!

```
# time dd if=./bigfile of=./bigfile2 bs=32k
```

System 1:

 *Quote:*   

> 
> 
> 70384+0 records in
> 
> 70384+0 records out
> ...

 

System2:

 *Quote:*   

> 
> 
> 70384+0 records in
> 
> 70384+0 records out
> ...

 

Also if one is copying lost of files to both systems creating lost of IO and Load "System 1" becomes unresponsive or very slow, where System 2 is fast as normal.

E.g 

During the "dd" test as above, if I do a "CTRL + C" after 30 sec or so, "System 1" needs about 5 to 8 sec to stop, where the "System 2" is less than 1 sec. 

Samba crawls on "System 1" but flyes in "System 2" and i cant find anything in the logs 

but did notice that in TOP the "CPU wa" value is always very high.

```

top - 23:38:39 up  5:26,  2 users,  load average: 1.19, 0.29, 0.09

Tasks:  35 total,   2 running,  33 sleeping,   0 stopped,   0 zombie

Cpu(s):  0.0% us,  3.0% sy,  0.0% ni,  0.0% id, 97.0% wa,  0.0% hi,  0.0% si

Mem:    905592k total,   896348k used,     9244k free,    25024k buffers

Swap:  1092412k total,      192k used,  1092220k free,   839548k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                 

8536 root      18   0  1464  420  328 D  3.0  0.0   0:01.39 dd if ./bigfile of ./bigfile2 bs 32k                                                    

     1 root      16   0  1464  516  452 S  0.0  0.1   0:00.14 init [3]                                                                                

     2 root      35  19     0    0    0 S  0.0  0.0   0:00.00 [ksoftirqd/0]                                                                           

```

Both have a 1 raid5 disk configured 64k stripe and have the 3ware trick "blockdev --setra 16384 /dev/sda" on.

----------------------------------------------------------------------------------------------------

System 1 ( The Should Be Fast One - But Is Not):

HARDWARE:

AMD Athlon(tm) 64 Processor 3000+

Gigabyte K8VT800 Pro

1GB RAM

3ware 9500s RAID5 

4x 200GB Samsung 200GB 7200rpm 8MB

SOFTWARE:

# cat /etc/gentoo-release 

Gentoo Base System version 1.6.14

# uname -a

Linux gentoo32 2.6.15-gentoo-r1Y #3 Wed Feb 22 10:15:01 CET 2006 i686 AMD Athlon(tm) 64 Processor 3000+ AuthenticAMD GNU/Linux

Config file can be seen here --->>>> http://www.wosindwirhier.de/2.6.15-gentoo-r1Y.txt

dmes file for this box is here -->>>> http://www.wosindwirhier.de/dmes.txt

#ls -la /etc/make.profile 

lrwxrwxrwx  1 root root 46 Feb 22 00:18 /etc/make.profile -> /usr/portage/profiles/default-linux/x86/2005.1

#cat /etc/make.conf

CFLAGS="-O2 -march=athlon-xp -pipe -fomit-frame-pointer"

CXXFLAGS="${CFLAGS}"

CHOST="i686-pc-linux-gnu"

ARCH="x86"

ACCEPT_KEYWORDS="x86"

MAKEOPTS="-j2"

PORT_LOGDIR=/var/log/portage

PORTDIR=/usr/portage

DISTDIR=${PORTDIR}/distfiles

PKGDIR=${PORTDIR}/packages

PORTAGE_TMPDIR=/tmp/.portage

CCACHE_SIZE="2G"

AUTOCLEAN="yes"

SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"

GENTOO_MIRRORS="ftp://ftp.uni-erlangen.de/pub/mirrors/gentoo/ http://ftp.uni-erlangen.de/pub/mirrors/gentoo/ ftp://ftp.belnet.be/mirror/rsync.gentoo.

org/gentoo/"

FEATURES="buildpkg fixpackages sandbox ccache"

#My use settings

USE="nptl nptlonly sse mmx 3dnow ccache nvidia -cups -ipv6"

# cat /etc/fstab    

#Normal FileSystems 

/dev/sda3                       /               reiserfs        noatime         0 1

/dev/sda1                       /boot           ext2            noauto,noatime  0 2

/dev/sda4                       /rep            reiserfs        noatime         0 3

----------------------------------------------------------------------------------------------------

System 2 ( Is Fast as Gentoo Should Be! ): 

HARDWARE:

AMD Athlon(TM) XP 2000+ 

Asus A7V333

1GB RAM

3ware 7506 Series

4x Maxtor 160GB 7200rpm 2MB

SOFTWARE:

# cat /etc/gentoo-release 

Gentoo Base System version 1.6.13

# uname -a

Linux tux32 2.6.13-gentoo-r3Y #3 Tue Feb 14 16:01:06 CET 2006 i686 AMD Athlon(TM) XP 2000+ AuthenticAMD GNU/Linux

Config file can be seen here --->>>> http://www.wosindwirhier.de/2.6.13-gentoo-r3Y.txt

#ls -la /etc/make.profile

lrwxrwxrwx  1 root root 48 Feb  5 22:13 /etc/make.profile -> ../usr/portage/profiles/default-linux/x86/2005.1

# cat /etc/make.conf 

CFLAGS="-O2 -march=athlon-xp -pipe"

CXXFLAGS="${CFLAGS}"

CHOST="i686-pc-linux-gnu"

ARCH="x86"

ACCEPT_KEYWORDS="x86"

MAKEOPTS="-j2"

PORT_LOGDIR=/var/log/portage

PORTDIR=/usr/portage

DISTDIR=${PORTDIR}/distfiles

PKGDIR=${PORTDIR}/packages

PORTAGE_TMPDIR=/tmp/.TMP

CCACHE_SIZE="2G"

AUTOCLEAN="yes"

SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"

GENTOO_MIRRORS="ftp://ftp.uni-erlangen.de/pub/mirrors/gentoo/ http://ftp.uni-erlangen.de/pub/mirrors/gentoo/ ftp://ftp.belnet.be/mirror/rsync.gentoo.org/gentoo/"

FEATURES="sandbox ccache buildpkg fixpackages sfperms"

#My use settings

USE="-gnome posix -kde sse mmx 3dnow java php gd gd-external nptl nptlonly apache2 ncurses samba ssl usb zlib crypt bzlib libclamav ldap crypt postgres mysql" 

#  cat /etc/fstab 

#Normal FileSystems 

/dev/sda2                       /               reiserfs        noatime         0 2

#Special FileSystems

/dev/sda1                       none            swap            sw              0 0

Last edited by heinzg on Fri Feb 24, 2006 4:33 pm; edited 1 time in total

----------

## heinzg

Hi 

Has anybody got a tip for me were to look... It can't be that the box with superior hardware in all respects is half the speed of the other system.

Cheers

Heinzg

----------

## bollucks

Do you take into account how much time is spent with your dd data in dirty ram being written out by pdflush after dd has returned? Make sure you do

time dd if=.... && sync

to make sure the sync writeout time is included in your benchmark.

However if you have high iowait times and poor performance under disk load that sounds like a badly performing device driver. You are on SATA and perhaps your driver config for that isn't ideal or is a dud driver. Check the dmesg output for info about the drivers.

----------

## heinzg

Hi 

Thanx for the post.

I am using, in both cases using the built in driver that comes with the 2.6. kernel. 

Last week I install suse SLES 9 with the latest 3ware driver from their web site. and the performance was even worse.   

Here is the driver info from dmesg: full dmesg at http://www.wosindwirhier.de/dmes.txt

 *Quote:*   

> 3ware 9000 Storage Controller device driver for Linux v2.26.02.004.
> 
> ACPI: PCI Interrupt 0000:00:0d.0[A] -> GSI 17 (level, low) -> IRQ 16
> 
> scsi0 : 3ware 9000 Storage Controller
> ...

 

I re-ran the bench with the sync option

```
sync && time dd if=./bigfile of=./bigfile2 bs=32k && sync
```

System 1: 

 *Quote:*   

> 
> 
> 70384+0 records in
> 
> 70384+0 records out
> ...

 

System 2: 

 *Quote:*   

> 
> 
> 70384+0 records in
> 
> 70384+0 records out
> ...

 

I will install Bonnie or somthing for a better bench test 

Cheers Heinzg

----------

## heinzg

From the test I have been doing today I see that the read performance seems Ok 

But the WRITE performance is as the box, in the celler. Realy bad (SUX). 

Why does the system suffer such bad responsiveness ? Because when I was doing the write lmdd test I did a CTRL + C and it took over 30 sec to abort!!

lmdd Write Test

System 1

```

# for a in 1 2 3 4 5; do lmdd of=bob$a ; done

write: wanted=8192 got=8191

2147.4755 MB in 233.4636 secs, 9.1983 MB/sec

write: wanted=8192 got=8191

2147.4755 MB in 279.9048 secs, 7.6722 MB/sec

write: wanted=8192 got=8191

2147.4755 MB in 280.9912 secs, 7.6425 MB/sec

write: wanted=8192 got=8191

2147.4755 MB in 275.8492 secs, 7.7850 MB/sec

write: wanted=8192 got=8191

2147.4755 MB in 283.1327 secs, 7.5847 MB/sec

```

System 2

```

# for a in 1 2 3 4 5; do lmdd of=bob$a ; done

write: wanted=8192 got=8191

2147.4755 MB in 123.4821 secs, 17.3910 MB/sec

write: wanted=8192 got=8191

2147.4755 MB in 166.8202 secs, 12.8730 MB/sec

write: wanted=8192 got=8191

2147.4755 MB in 195.1745 secs, 11.0028 MB/sec

write: wanted=8192 got=8191

2147.4755 MB in 145.3806 secs, 14.7714 MB/sec

write: wanted=8192 got=8191

2147.4755 MB in 168.8912 secs, 12.7151 MB/sec

```

lmdd Read

System 1

```

for a in 1 2 3 4 5 ; do lmdd if=bob$a ; done

2147.4836 MB in 18.9463 secs, 113.3461 MB/sec

2147.4836 MB in 18.9795 secs, 113.1473 MB/sec

2147.4836 MB in 19.1919 secs, 111.8952 MB/sec

2147.4836 MB in 19.0852 secs, 112.5209 MB/sec

2147.4836 MB in 19.0269 secs, 112.8657 MB/sec

```

System2

```

# for a in 1 2 3 4 5 ; do lmdd if=bob$a ; done

2147.4836 MB in 34.1135 secs, 62.9511 MB/sec

2147.4836 MB in 36.1482 secs, 59.4077 MB/sec

2147.4836 MB in 34.8607 secs, 61.6019 MB/sec

2147.4836 MB in 32.8862 secs, 65.3004 MB/sec

2147.4836 MB in 32.8077 secs, 65.4567 MB/sec

```

Tiotest

System 1

```

tiotest -d ./del/ -f 50

Tiotest results for 4 concurrent io threads:

,----------------------------------------------------------------------.

| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |

+-----------------------+----------+--------------+----------+---------+

| Write         200 MBs |   28.2 s |   7.098 MB/s |   0.0 %  |  10.6 % |

| Random Write   16 MBs |   30.8 s |   0.508 MB/s |   0.0 %  |   0.2 % |

| Read          200 MBs |    0.2 s | 1257.395 MB/s |   5.0 %  | 254.0 % |

| Random Read    16 MBs |    0.0 s | 1102.060 MB/s |   0.0 %  |  84.6 % |

`----------------------------------------------------------------------'

Tiotest latency results:

,-------------------------------------------------------------------------.

| Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |

+--------------+-----------------+-----------------+----------+-----------+

| Write        |        0.054 ms |      312.076 ms |  0.00000 |   0.00000 |

| Random Write |        0.006 ms |        0.045 ms |  0.00000 |   0.00000 |

| Read         |        0.008 ms |       96.023 ms |  0.00000 |   0.00000 |

| Random Read  |        0.003 ms |        0.019 ms |  0.00000 |   0.00000 |

|--------------+-----------------+-----------------+----------+-----------|

| Total        |        0.029 ms |      312.076 ms |  0.00000 |   0.00000 |

`--------------+-----------------+-----------------+----------+-----------'

```

System 2

```

tiotest -d ./del/ -f 50

Tiotest results for 4 concurrent io threads:

,----------------------------------------------------------------------.

| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |

+-----------------------+----------+--------------+----------+---------+

| Write         200 MBs |   20.0 s |   9.998 MB/s |   0.5 %  |  31.6 % |

| Random Write   16 MBs |   28.0 s |   0.559 MB/s |   0.0 %  |   0.7 % |

| Read          200 MBs |    0.4 s | 511.822 MB/s |   9.2 %  | 206.8 % |

| Random Read    16 MBs |    0.0 s | 470.052 MB/s |   0.0 %  |  96.3 % |

`----------------------------------------------------------------------'

Tiotest latency results:

,-------------------------------------------------------------------------.

| Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |

+--------------+-----------------+-----------------+----------+-----------+

| Write        |        0.108 ms |      368.090 ms |  0.00000 |   0.00000 |

| Random Write |        0.011 ms |        0.271 ms |  0.00000 |   0.00000 |

| Read         |        0.016 ms |       69.615 ms |  0.00000 |   0.00000 |

| Random Read  |        0.008 ms |        0.026 ms |  0.00000 |   0.00000 |

|--------------+-----------------+-----------------+----------+-----------|

| Total        |        0.058 ms |      368.090 ms |  0.00000 |   0.00000 |

`--------------+-----------------+-----------------+----------+-----------'

```

Bonnie 1024MB

System 1

```

sync && bonnie -d ./del/ -s 1024  -m System1

              -------Sequential Output-------- ---Sequential Input-- --Random--

              -Per   Char-   --Block---   -Rewrite--   -Per Char-   --Block---   --Seeks---

Machine    MB   K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU

System1  1024 11975   23.9 10378    2.5  6442  1.5     21372 38.6   54748  7.5    1000.8  1.5

```

System 2

```

sync && bonnie -d ./del/ -s 1024  -m System2

              -------Sequential Output-------- ---Sequential Input-- --Random--

              -Per   Char-   --Block---   -Rewrite--   -Per Char-   --Block---   --Seeks---

Machine    MB   K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU

System2  1024 14435   39.6 27656 12.7   7671  3.0     25220  65.4  74271 16.0   1667.7  6.8

```

Bonnie++

System 1

```

smb # bonnie++ -d /tmp/ -m System2 -u heinzg

Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-

Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

System1          2G   281  98  7404   2  6776   1  1482  95 105688  15 311.3   3

Latency             42484us   49458ms    5584ms   44876us     276ms     556ms

Version 1.93c       ------Sequential Create------ --------Random Create--------

System1             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

                 16  9539  36 +++++ +++ 18009  76 11958  42 +++++ +++ 18332  80

Latency             16692us    1469us    1636us   10415us      15us    1673us

1.93c,1.93c,System1,1,1140691445,2G,,281,98,7404,2,6776,1,1482,95,105688,15,311.3,3,16,,,,,9539,36,+++++,+++,18009,76,11958,42,+++++,+++,18332,80,42484us,49458ms,5584ms,44876us,276ms,556ms,16692us,1469us,1636us,10415us,15us,1673us

```

System 2

```

smb # bonnie++ -d /tmp/ -m System2 -u heinzg

Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-

Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

System2          2G   174  99 12482   7  8868   3  1412  96 79073  20 327.5   8

Latency             67507us   34927ms    2481ms   35811us     248ms     251ms

Version 1.93c       ------Sequential Create------ --------Random Create--------

System2             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

                 16  9168  54 +++++ +++ 11983  80  9460  54 +++++ +++ 12458  90

Latency             42826us    2075us   19150us   46648us      27us     316us

1.93c,1.93c,System2,1,1140703800,2G,,174,99,12482,7,8868,3,1412,96,79073,20,327.5,8,16,,,,,9168,54,+++++,+++,11983,80,9460,54,+++++,+++,12458,90,67507us,34927ms,2481ms,35811us,248ms,251ms,42826us,2075us,19150us,46648us,27us,316us

```

Cheers

heinzgLast edited by heinzg on Fri Feb 24, 2006 4:13 pm; edited 1 time in total

----------

## heinzg

Hi 

I don't know if this is the way to do things... I downloaded the latest 3ware driver unpacked it.

Then I copied the two source files from 3ware - "3w-9xxx.c  3w-9xxx.h" into the kernel source tree, replacing the existing files.

Re compiled the kernel. update Grub. reboot 

The Box came up with a different driver version... So I take it that it worked.

Did some of the test as above, and it seems that the read times are a little better and the write times a little worse.

```

# sync ; sync && tiotest -d ./del/ -f 50

Tiotest results for 4 concurrent io threads:

,----------------------------------------------------------------------.

| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |

+-----------------------+----------+--------------+----------+---------+

| Write         200 MBs |   28.3 s |   7.066 MB/s |   0.3 %  |   9.3 % |

| Random Write   16 MBs |   31.6 s |   0.494 MB/s |   0.0 %  |   0.2 % |

| Read          200 MBs |    0.2 s | 1292.708 MB/s |  18.1 %  | 157.7 % |

| Random Read    16 MBs |    0.0 s | 1132.246 MB/s |  29.0 %  |  58.0 % |

`----------------------------------------------------------------------'

Tiotest latency results:

,-------------------------------------------------------------------------.

| Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |

+--------------+-----------------+-----------------+----------+-----------+

| Write        |        0.043 ms |      304.055 ms |  0.00000 |   0.00000 |

| Random Write |        0.006 ms |        0.049 ms |  0.00000 |   0.00000 |

| Read         |        0.005 ms |       64.018 ms |  0.00000 |   0.00000 |

| Random Read  |        0.003 ms |        0.018 ms |  0.00000 |   0.00000 |

|--------------+-----------------+-----------------+----------+-----------|

| Total        |        0.023 ms |      304.055 ms |  0.00000 |   0.00000 |

`--------------+-----------------+-----------------+----------+-----------'

# 

# for a in 1 2 3; do lmdd of=bob$a ; done

write: wanted=8192 got=8191

2147.4755 MB in 244.0551 secs, 8.7991 MB/sec

write: wanted=8192 got=8191

2147.4755 MB in 271.7055 secs, 7.9037 MB/sec

write: wanted=8192 got=8191

2147.4755 MB in 293.9404 secs, 7.3058 MB/sec

```

So this problem goes on....

Heinzg

----------

## heinzg

I think this one is solved... The Write Cache on the card was disabled   :Embarassed: 

Now it goes as it should....

```
 

# tiotest -d ./del/ -f 50

Tiotest results for 4 concurrent io threads:

,----------------------------------------------------------------------.

| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |

+-----------------------+----------+--------------+----------+---------+

| Write         200 MBs |    4.5 s |  44.307 MB/s |   0.7 %  |  65.0 % |

| Random Write   16 MBs |    9.2 s |   1.703 MB/s |   0.0 %  |   1.0 % |

| Read          200 MBs |    0.2 s | 1233.289 MB/s |   4.9 %  |  93.7 % |

| Random Read    16 MBs |    0.0 s | 1093.192 MB/s |  28.0 %  |  56.0 % |

`----------------------------------------------------------------------'

Tiotest latency results:

,-------------------------------------------------------------------------.

| Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |

+--------------+-----------------+-----------------+----------+-----------+

| Write        |        0.048 ms |      508.030 ms |  0.00000 |   0.00000 |

| Random Write |        0.006 ms |        0.036 ms |  0.00000 |   0.00000 |

| Read         |        0.003 ms |        0.022 ms |  0.00000 |   0.00000 |

| Random Read  |        0.003 ms |        0.019 ms |  0.00000 |   0.00000 |

|--------------+-----------------+-----------------+----------+-----------|

| Total        |        0.024 ms |      508.030 ms |  0.00000 |   0.00000 |

`--------------+-----------------+-----------------+----------+-----------'

# sync && sync && for a in 1 2 3 4 5; do lmdd of=bob$a ; done

write: wanted=8192 got=8191

2147.4755 MB in 27.2877 secs, 78.6975 MB/sec

write: wanted=8192 got=8191

2147.4755 MB in 35.1331 secs, 61.1241 MB/sec

write: wanted=8192 got=8191

2147.4755 MB in 36.1987 secs, 59.3246 MB/sec

write: wanted=8192 got=8191

2147.4755 MB in 33.8631 secs, 63.4163 MB/sec

write: wanted=8192 got=8191

2147.4755 MB in 35.3913 secs, 60.6780 MB/sec

# sync && sync && for a in 1 2 3 4 5; do lmdd if=bob$a ; done

2147.4836 MB in 18.8087 secs, 114.1748 MB/sec

2147.4836 MB in 19.0856 secs, 112.5183 MB/sec

2147.4836 MB in 18.8823 secs, 113.7300 MB/sec

2147.4836 MB in 18.8632 secs, 113.8453 MB/sec

2147.4836 MB in 18.4551 secs, 116.3628 MB/sec

```

Cheers

Heinzg

----------

