# Performance issues

## SweepingOar

I've recently re-installed Gentoo on my system because I was having trouble outlined in this thread:

https://forums.gentoo.org/viewtopic-p-5009630.html#5009630

This time I used 2008.1 beta amd64 on a raid1 using two sata drives. My mb is an Intel G965RY (exact model number: BOXDG965RYCK socket 775) with a Intel Core 2 duo E6400 cpu and 1gb of ram. Things seemed to be ok for a while but the server is not in "production" (i.e. it's online but not taking traffic). I installed exim and sent some of our email through it and then I noticed the same sort of performance wallop that occurred before. Running top shows the cpu only at 50% idle when there is ANY apache request to the php/mysql site that the machine serves. Our idle never used to drop below 90%. Our traffic hasn't increased, our hardware and site has stayed the same. The only thing that's changed is a newer kernel, mysql and other system sw.

```
$ free -m

             total       used       free     shared    buffers     cached

Mem:           976        886         90          0        298        192

-/+ buffers/cache:        394        582

Swap:         1968         18       1950
```

Here's an example of what happens when the machine is idle and receives one (1) web request (often idle drops well below 80%, but here it's just 86%):

```
top - 10:05:09 up 12 days,  4:38,  1 user,  load average: 0.00, 0.00, 0.00

Tasks:  79 total,   1 running,  78 sleeping,   0 stopped,   0 zombie

Cpu(s):  5.2%us,  0.5%sy,  0.0%ni, 86.2%id,  8.2%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:   1000360k total,   922020k used,    78340k free,   306120k buffers

Swap:  2016056k total,    18652k used,  1997404k free,   198580k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

 5778 mysql     20   0  174m  63m 5192 S    6  6.5   0:09.32 mysqld

 1982 apache    20   0  189m  10m 2996 S    4  1.0   0:00.42 apache2

    1 root      20   0  3720  532  476 S    0  0.1   0:06.70 init

    2 root      15  -5     0    0    0 S    0  0.0   0:00.00 kthreadd

    3 root      RT  -5     0    0    0 S    0  0.0   0:00.14 migration/0

    4 root      15  -5     0    0    0 S    0  0.0   0:00.76 ksoftirqd/0

    5 root      RT  -5     0    0    0 S    0  0.0   0:00.30 migration/1

    6 root      15  -5     0    0    0 S    0  0.0   0:01.10 ksoftirqd/1

    7 root      15  -5     0    0    0 S    0  0.0   0:10.10 events/0

    8 root      15  -5     0    0    0 S    0  0.0   0:07.46 events/1

    9 root      15  -5     0    0    0 S    0  0.0   0:00.00 khelper

   78 root      15  -5     0    0    0 S    0  0.0   0:04.64 kblockd/0

   79 root      15  -5     0    0    0 S    0  0.0   0:00.42 kblockd/1

   82 root      15  -5     0    0    0 S    0  0.0   0:00.00 kacpid

   83 root      15  -5     0    0    0 S    0  0.0   0:00.00 kacpi_notify

  162 root      15  -5     0    0    0 S    0  0.0   0:00.00 ata/0

  163 root      15  -5     0    0    0 S    0  0.0   0:00.00 ata/1

  164 root      15  -5     0    0    0 S    0  0.0   0:00.00 ata_aux

  165 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksuspend_usbd

  171 root      15  -5     0    0    0 S    0  0.0   0:00.00 khubd

  174 root      15  -5     0    0    0 S    0  0.0   0:00.00 kseriod

  222 root      20   0     0    0    0 S    0  0.0   0:00.06 pdflush

  223 root      20   0     0    0    0 S    0  0.0   0:04.78 pdflush

  224 root      15  -5     0    0    0 S    0  0.0   0:12.36 kswapd0

  225 root      15  -5     0    0    0 S    0  0.0   0:00.00 aio/0
```

I was thinking of trying a new motherboard, but that would just be a stab in the dark. Any advice?

----------

## BradN

8.2%wa seems a bit high for something like this.  This indicates that no actual processing is going on during this time, but processes are waiting for disk or other I/O.

Is DMA enabled on the hard drive(s)?  Can you test with another network card?

----------

## SweepingOar

Isn't 5.x% us high as well for one web request? Did I mention there's a software raid1? Thanks. Yes, I've got two nics in the machine. I'll try with eth1 and post results. I think dma is on, but this is the output so I'm not positive:

```
# hdparm -d /dev/sda

/dev/sda:

 HDIO_GET_DMA failed: Inappropriate ioctl for device
```

----------

## BradN

Yeah, I get that for my SATA controller as well.  If you get more than a few MB/s running "hdparm -t" on it, then it's got DMA on.

Depending how much processing is really being done with the request, 5% isn't too unreasonable.  I noticed it's triggering mysql queries, so there must be at least some data crunching involved.  If you've got some kind of speedstep or frequency scaling in effect, this might be worth investigating too.

----------

## SweepingOar

I'll have to check the eth1 vs eth0 issue tomorrow because I'll need to change the plug the ethernet is in on the machine obviously. I think dma is on from these numbers:

```
# hdparm -tT /dev/sda

/dev/sda:

 Timing cached reads:   2706 MB in  2.00 seconds = 1353.32 MB/sec

 Timing buffered disk reads:  188 MB in  3.02 seconds =  62.30 MB/sec

# hdparm -tT /dev/sdb

/dev/sdb:

 Timing cached reads:   2688 MB in  2.00 seconds = 1343.85 MB/sec

 Timing buffered disk reads:  178 MB in  3.03 seconds =  58.77 MB/sec
```

I thought that I disabled the frequency scaling, but I just ran make menuconfig and it looks like it's on. I'll also recompile the kernel tomorrow when I change the nic plug:

```
[*] CPU Frequency scaling

[*]   Enable CPUfreq debugging

<*>   CPU frequency translation statistics

[ ]     CPU frequency translation statistics details

      Default CPUFreq governor (performance)  --->

-*-   'performance' governor

< >   'powersave' governor

<*>   'userspace' governor for userspace frequency scaling

<*>   'ondemand' cpufreq policy governor

< >   'conservative' cpufreq governor

      *** CPUFreq processor drivers ***

<*>   ACPI Processor P-States driver

< >   AMD Opteron/Athlon64 PowerNow!

< >   Intel Enhanced SpeedStep (deprecated)

< >   Intel Pentium 4 clock modulation

      *** shared options ***

[*]   /proc/acpi/processor/../performance interface (deprecated)
```

----------

## SweepingOar

I changed the network connection to eth1 and although it does seem to give slightly lower wait numbers in top, it's still in the same ballpark (5.2 wa for a single apache/php/mysqld request). Do you think frequency scaling could be affecting this?

----------

## DaggyStyle

you lucky, mine outputs

```

/dev/sda:

 Timing cached reads:   1200 MB in  2.00 seconds = 599.43 MB/sec

 Timing buffered disk reads:   44 MB in  3.01 seconds =  14.63 MB/sec

```

----------

## SweepingOar

Is your drive 7200 rpm? Maybe your DMA isn't enabled either for the drive or in the kernel. Still looking for advice re: processor nose-dives as mentioned above by the way. Thanks.

----------

## DaggyStyle

 *SweepingOar wrote:*   

> Is your drive 7200 rpm? Maybe your DMA isn't enabled either for the drive or in the kernel. Still looking for advice re: processor nose-dives as mentioned above by the way. Thanks.

 

when using the kernel sata driver, dma is turned on by default

----------

## SweepingOar

So, do you know what the story is with the drive? That seems slow compared to the drives in the machines here. Any suggestions on the performance trouble mentioned above re: mysql/php/apache? Thanks.

----------

## Monkeh

 *DaggyStyle wrote:*   

> you lucky, mine outputs
> 
> ```
> 
> /dev/sda:
> ...

 

Try turning NCQ off. Some drives/controllers are broken.

----------

## DaggyStyle

cant seems to find it

----------

