# How to troubleshoot high load average?

## Moreaulf

Hi!

I'm running a Gentoo server which uses apache, mysql, php and qmail with spamassassin. The web server handles about 40 000 sessions (which most are from search engines! - my guess is that 10 000 of them are actual users) a day.

I've been Googling the net for soo many hours trying to find solutions to the problem of high load average but although I've seen others having the same problem I've never been able to benefit from the suggested actions.

The problem is that every once in a while the server seem to hog up on something which causes it to rocket rowards numbers up to 40 in load average. When passing 5 the page gets really slow and above 10 there will be more than a 30 seconds wait to load one page - if it doesn't time out or loses the MySQL connection during this time.

I've been monitoring the system during these peeks and I can't seem to find something that really causes this. Below is an extract from a log (CPU:s are 2 * Intel 2.4GHz):

```
Wed Jul 26 21:30:06 CEST 2006

             total       used       free     shared    buffers     cached

Mem:       1035728    1029348       6380          0       3564     131732

-/+ buffers/cache:     894052     141676

Swap:       999928     547292     452636

top - 21:30:07 up 2 days,  6:17,  1 user,  load average: 23.14, 21.58, 18.12

Tasks: 294 total,   1 running, 293 sleeping,   0 stopped,   0 zombie

Cpu(s):  5.4% us,  4.1% sy,  0.0% ni, 73.0% id, 17.5% wa,  0.1% hi,  0.0% si

Mem:   1035728k total,  1028820k used,     6908k free,     3604k buffers

Swap:   999928k total,   546060k used,   453868k free,   131852k cached
```

A disk test does give various result, anywhere from 15 to 60 MB/sec, like this:

 *Quote:*   

> hdparm -t /dev/hda
> 
> /dev/hda:
> 
>  Timing buffered disk reads:  122 MB in  3.03 seconds =  40.23 MB/sec
> ...

 

However, it doesn't seem to be connectedto the load average since I can get as low as 15 MB/sec when load average is below 0.5 and over 40 MB/sec when load average is high.

Feel very free to come up with suggestions and I'll be glad to publish results for further troubleshooting.

Thank you very much!

/Thomas

----------

## !equilibrium

i have an ISP/Data center with a lot of very heavy load webservers, and to reduce load on apache i suggest you:

[1] disable FollowSymLinks and AllowOverride globally (this reduce apache's memory usage drastically and let you gain a lot of performances).

into http.conf there is 

```
<Directory />

    Options FollowSymLinks

    AllowOverride None

</Directory>
```

change to:

```
<Directory />

    Options -FollowSymLinks

    AllowOverride None

</Directory>
```

modify your virtual hosts (/etc/apache/vhosts/*) to disable FollowSymLinks and AllowOverride, or enable them only if really required. If you don't know what is the option FollowSymLinks, you do not need it, so disable globaly.

[2] increase apache buffers:

```
SendBufferSize 32768
```

[3] compress the output static pages:

```
<Location /var/www/>

        # Insert filter

        AddOutputFilterByType DEFLATE text/plain

        AddOutputFilterByType DEFLATE text/xml

        AddOutputFilterByType DEFLATE application/xhtml+xml

        AddOutputFilterByType DEFLATE text/css

        AddOutputFilterByType DEFLATE application/xml

        AddOutputFilterByType DEFLATE image/svg+xml

        AddOutputFilterByType DEFLATE application/rss+xml

        AddOutputFilterByType DEFLATE application/atom_xml

        AddOutputFilterByType DEFLATE application/x-javascript

        AddOutputFilterByType DEFLATE application/x-httpd-php

        AddOutputFilterByType DEFLATE application/x-httpd-fastphp

        AddOutputFilterByType DEFLATE application/x-httpd-eruby

        AddOutputFilterByType DEFLATE text/html

        # Netscape 4.x has some problems...

        BrowserMatch ^Mozilla/4 gzip-only-text/html

        # Netscape 4.06-4.08 have some more problems

        BrowserMatch ^Mozilla/4\.0[678] no-gzip

        # MSIE masquerades as Netscape, but it is fine

        # BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

        # NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48

        # the above regex won't work. You can use the following

        # workaround to get the desired effect:

        BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html

        # Don't compress images

        SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary

        SetEnvIfNoCase Request_URI \.(?:exe|t?gz|zip|bz2|sit|rar)$ no-gzip dont-vary

        SetEnvIfNoCase Request_URI \.pdf$ no-gzip dont-vary

        SetEnvIfNoCase Request_URI \.avi$ no-gzip dont-vary

        SetEnvIfNoCase Request_URI \.mov$ no-gzip dont-vary

        SetEnvIfNoCase Request_URI \.mp3$ no-gzip dont-vary

        SetEnvIfNoCase Request_URI \.mp4$ no-gzip dont-vary

        SetEnvIfNoCase Request_URI \.rm$ no-gzip dont-vary

        # Make sure proxies don't deliver the wrong content

        Header append Vary User-Agent env=!dont-vary

</Location>
```

[4] increase daemons performances, tuning the MinSpareServers, MaxSpareServers, MaxClients, MaxRequestsPerChild. You need to increase this values accordling to your load, read the apache 2.0 manual to understand better (it's hard to give you exact configuration without knowing the server and the problem). Please make sure to modify the righet MPM section into your http.conf

[5] disable in http.conf all the apache modules that you do not need (such as the proxy)

[6] disable reverse dns lookup into http.conf.

```
HostnameLookups Off
```

[7] enable SendFile support into http.conf

```
EnableSendfile on
```

[8] use a better filesystem, like XFS and tuning it to reach the highest performances:

format partition with cluster of 512byte size (you are on a server, not on a workstation, so you need little clustes size)

```
mkfs.xfs -d unwritten=0 -b size=512 -s size=512
```

into /etc/fstab mount XFS partitions with tuned options

```
/dev/sdbX               /               xfs     logbufs=8,logbsize=64K,barrier  0 2
```

obiovusly you need also to tweak your IDE/SATA/SCSI drives (enable write barriers, enable write cache ecc ecc use hdparm/sdparm)

[9] use PHP with error_reporting disabled, store the users sessions into a TMPFS, install a PHP pre-compiler such as dev-php/eaccelerator (this let you gain a lot of performances)

[10] there are a lot of others tips to speedup your server and reduce the load on it, for examples tweaks for spamassassin, qmail, courier-auth, mysql ecc ecc but i need more infos from you to help better  :Wink: 

----------

## think4urs11

you also may want to read

https://forums.gentoo.org/viewtopic-t-320970.html

https://forums.gentoo.org/viewtopic-t-436013.html

----------

## mudrii

!equilibrium

Big Thank you for sharing information is really helpful

Thx

----------

## Janne Pikkarainen

Moreaulf:  Still having these problems? This thread contains some nice tuning tips which do help in really high loaded environments, but I suspect your problem is more related to memory usage. There's a lot of swap used in your server and not so much cached or buffered memory. It usually means your server would need more memory, but since you don't seem to have very many hits per day and your server has 1 GB of RAM, it really should be plenty and most probably in this case some nasty-behaving script is sucking up all the RAM.

I suspect that during the load spikes your server has to swap. That has tendency to kill the performance very badly. What if you monitor the server with vmstat 1 or sar -W 1 999 whenever the load average is growing? And do you have an overall idea about what kind of scripts are regularly accessed? Maybe some wrong-behaving script is always the reason and sucks the resources away from all the other processes.

----------

## Moreaulf

Thank you everyone for your tips in this thread, especially to you !equilibrium.

I've spent a lot of time during this summer (when one should be out in the sun?) and really learned a lot.

I just want to thank you for helping out!   :Very Happy: 

----------

