# resolving net slownes compared to a RHEL5 install (same box)

## m27315

Hi,

I have a dual boot computer, which boot either Gentoo or CentOS5.2 (similar to RHEL5.2).  On HDD limited apps, the Gentoo installation is faster; however, on NFS (network traffic) limited apps, the CentOS5.2 box is much faster.  Here are my results:

```
Distro          App#1   App#2   App#3

Gentoo 2008.0   174.7   176.67   2346.91

CentOS 5.2        147.9   142.33   1539.57
```

In both cases, I am using the exact same hardware, except HDD.  However, since this is a network based problem (NFS), I don't think HDD is that significant.  Plus in my HDD limited apps, the Gentoo conf was slightly faster.

What could be different that would account for this difference?

Thanks!

----------

## aronparsons

Post the mount options from /proc/mounts for each OS.  What are the export options from the NFS server?

----------

## m27315

 *aronparsons wrote:*   

> What are the export options from the NFS server?

 

I don't have logon access to the NFS server.  Is there some way I can find the info on the client? ... Thx!  (BTW, I'll post the other data shortly...)

----------

## m27315

Slow Gentoo:

```
$ uname -a

Linux ratchet 2.6.25-gentoo-r7 #1 SMP Mon Jul 28 14:32:02 CDT 2008 x86_64 Intel(R) Core(TM)2 Extreme CPU X9650 @ 3.00GHz GenuineIntel GNU/Linux

$ cat /proc/mounts 

rootfs / rootfs rw 0 0

/dev/root / ext3 rw,noatime,errors=continue,data=ordered 0 0

proc /proc proc rw,nosuid,nodev,noexec 0 0

sysfs /sys sysfs rw,nosuid,nodev,noexec 0 0

udev /dev tmpfs rw,nosuid,size=10240k,mode=755 0 0

devpts /dev/pts devpts rw,nosuid,noexec,gid=5,mode=620 0 0

/dev/sda5 /var ext3 rw,noatime,errors=continue,data=ordered 0 0

/dev/sda6 /local ext3 rw,noatime,errors=continue,data=ordered 0 0

shm /dev/shm tmpfs rw,nosuid,nodev,noexec 0 0

usbfs /proc/bus/usb usbfs rw,nosuid,noexec,devgid=85,devmode=664 0 0

binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec 0 0

securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec 0 0

emcnas:/project /project nfs ro,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=20,retrans=10,sec=sys,addr=172.22.16.109 0 0

emcnas:/proje /project/e nfs rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=20,retrans=10,sec=sys,addr=172.22.16.109 0 0

emcnas:/home /home nfs rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=20,retrans=10,sec=sys,addr=172.22.16.109 0 0

emcnas:/tools /tools nfs ro,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=20,retrans=10,sec=sys,addr=172.22.16.109 0 0
```

Fast CentOS 5.2:

```
$ uname -a

Linux ratchet.mydomain.com 2.6.18-92.el5 #1 SMP Tue Jun 10 18:51:06 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

$ cat /proc/mounts

rootfs / rootfs rw 0 0

/dev/root / ext3 rw,data=ordered 0 0

/dev /dev tmpfs rw 0 0

/proc /proc proc rw 0 0

/sys /sys sysfs rw 0 0

none /selinux selinuxfs rw 0 0

/proc/bus/usb /proc/bus/usb usbfs rw 0 0

devpts /dev/pts devpts rw 0 0

/dev/sdb1 /boot ext2 rw 0 0

/dev/sdb5 /.snapshots ext3 rw,data=ordered 0 0

/dev/sdb6 /var ext3 rw,data=ordered 0 0

/dev/sdb7 /local ext3 rw,data=ordered 0 0

tmpfs /dev/shm tmpfs rw 0 0

none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0

sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0

emcnas:/project /project nfs ro,vers=3,rsize=32768,wsize=32768,hard,intr,proto=tcp,timeo=20,retrans=10,sec=sys,addr=emcnas 0 0

emcnas:/proje /project/e nfs rw,vers=3,rsize=32768,wsize=32768,hard,intr,proto=tcp,timeo=20,retrans=10,sec=sys,addr=emcnas 0 0

emcnas:/home /home nfs rw,vers=3,rsize=32768,wsize=32768,hard,intr,proto=tcp,timeo=20,retrans=10,sec=sys,addr=emcnas 0 0

emcnas:/tools /tools nfs ro,vers=3,rsize=32768,wsize=32768,hard,intr,proto=tcp,timeo=20,retrans=10,sec=sys,addr=emcnas 0 0
```

Incidentally, I also created a small perl script to record all the /proc/sys/(net|fs|kernel) settings in the CentOS environment, so I could duplicate most of them in the Gentoo environment.  Unfortunately, it did not help this problem, but here's the script, FWIW:

```
#!/usr/bin/perl

my (%data, $maxValue, $maxFile);

sub write_sys {

  my ($dir) = @_;

  opendir(DIR, $dir) || die("(E) record_net.pl:  Unable to open dir ($dir) - $!");

  my $file;

  foreach $file (readdir(DIR)) {

    unless (($file eq ".") || ($file eq "..")) {

      $file = "$dir/$file";

      if (-d $file) {

        write_sys($file);

      } else {

        my $value = `cat $file`;

        chomp $value;

        $maxValue = length($value) if length($value) > $maxValue;

        $maxFile = length($file) if length($file) > $maxFile;

        $data{$file} = $value;

      }

    }

  }

}

write_sys(@ARGV);

my $format = "echo %${maxValue}s > %s\n";

my $key;

foreach $key (sort keys %data) {

  printf $format, $data{$key}, $key;

}
```

I can post the file this created somewhere, but it's huge!  Again, it did not make a difference anyway.   :Sad: 

(Note and Warning:  After I rebooted from importing the CentOS settings, my Gentoo box would either hang during startup, act incredibly slow, or be unable to access the network in general.  Don't try the above step!  I was able to fix the problem by applying the Gentoo settings, which I saved before applying the CentOS settings, rebooting several times and finally turning the box off for a while. - edited 8/7/2008)

Since the NFS mount options look basically the same, I'm guessing the problem is in the kernel or driver.  Is there any other place there could be a problem?

Thanks!Last edited by m27315 on Thu Aug 07, 2008 3:58 pm; edited 1 time in total

----------

## m27315

Well, I have tried several things, but I am not getting anywhere.  First, does any one know of a good way to profile a process (possibly process tree) to see where it is spending its time?

I have used gkrellm to watch the job execute in both environments, and I noticed the following things:The CentOS process camps out on Core #0 (QX9650 - Quad-core), and doesn't budge until the process is complete.  However, the Gentoo versions seems to spread the work evenly across all 4 processes.  Since this is a single-thread app, this approach means that I could be losing time as the process is consistently shifted from one process to the other.  However, I have used "taskset -c 0 <myapp>" to force the process affinity to core 0.  This shaved off a second, maybe two, but it did not buy back the 20-40 seconds difference in my small benchmark.  Although interesting, I do not think this is the source of my problem.

During run-time, there is a period where there is some moderate disk IO (peak 1-15 MB/s, but I can sustain 20-30 MB/s, so I do not consider it heavy).  The CentOS box seems to work through this period more quickly, which makes me think this is a network, NFS client issue for the Gentoo box....  To tetst this theory, I changed my app's "run directory" to local disk.  This dropped the run-time significantly to 20% faster than the CentOS box.

I am wondering if it is a kernel difference, so I tried booting into the Gentoo env with the CentOS5.2 kernel and vice versa; however, I ran into various problems because the kernels or environments are so different...

I tried stripping down the Gentoo kernel, turning off everything that I could (netfilter, QoS routing, Power management, even SMP and HT), and although these reductions helped to shave off a few seconds, I could never get the Gentoo 2.6.25 version to perform close to the CentOS 2.6.18 version.So, basically, I'm still stuck.  I'm trying to move the NFS data to local disk to isolate the network variable, but I would appreciate any suggestions, because if I can't get this resolved, I will be forced to migrate to CentOS, because of performance reasons!    :Shocked: 

----------

## m27315

After looking a the CentOS 5.2 kernel config, I realized that it has been patched to use FS-Cache, and the Gentoo kernel has not!  I will try to patch my Gentoo kernel to see if it makes a difference....

----------

## SeaTiger

Thank you very much for sharing your experience in this topic.

I am working with diskless client environment and find it very difficult to tune NFS performance. You posting may share some clues on it. I will try to convince my boss to get a testing environment to test this. I will post more once I get some result. But it may takes up to a week to get things ready so please be patient.

----------

## m27315

I have seen more dramatic difference on outbound (WAN) traffic, but tuning the IP stack parameters can help also.  Try comparing these (execute at startup, or before benchmarking):

Gentoo 2008:

```
# TCP Tunining:

echo 1                                          > /proc/sys/net/ipv4/tcp_moderate_rcvbuf

echo 1                                          > /proc/sys/net/ipv4/tcp_sack

echo 1                                          > /proc/sys/net/ipv4/tcp_timestamps

echo 1                                          > /proc/sys/net/ipv4/tcp_window_scaling

echo 0                                          > /proc/sys/net/ipv4/conf/all/arp_ignore

echo 0                                          > /proc/sys/net/ipv4/conf/all/arp_announce

echo 1                                          > /proc/sys/net/ipv4/conf/eth1/rp_filter

echo 1                                          > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts

echo 0                                          > /proc/sys/net/ipv4/icmp_echo_ignore_all

echo 1                                          > /proc/sys/net/ipv4/icmp_ignore_bogus_error_responses

echo 100                                        > /proc/sys/net/ipv4/icmp_ratelimit

echo 500                                        > /proc/sys/net/ipv4/route/error_burst

echo 100                                        > /proc/sys/net/ipv4/route/error_cost

echo 2                                          > /proc/sys/net/ipv4/route/redirect_load

echo 2048                                       > /proc/sys/net/ipv4/route/redirect_silence

echo 32768      61000                           > /proc/sys/net/ipv4/ip_local_port_range

echo 60                                         > /proc/sys/net/ipv4/tcp_fin_timeout

echo 2                                          > /proc/sys/net/ipv4/tcp_frto

echo 75                                         > /proc/sys/net/ipv4/tcp_keepalive_intvl

echo 9                                          > /proc/sys/net/ipv4/tcp_keepalive_probes

echo 65536                                      > /proc/sys/net/ipv4/tcp_max_orphans

echo 0                                          > /proc/sys/net/ipv4/tcp_orphan_retries

echo 0                                          > /proc/sys/net/ipv4/tcp_rfc1337

echo 0                                          > /proc/sys/net/ipv4/tcp_tw_recycle

echo 0                                          > /proc/sys/net/ipv4/tcp_tw_reuse

echo 5                                          > /proc/sys/net/ipv4/tcp_syn_retries

echo 5                                          > /proc/sys/net/ipv4/tcp_synack_retries

echo 385824     514432          771648          > /proc/sys/net/ipv4/tcp_mem

echo 4096       87380           4194304         > /proc/sys/net/ipv4/tcp_rmem

echo 4096       16384           4194304         > /proc/sys/net/ipv4/tcp_wmem

#echo 4096      16384           131072          > /proc/sys/net/ipv4/tcp_wmem

#echo 4096      87380           174760          > /proc/sys/net/ipv4/tcp_rmem

echo 1000                                       > /proc/sys/net/core/netdev_max_backlog

echo 128                                        > /proc/sys/net/core/somaxconn

echo 20480                                      > /proc/sys/net/core/optmem_max

echo 124928                                     > /proc/sys/net/core/rmem_default

echo 131071                                     > /proc/sys/net/core/rmem_max

echo 124928                                     > /proc/sys/net/core/wmem_default

echo 131071                                     > /proc/sys/net/core/wmem_max

# Other Tuning

ulimit -n 1024

ulimit -f unlimited

echo 356838                                     > /proc/sys/fs/file-max

ifconfig eth1 mtu 1500

ifconfig eth1 txqueuelen 1000

echo 0                                          > /proc/sys/vm/nr_hugepages

echo 250     32000   32      128                > /proc/sys/kernel/sem

echo 33554432                                   > /proc/sys/kernel/shmmax
```

CentOS5:

```
# TCP Tunining: RHEL5 (CentOS5)

echo 1                                          > /proc/sys/net/ipv4/tcp_moderate_rcvbuf

echo 1                                          > /proc/sys/net/ipv4/tcp_sack

echo 1                                          > /proc/sys/net/ipv4/tcp_timestamps

echo 1                                          > /proc/sys/net/ipv4/tcp_window_scaling

echo 0                                          > /proc/sys/net/ipv4/conf/all/arp_ignore

echo 0                                          > /proc/sys/net/ipv4/conf/all/arp_announce

echo 1                                          > /proc/sys/net/ipv4/conf/eth1/rp_filter

echo 1                                          > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts

echo 0                                          > /proc/sys/net/ipv4/icmp_echo_ignore_all

echo 1                                          > /proc/sys/net/ipv4/icmp_ignore_bogus_error_responses

echo 1000                                       > /proc/sys/net/ipv4/icmp_ratelimit

echo 5000                                       > /proc/sys/net/ipv4/route/error_burst

echo 1000                                       > /proc/sys/net/ipv4/route/error_cost

echo 20                                         > /proc/sys/net/ipv4/route/redirect_load

echo 20480                                      > /proc/sys/net/ipv4/route/redirect_silence

echo 32768      61000                           > /proc/sys/net/ipv4/ip_local_port_range

echo 60                                         > /proc/sys/net/ipv4/tcp_fin_timeout

echo 0                                          > /proc/sys/net/ipv4/tcp_frto

echo 75                                         > /proc/sys/net/ipv4/tcp_keepalive_intvl

echo 9                                          > /proc/sys/net/ipv4/tcp_keepalive_probes

echo 65536                                      > /proc/sys/net/ipv4/tcp_max_orphans

echo 0                                          > /proc/sys/net/ipv4/tcp_orphan_retries

echo 0                                          > /proc/sys/net/ipv4/tcp_rfc1337

echo 0                                          > /proc/sys/net/ipv4/tcp_tw_recycle

echo 0                                          > /proc/sys/net/ipv4/tcp_tw_reuse

echo 5                                          > /proc/sys/net/ipv4/tcp_syn_retries

echo 5                                          > /proc/sys/net/ipv4/tcp_synack_retries

echo 196608     262144          393216          > /proc/sys/net/ipv4/tcp_mem

echo 4096       87380           174760          > /proc/sys/net/ipv4/tcp_rmem

echo 4096       16384           131072          > /proc/sys/net/ipv4/tcp_wmem

echo 4096               87380   4194304         > /proc/sys/net/ipv4/tcp_rmem

echo 4096               16384   4194304         > /proc/sys/net/ipv4/tcp_wmem

echo 1000                                       > /proc/sys/net/core/netdev_max_backlog

echo 128                                        > /proc/sys/net/core/somaxconn

echo 20480                                      > /proc/sys/net/core/optmem_max

echo 126976                                     > /proc/sys/net/core/rmem_default

echo 131071                                     > /proc/sys/net/core/rmem_max

echo 126976                                     > /proc/sys/net/core/wmem_default

echo 131071                                     > /proc/sys/net/core/wmem_max

# Other Tuning

ulimit -n 1024

ulimit -f unlimited

echo 305999                                     > /proc/sys/fs/file-max

echo 357678                             > /proc/sys/fs/file-max

ifconfig eth1 mtu 1500

ifconfig eth1 txqueuelen 1000

echo 0                                          > /proc/sys/vm/nr_hugepages

echo 250     32000   32      128                > /proc/sys/kernel/sem

echo 68719476736                                > /proc/sys/kernel/shmmax
```

Here are some more links, from which I created other similar files for testing:

http://www.psc.edu/networking/projects/tcptune/#Linux

http://www.soe.ucsc.edu/pipermail/genome-mirror/2007-October/000633.html

http://www.speedguide.net/read_articles.php?id=121

http://janforman.com

http://proj.sunet.se/E2E/tcptune.html

http://www.spec.org/jAppServer2004/results/res2006q3/jAppServer2004-20060719-00031.txt

http://lists.digium.com/pipermail/asterisk-dev/2003-May/000702.html

HTH

----------

