# Tuning network speed - benchmark tools?

## csim

Hi,

i just made a couple of tests running rsync over a 1000Mbps (Marvell 88E8056, sky2 module, kernel 2.6.31 on server, Realtek RTL8111/8168B, r8169 module, kernel 2.6.31, both capable of 1000Mbps)

the results:

```

rsync -rtgov (...)

```

gives over 1Gbps network(destination is RAID6, 6 7200rpm drives):

```

sent 4033886291 bytes  received 148 bytes  18632269.93 bytes/sec

total size is 4033393424  speedup is 1.00

```

over USB2.0 (destination is external 2,5" drive, 5400rpm) gives:

```

sent 4033886291 bytes  received 148 bytes  28307975.01 bytes/sec

total size is 4033393424  speedup is 1.0

```

Question is - shouldn't 1Gbps network be up to 2.5x faster?

It's a very simple network, no other data transfer is happening between server and client, both server and client have no firewall.

Does anyone know of any tricks how to speed it up? Kernel parameters etc. or tools?

Are there any benchmark tools available for measuring data transfer speed?

----------

## Sadako

net-misc/iperf is a "raw" network benchmarking tool in portage, and there are others too.

As for tricks to speed it up, on gigabit hardware you should try increasing the MTU, which can lead to impressive performance increases.

I have two nics using the r8169 module, both of which can handle an MTU of up to 7,200, compared to the typical ethernet MTU of 1,500.

9,000 bytes is typically the maximium with such "jumbo" frames, but you should set all devices on the lan segment to the same value, so you're stuck with the lowest maximum any of your nics can handle.

You can try `ifconfig eth0 mtu 7200` to change it, if your hardware can't handle it it'll simply return "SIOCSIFMTU: Invalid argument".

Also, you can simply add mtu_eth0="7200" to /etc/conf.d/net to have the mtu set upon boot.

Note, any network switch or anything between the machines must be able to handle jumbo frames also.

Other than that, you can tweak some tcp options via sysctl, take a look at what's under /proc/sys/net/ to see what's available.

I have the following in my /stc/sysctl.conf

```
net.core.rmem_max = 16777216

net.core.wmem_max = 16777216

net.ipv4.tcp_rmem = 4096 87380 16777216

net.ipv4.tcp_wmem = 4096 65536 16777216
```

They seem to be common increases over the defaults.

If your going to try any (or all) of this, you should see what the maximum you can get with iperf is before making any changes first, I'd be very interesting in seeing what the actual benefit of all this will be.

----------

## kernelOfTruth

*subscribes*

interesting topic

----------

## csim

Hi Hopeless,

 *Hopeless wrote:*   

> net-misc/iperf is a "raw" network benchmarking tool in portage, and there are others too.
> 
> 

 

Thanks  :Smile: 

 *Hopeless wrote:*   

> 
> 
> As for tricks to speed it up, on gigabit hardware you should try increasing the MTU, which can lead to impressive performance increases.
> 
> I have two nics using the r8169 module, both of which can handle an MTU of up to 7,200, compared to the typical ethernet MTU of 1,500.
> ...

 

I tried 7200 for both (marvell 88E8056 can handle up to 9000) but it actually degraded the performace, one time I got as low as 25.5Kbits/sec then went up to ~850Mbits/sec.

 *Hopeless wrote:*   

> 
> 
> Other than that, you can tweak some tcp options via sysctl, take a look at what's under /proc/sys/net/ to see what's available.
> 
> I have the following in my /stc/sysctl.conf
> ...

 

I googled up some sources that mention the same or similar values you've mentioned above:

```
http://blogs.techrepublic.com.com/opensource/?p=62?

http://fasterdata.es.net/TCP-tuning/linux.html

http://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.php

http://www.acc.umu.se/~maswan/linux-netperf.txt
```

but one source 

```
http://www.psc.edu/networking/projects/tcptune/#Linux
```

actually suggested to leave them alone for recent kernels that are capable of autotuning and in case autotuning is turned on, as it might degrade performance rather than improve it (I think it's what i was seeing with rsync after i set the values with sysctl -w on both server and client).

Autotuning can be checked with:

```
cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf
```

If set to 1, autotuning is in effect.

Here's an output of iperf which was run with following options:

```
iperf -c 10.0.0.1 -p 5001 -l 64K -r -t 21
```

The output:

```
------------------------------------------------------------

Client connecting to 10.0.0.1, TCP port 5001

TCP window size: 64.0 KByte (default)

------------------------------------------------------------

[  3] local 10.0.0.10 port 50162 connected with 10.0.0.1 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-21.0 sec  1.39 GBytes    569 Mbits/sec

------------------------------------------------------------

Server listening on TCP port 5001

TCP window size: 85.3 KByte (default)

------------------------------------------------------------

[  4] local 10.0.0.10 port 5001 connected with 10.0.0.1 port 37432

[ ID] Interval       Transfer     Bandwidth

[  4]  0.0-21.1 sec  2.27 GBytes    924 Mbits/sec

```

It seems as if the transfer rate would climb up to nearly 100%. The values changed only slightly if i repeated the test.

One thing i don't understand that rsync (and also cp) is still ~40% slower on my network(simple computer 2 computer network) compared to usb2.

Doesn't matter if i use rsync over NFS or directly.

----------

## csim

Hi,

i made some new benchmarks:

```
iperf -c 10.0.0.1 -p 5001 -l 64K -r -t ... 
```

```
------------------------------------------------------------

Client connecting to 10.0.0.1, TCP port 5001

TCP window size: 64.0 KByte (default)

------------------------------------------------------------

[  3] local 10.0.0.10 port 48605 connected with 10.0.0.1 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-42.0 sec  1.49 GBytes    305 Mbits/sec

------------------------------------------------------------

Server listening on TCP port 5001

TCP window size: 85.3 KByte (default)

------------------------------------------------------------

[  4] local 10.0.0.10 port 5001 connected with 10.0.0.1 port 45743

[ ID] Interval       Transfer     Bandwidth

[  4]  0.0-42.0 sec  3.79 GBytes    774 Mbits/sec
```

Here I have used -t 42 secs, the numbers vary only slightly when run repeatedly.

Strange thing here is, that when i run with -t 1 second i get the same results:

```
------------------------------------------------------------

Client connecting to 10.0.0.1, TCP port 5001

TCP window size: 64.0 KByte (default)

------------------------------------------------------------

[  3] local 10.0.0.10 port 38086 connected with 10.0.0.1 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0- 1.0 sec  30.4 MBytes    254 Mbits/sec

------------------------------------------------------------

Server listening on TCP port 5001

TCP window size: 85.3 KByte (default)

------------------------------------------------------------

[  4] local 10.0.0.10 port 5001 connected with 10.0.0.1 port 45744

[ ID] Interval       Transfer     Bandwidth

[  4]  0.0- 1.0 sec  92.3 MBytes    772 Mbits/sec
```

I believe it's got something to do with the autotune feature - each time a second transfer (within a session) is initiated, it's tuned to give max. results.

Another strange thing is that running this script (based on suggestions made by Hopeless, couple of sources and iperf manual)on both server and client doesn't seem to have any impact on performance or behavior:

```
ifconfig eth0 mtu 7200

sysctl -w net.ipv4.tcp_moderate_rcvbuf=0

sysctl -w net.core.rmem_max=16777216

sysctl -w net.core.wmem_max=16777216

sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"

sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"
```

I also made a test with cat running 

```
cat some_file | nc 10.0.0.1 2342
```

 on client and 

```
nc -l -p 2342 > some_file
```

 on server and was able to get ~40MBytes/sec. Perhaps the performance is limited as indicated above, but I'd need to test performance of HDDs of my client and server in order to know the limits.

EDIT:

my external USB2.0 disk gives me 

effective write speed of ~26 MByte/sec, which is exactly what rsync is reporting when copying to the drive.

My server (RAID6 consisting of 6 7200rpm SATA drives, chunk size 512k) gives  ~47 MByte/sec write speed almost exactly as 7200rpm client.

I have no idea why rsync is reporting sub ~20MBytes/sec speed on a gigabit link.

----------

## Sadako

You could eliminate disk io in the above test by reading from /dez/zero and to /dev/null, like so;

```
cat /dev/zero | nc 10.0.0.1 2342

nc -l -p 2342 > /dev/null
```

----------

## csim

But how to check how much data has been transfered?

Time could be watched with:

```
time -p cat /dev/zero | nc 10.0.0.1 2342 
```

i guess i could do 

```
watch ifconfig
```

 but is there any better way? Or even better - restrict transfer to some size?

----------

## Sadako

 *csim wrote:*   

> But how to check how much data has been transfered?
> 
> Time could be watched with:
> 
> ```
> ...

 Use dd;

```
time dd if=/dev/zero bs=4k count=1000000 | nc 10.0.0.1 2342
```

----------

## csim

Thanks Hopeless!   :Very Happy: 

Turns out time isn't needed at all

source:

```
dd if=/dev/zero bs=4k count=1000000 | nc 10.0.0.1 2342
```

destination:

```
nc -l -p 2342 > /dev/null
```

I'm getting ~65MBytes/sec, it's what dd is reporting.

----------

