# Network "ricing"

## pappy_mcfae

I have read many postings about ricers; folks who tweak their systems to within inches of complete chaos. I was wondering if there was a subset of ricing that concerns itself with tweaking network throughput. I have read a few tips and tricks on how to make your TCP zoom...and I have implemented some of them. However, I am wondering what all can be done, what settings can be manipulated, and so on. 

I get better network throughput now than I ever got from any other operating system. I've spent some time the last few days twiddling and tweaking, and I can get an initial throughput of twenty-seven MB/sec. It usually throttles down to something in the neighborhood of twelve after about ten seconds, and now only goes as low as about eight point five. I'd say that's pretty awesome.

I am interested in this because my roommate and I do sound recording work, and we make some pretty big files...and a lot of times, they have to move from machine to machine to be mixed, or to have special plugins used, plugins that only exist on one machine because it requires a parallel port dongle, and windoze.

So naturally, the faster I can move files from one machine to another, the better the work flows. 

Anyway, if there are other folks out there who live to see how many megs per second they can push through their switches/routers, I'd like to start a discussion on the tricks you have used to boost network throughput.

Thanks in advance for your time and consideration.

Blessed be!

Pappy

----------

## NeddySeagoon

pappy_mcfae,

Like any other attempt to improve the performance of the bottlenecks in a system, it needs to start with the identification of the bottlenecks. Its a serious subject, unlike ricing.

To start, what makes you think your network is the bottleneck in your end to end data transfer rate ?

Data is moved around the first system, hard drive, PCI bus, memory, PCI bus, network card

then over the network to the second system.  At the same time, there is a small amount of traffic the other way for ACK packets.

in the second system it goes network card, PCI bus, memory PCI bus, hard drive. So your system looks like   

```
(hard drive, PCI bus, memory, PCI bus, network card) == (network card, PCI bus, memory PCI bus, hard drive)
```

You need to test the transfer rates of each step before you can proceed.

----------

## pappy_mcfae

 *NeddySeagoon wrote:*   

> pappy_mcfae,
> 
> Like any other attempt to improve the performance of the bottlenecks in a system, it needs to start with the identification of the bottlenecks. Its a serious subject, unlike ricing.
> 
> To start, what makes you think your network is the bottleneck in your end to end data transfer rate ?

 

Experimentation with settings as listed in this article http://gentoo-wiki.com/HOWTO_TCP_Tuning. After simply cranking up the buffer settings as listed, I noticed an immediate improvement in throughput. That was without playing with the congestion settings. When I started playing with those, I noticed a decent increase yet again in throughput. So much so, that somehow, I have not only improved throughput on my home LAN, but I have also noticed that I get better data throughput coming in from my DSL connection. Where once, the most I could pull from the net was approximately 155 kbps, I reached peaks of 210 kbps, with a steady state average at least 165 kbps. That is up from a peak of 156, and a steady state average in the 130's. I have to believe that it was my fiddling with the settings that brought this into being. 

Such incremental improvements make me think that the bottleneck is in the transmission, not in the rest of the machine. And I say this because even my PII 450 can achieve such blazing throughput, although it can't maintain the speed as well as this machine or my other laptop.

 *Quote:*   

> Data is moved around the first system, hard drive, PCI bus, memory, PCI bus, network card
> 
> then over the network to the second system.  At the same time, there is a small amount of traffic the other way for ACK packets.
> 
> in the second system it goes network card, PCI bus, memory PCI bus, hard drive. So your system looks like   
> ...

 

I understand where the bottlenecks are. Take for instance the PII 450. I know that as it is set now, it is pretty much maxed out in the throughput department. While I'm sure the SATA drive and interface card would be blazingly fast in another machine, in that particular machine, the major limiting factor is processor speed. I know it's not memory, since it has a gig installed...and only uses a fourth of it, and although it has a swap partition, I have yet to show an indication that swapping is even happening. 

On my other laptop, the bottleneck is the fact that it only has 256 megs of ram. If it had more, I'd be able to goose the buffer sizes as listed in the HOWTO above. However, because of it's limited resources, I can only crank things up so far.

Speaking of which, as I was playing around, I noticed that the PI 450 could achieve better average throughput than the older laptop. Curious, but interesting. That pretty much cements my idea about where the bottleneck lies on the laptop.

Anyway, I know what you are saying. If you could give me some idea of benchmarking programs that would help me better identify where the true bottlenecks exist, I'd be very interested to know. I am serious about wanting my systems to turn and burn when it comes to TCP throughput. I want to squeeze as much out of my network as I can.

Thanks for your reply.

Blessed be!

Pappy

----------

## NeddySeagoon

pappy_mcfae,

CPU speed is rarely an issue as data is moved around using DMA.

The processor sets up the DMA engines and keeps track of transfers on an interrupt basis.

I wasn't thinking so much of amount of memory as memory bandwidth. Bigger buffers help the throughput but may increase the latency, for many things, thats an acceptable trade off.

You can use bonnie or hdparm (not as good) for checking disk to memory speed.

I don't know any memory to network card speed test programs but you can set a top limit on it by knowing that the theoretical max for a 32 bit 33MHz PCI bus is 133MB/sec. Thats for the entire bus.  You may have several PCI busses too.  This snippet of lspci shows three plus an AGP slot.

```
00:1e.0 PCI bridge: nVidia Corporation nForce2 AGP (rev c1)

01:07.0 Multimedia video controller: Brooktree Corporation Bt878 Video Capture (rev 02)

02:01.0 Ethernet controller: 3Com Corporation 3C920B-EMB Integrated Fast Ethernet Controller [Tornado] (rev 40)

03:00.0 VGA compatible controller: nVidia Corporation NV28 [GeForce4 Ti 4200 AGP 8x] (rev a1)
```

the first two digits are the bus number, so speading the load over a number of buses is good.

Network speed tests can be improvised . If you use NFS, you can do NFS transfers between files all on the same machine.

When you use /dev/null as a destination, you can check just the read speed. When you use /dev/zero for input, you can chek the write speed. However, /dev/zero compresses particularly well so make sure any compression is off.

If you are sure the network is your problem, you can play with bonding, which several physical interfaces behave as a single interface. Your switches need to support it.

I hope thats provided food for thought.

----------

## pappy_mcfae

Yes, it does clear up some issues...and I thank you for your speedy reply. Later on this evening, I'll see what I can do to crank up speed even more. I appreciate your advice and thoughts on this issue. When I'm done, I'll publish my results.

Blessed be!

Pappy

----------

## pappy_mcfae

I have just completed my first round of experimentation with network settings tweaks, and I have come up with some curious and completely unexpected results. Due to that fact, I am going to expand the testing slightly to not only time the copying of one large file, I am also going to try copying a directory with smaller files. 

The results are fascinating. In a nutshell, it seems at this point that the congestion control algorithm is more important than messing with the TCP buffer settings. I am really amazed by the results so far. As soon as I finish the next round of experiments, I'm going to post all my findings.

This is so cool! I am getting the serious geek woody doing this.

Blessed be!

Pappy

----------

## NeddySeagoon

pappy_mcfae,

Rule 1 of investigating anything like this is Assume Nothing.

Rule 2 is keep an open mind

Above all, enjoy yourself

----------

## pappy_mcfae

Below are the results of my experiments with increasing network speed. The results are pretty interesting. Unfortunately, the results disagree with the hypothesis. Forgive the length of this post. There's a lot of information contained. 

**Data flow tests to determine highest network throughput.**

The purpose of this experiment is to find out which settings give the best overall network throughput.

Hypothesis:

That the fully tweaked TCP settings will provide the greatest throughput, and the default settings will provide the least.

Procedure:

Each machine had three different sized file units copied from the local drive to the remote drive under test. The computers were tested with stock settings, which are determined by installed memory size and kernel configuration settings. Tweaked buffer settings were entered in /etc/sysctl.conf, and called by /etc/conf.d/local.start. The machines were then tested with and without the tweaked congestion protocol algorithm. Finally, the machines were tested with just the tweaked congestion algorithm, and the default buffer settings. The settings in the /etc/sysctl.conf file were reset and the machine was rebooted to insure stability after modifying settings.

For all systems, data was transfered via drag/drop using konqueror as the file manager. The link between machines was established by use of the "mount.cifs" command. Files were dragged from local drive to remote drive on the local system under test. Ktimer was set for 1500 seconds, and the file transfer was began at the same time as ktimer was started. The resulting time was subtracted from 1500, thereby giving the elapsed time for file transfer.

Test file information:

large file (LF) = 1 file sized 1.3 GB (1,374,377,984 bytes) 

medium file (MF) = 24 files sized 1.2 - 138 MB plus 4 directories (660,831,538 bytes total)

small file (SF) = 250 files sized 13.1 - 366.9 KB plus 1 directory (37,369,559 bytes total)

Systems under test:

(System A):

debby-anne-ii

PII 450, 851 Megs RAM, 20 Gig PATA, 160 Gig SATA, Rhine/Via 530TX PCI

(System B):

gen_tosh

Celeron 1.06 GHz laptop, 256 Megs RAM, 60 gig PATA, PCMCIA ADMtek 21x4x DEC-Tulip compatible  

(System C):

pappy-lap

Celeron 1.86 GHz laptop, 1 Gig RAM 120 Gig SATA, built in Realtek 8139C

Router/switch:

D-Link WBR-1310 switch/wireless access point

Test results:

Test 1: Default settings*

```
Data Flow..............   (LF)    (MF)    (SF)

(from > to)............   (sec)   (sec)   (sec)

(System A) > (System B)   142      98      9

(System A) > (System C)   138      92      9

(System B) > (System A)   160      108      14

(System B) > (System C)   151      123      37

(System C) > (System A)   138      90      6

(System C) > (System B)   132      82      7
```

This yields throughput ratings of:

```
Throughput(MB/s)          (LF)     (MF)     (SF)

(from > to)

(System A) > (System B)   9.7      6.7      4.1

(System A) > (System C)   9.9      7.2      4.1

(System B) > (System A)   8.6      6.1      2.7

(System B) > (System C)   9.1      5.4      1.0

(System C) > (System A)   10.0      7.3      6.2

(System C) > (System B)   10.4      8.1      5.3
```

Test 2: Full tweaks**

```
Data Flow..............   (LF)    (MF)    (SF)

(from > to)............   (sec)   (sec)   (sec)

(System A) > (System B)   165      98      9

(System A) > (System C)   138      92      9

(System B) > (System A)   168      83      10

(System B) > (System C)   215      76      9

(System C) > (System A)   143      80      7

(System C) > (System B)   133      72      6
```

This yields throughput ratings of:

```
Throughput(MB/s)          (LF)     (MF)     (SF)

(from > to)

(System A) > (System B)   8.3      6.7      4.1

(System A) > (System C)   10.0      7.2      4.1

(System B) > (System A)   8.2      7.9      3.7

(System B) > (System C)   6.4      8.7      4.1

(System C) > (System A)   9.6      8.3      5.3

(System C) > (System B)   10.3      9.2      6.2
```

Test 3: Tweaked buffers only***

```
Data Flow..............   (LF)    (MF)    (SF)

(from > to)............   (sec)   (sec)   (sec)

(System A) > (System B)   141      92      10

(System A) > (System C)   136      89      9

(System B) > (System A)   136      80      11

(System B) > (System C)   154      74      9

(System C) > (System A)   136      97      8

(System C) > (System B)   130      89      6
```

This yields throughput ratings of:

```
Throughput(MB/s)          (LF)     (MF)     (SF)

(from > to)

(System A) > (System B)   9.7      7.2      3.7

(System A) > (System C)   10.1      7.4      4.1

(System B) > (System A)   10.1      8.3      3.4

(System B) > (System C)   8.9      8.9      4.1

(System C) > (System A)   10.0      6.8      4.7

(System C) > (System B)   10.6      7.4      5.3
```

Test 4: Tweaked congestion protocol only****                                              

```
Data Flow..............   (LF)    (MF)    (SF)

(from > to)............   (sec)   (sec)   (sec)

(System A) > (System B)   132      98      9

(System A) > (System C)   136      91      9

(System B) > (System A)   136      82      10

(System B) > (System C)   150      75      9

(System C) > (System A)   139      87      7

(System C) > (System B)   130      80      6
```

This yields throughput ratings of:

```
Throughput(MB/s)          (LF)     (MF)     (SF)

(from > to)

(System A) > (System B)   10.4      6.7      4.1

(System A) > (System C)   10.1      7.3      4.1

(System B) > (System A)   10.1      8.0      3.7

(System B) > (System C)   9.2      8.8      4.1

(System C) > (System A)   9.9      7.6      5.3

(System C) > (System B)   10.6      8.3      6.2
```

* Default settings are the settings of the machine determined by installed memory size, and kernel parameters.

Default settings (debby-anne-ii):

net.core.wmem_max = 131071

net.core.rmem_max = 131071

net.core.wmem_default = 106496

net.core.rmem_default = 106496

net.ipv4.tcp_mem = 79200 105600 15840

net.ipv4.tcp_wmem = 4096 16384 3379200

net.ipv4.tcp_rmem = 4096 87380 3379200 

net.ipv4.tcp_congestion_control=cubic

Default settings (gen_tosh):

net.core.wmem_max = 131071

net.core.rmem_max = 131071

net.core.wmem_default = 106496

net.ipv4.tcp_mem = 22509 30012 45018

net.ipv4.tcp_wmem = 4096 16384 960384

net.ipv4.tcp_rmem = 4096 87380 960384

net.ipv4.tcp_congestion_control=cubic

Default settings (pappy-lap):

net.core.wmem_max = 131071

net.core.rmem_max = 131071

net.core.wmem_default = 105472

net.core.rmem_default = 105472

net.ipv4.tcp_mem = 96576 128768 193152

net.ipv4.tcp_wmem = 4096 16384 4120576

net.ipv4.tcp_rmem = 4096 87380 4120576

net.ipv4.tcp_congestion_control=cubic 

**Full tweaks involved using the settings indicated below. 

sysctl.conf settings (pappy-lap, debby-anne-ii):

net.core.rmem_max=16777216

net.core.wmem_max=16777216

net.ipv4.tcp_rmem=4096 87380 16777216

net.ipv4.tcp_wmem=4096 65536 16777216

net.ipv4.tcp_congestion_control=highspeed

sysctl.conf settings (gen_tosh):

net.core.rmem_max=4194304

net.core.wmem_max=4194304

net.ipv4.tcp_rmem=4096 87380 4194304

net.ipv4.tcp_wmem=4096 65536 4194304

net.ipv4.tcp_congestion_control=highspeed

***Tweaked buffers only involved tweaking the buffers as listed above. The default congestion algorithm (cubic) is left as is.

****Tweaked congestion protocol involved using default buffer settings. The congestion protocol is changed to Highspeed

Conclusions:

Firstly, it is clear from the data that the size of the TCP buffers is important to the flow of datagrams across a TCP/IP connection. As can be clearly seen, the meagerness of the default settings for gen_tosh gave it significantly slower throughput than the other two machines, even though it was the second fastest machine under test.

Secondly, for machines with limited resources, clearly, any tweaking will make the system much faster on the network. However, it is advisable to make sure you don't over do it. As can be seen, tweaking either the buffer size or the congestion algorithm results in a significant increase in throughput. While tweaking both also improves throughput on gen_tosh, note that giving it full tweaks yields less throughput than simply changing one or the other. This disagrees with the hypothesis. 

Third, even modest increases in buffer size greatly effects the speed of data transmission, especially when the default settings are so small due to a smaller amount of installed memory. 

Fourth, if your machine is fairly new, and fairly fast, you don't reap that large of a benefit from tweaking your settings over and above default. However, you do get a small performance increase, and if you are looking for all out performance, any plus is a good plus.

Fifth, and most importantly, if you truly desire to have increased network throughput, you will have to do some experimenting to get the maximum out of your machines. As can be seen, both debby-anne-ii and gen_tosh achieve their best overall throughput by tweaking the buffer settings, whereas pappy-lap achieves its best throughput with the full tweaks. With this information, I can now customize my networking so that each machine performs its best.

Blessed be!

PappyLast edited by pappy_mcfae on Tue Jan 22, 2008 11:46 pm; edited 3 times in total

----------

## servilvo

From speedguide.net

You can place the following code in /etc/rc.local or /etc/boot.local, depending on your distribution (so it gets automatically reapplied at startup). The TCP/IP parameters should be self-explanatory: we're basically setting the TCP Window to 256960, disabling timestamps (to avoid 12 byte header overhead), enabling tcp window scaling, and selective acknowledgements:

echo 256960 > /proc/sys/net/core/rmem_default

echo 256960 > /proc/sys/net/core/rmem_max

echo 256960 > /proc/sys/net/core/wmem_default

echo 256960 > /proc/sys/net/core/wmem_max

echo 0 > /proc/sys/net/ipv4/tcp_timestamps  

echo 1 > /proc/sys/net/ipv4/tcp_sack 

echo 1 > /proc/sys/net/ipv4/tcp_window_scaling

----------

## pappy_mcfae

 *servilvo wrote:*   

> From speedguide.net
> 
> You can place the following code in /etc/rc.local or /etc/boot.local, depending on your distribution (so it gets automatically reapplied at startup). The TCP/IP parameters should be self-explanatory: we're basically setting the TCP Window to 256960, disabling timestamps (to avoid 12 byte header overhead), enabling tcp window scaling, and selective acknowledgements:
> 
> echo 256960 > /proc/sys/net/core/rmem_default
> ...

 

First off, thanks for the tip on speedguide.net. it seems that site is the mother load for info about tweaking TCP, which is what I am all about at the moment...as can be seen in the experiment above. Secondly, while changing the system around is possible as you have posted, I find it more efficient, and cleaner to use sysctl. Sysclt is part of the sys-process/procps package. You can get it by typing: 

```
emerge -av sys-process/procps
```

Once that is done, you can enter your desired changes into /etc/sysctl.conf, and call sysctl at boot time by simply placing the following command in your /etc/conf.d/local.start file: 

```
sysctl -q 
```

This achieves the goal of tweaking your settings, AND it keeps the local.start file much smaller and cleaner. Also, with sysctl, you can enter 

```
sysctl -a
```

 and get a snapshot of your global settings. I like that idea. There's nothing like really getting a look at the software guts of your machine while it's running.

Blessed be!

Pappy

----------

