# How do I optimize for Gigabit Ethernet?

## Malakin

I'm sharing files between Linux and winxp using samba, both have identical Yukon Marvel gigabit ethernet cards and the connection is running at 1.0Gbit over a single cat5e crossover cable connecting the computers.

Currently using samba WinXP->Linux I'm getting about 9.5MB/s and Linux->WinXP I'm getting about 10MB/s. On either computer I can hook up a usb 2.0 drive and get about 26MB/s so it seems like I should be able to speed this up a fair bit.

After transferring about 10GB I have zero errors on the interface. Windows is reporting the link speed to be 1Gb/s and Linux is reporting the link speed to be 1Gb/s with full duplex mode. Everything seems to be running fine as far as I can tell.

So the question is what should I do to speed it up?

What's a good way to test the network speed without creating disk access?

----------

## Paladine01

Make sure you're using jumbo frames

----------

## devon

What about NetPerf or TTCP?

----------

## Malakin

So far I did "ifconfig eth0 mtu 8000" in Linux and then in winxp I editted the registry and changed the mtu for that interface to 8000 (and rebooted of course) and under advanced configuration for the device I had to change "maximum frame size" so I put it to 9000. (without the last step pings over 1472 wouldn't work)

After doing this I can ping each computer with packets over 7000 with the the do not fragment bit set so I assume it's working fine. 

Performance linux->winxp has increased to about 14MB/s but performance winxp->linux is about the same.

I'll have some more time to play with this later.

----------

## ewan.paton

are they both 64bit cards if not the pci bus is limited to 133megs/sec

----------

## Malakin

I can't seem to get ttcp working on Linux, even running the server and client both on Linux it still doesn't work, works fine if I run both on windows though, maybe a problem with the newer kernel?

I got netperf working with the server on windows and got 83MB/s, but it won't work with the server on Linux I get this error "establish_control: control socket connect failed: No error"

I only want to get about 20MB/s transfering files and then I'll be happy, there's no way I'll be able to exceed about 26MB/s anyways since the drives can't write any faster so the pci bus isn't an issue.

I guess I'll look into samba optimizations next.

----------

## Malakin

I tried quadrupling the samba buffers and a few other things and it made no difference, starting to think it's windows bogging things down. Maybe I can use my Knoppix cd in the winxp system so I can do some linux->linux tests. I'll try that tomorrow I guess.

----------

## eNTi

i've set up a gb network too. i'm having gentoo installed on two machines and i only get about 5-7MB/s, even though my slowest harddisk has at least 25MB/s buffered reads (hdparm). netio shows lots of higher numbers, i guess because it does ram to ram transfers only. 

i've bought a sata promise controller + hdd to speed up my network to my third pc, which runs on slackware. it's an old one with only about 300mhz, but i didn't think that this could be a problem. i put in the sata and had no speed up at all!!! (was 2.5MB/s). the problem could be the hard disk, on which my the system itself is located, because it has only 9MB/s buffered reads. the sata reads 25MB/s bufferd reads, which is terribly low. i wonder what's turning down this harddisk? 

anyways... i'm stuck. i can't figure out what really is the problem. isn't there some network guru running about in the gentoo forums? please respond  :Very Happy: . what are other possible bottlenecks? could 300mhz to less to process the data fast enough?

----------

## Mnemia

To everyone having Gigabit problems:

```

emerge iperf

```

It's a lot easier to use than ttcp or Netperf and supports a lot of advanced features like sending multiple streams simultaneously from multiple threads.

This might help too: http://www.psc.edu/networking/perf_tune.html

----------

## Malakin

 *Quote:*   

> even though my slowest harddisk has at least 25MB/s buffered reads

 If you're transfering from one computer to another then the bottleneck will be the write speed on one system, write speeds are quite a bit slower then read speeds.

 *Quote:*   

> i've bought a sata promise controller

 sata is only going to help if your hard drive is faster then the pata interface it's on which is usually not the case. New hard drives do about 50-60MB/s and pata can handle up to 133MB/s so it will be some time before sata makes any difference with a single drive. If you bought a new drive and put it on an computer then it could help but then you'd probably bottleneck on the cpu.

 *Quote:*   

> could 300mhz to less to process the data fast enough?

 This is probably the problem, you could check the cpu utilization while you try to transfer a large file.

----------

## kamikaze_gerbil

 *Malakin wrote:*   

> single cat5e crossover cable connecting the computers.
> 
> 

 

I think you'll need a cat6 crossover to improve your speeds. As I understand it all the wires need to be crossed to get the full speed.

----------

## Malakin

 *Quote:*   

> I think you'll need a cat6 crossover to improve your speeds. As I understand it all the wires need to be crossed to get the full speed.

 Well I'm already getting 83MB/s with netperf so it seems the bottleneck now is software, I'm trying to find some windows tweaks right now for file sharing but it seems windows file sharing just sucks and there isn't much you can do about it.

I'm going to try nfs next but haven't had a chance yet.

----------

## eNTi

 *Malakin wrote:*   

> 
> 
>  *Quote:*   i've bought a sata promise controller sata is only going to help if your hard drive is faster then the pata interface it's on which is usually not the case. New hard drives do about 50-60MB/s and pata can handle up to 133MB/s so it will be some time before sata makes any difference with a single drive. If you bought a new drive and put it on an computer then it could help but then you'd probably bottleneck on the cpu.
> 
> 

 

sorry, i don't understand this... my sata hd is not on any pata interface. i bought a "promise controller card", which goes into the pci-bus and has it's own bios and stuff. so it's not connected to my old pata system in any way.

the only thing, i could imagine, is, that if i transfere large amounts of data, the old harddisk is somehow used for swapping an thus slowing the transfere down that significantly.

i will make a cpu utilization test...

----------

## eNTi

yep. it's my cpu, it seems:

```

root@overlord $ ./netperf -f M -H enti -c -C -t TCP_STREAM

TCP STREAM TEST to enti

Recv   Send    Send                          Utilization       Service Demand

Socket Socket  Message  Elapsed              Send     Recv     Send    Recv

Size   Size    Size     Time     Throughput  local    remote   local   remote

bytes  bytes   bytes    secs.    MBytes  /s  % T      % T      us/KB   us/KB

 87380  16384  16384    10.00        12.26   99.12    6.60     78.982  5.258 

```

wow, that's interesting.

----------

## Malakin

 *Quote:*   

> sorry, i don't understand this... my sata hd is not on any pata interface. i bought a "promise controller card", which goes into the pci-bus and has it's own bios and stuff. so it's not connected to my old pata system in any way. 

 If your hard drive can read data at 30MB/s and your pata interface runs at 66MB/s then moving to sata won't make any difference. The only time it would help is if your pata was 33MB/s on maybe an old bx board and you had a newer hard drive capable of 40MB/s or more, then moving to sata might help.

----------

## eNTi

i do have strange readings with netperf. these are netperf outcomes from an athlon xp2000+ (sata, udma6,160GB, ext3) to an athlon thunderbird 1200 (pata, udma5, 80GB, ext3) over my gigabit network (all network hardware being from netgear) on cat6:

```

netperf -H hydra -f M -c -C

TCP STREAM TEST to hydra

Recv   Send    Send                          Utilization       Service Demand

Socket Socket  Message  Elapsed              Send     Recv     Send    Recv

Size   Size    Size     Time     Throughput  local    remote   local   remote

bytes  bytes   bytes    secs.    MBytes  /s  % T      % T      us/KB   us/KB

 87380  16384  16384    10.00        28.68   90.80    32.61    30.921  11.104

```

and vice versa:

```
netperf -H enti -f M -c -C 

TCP STREAM TEST to enti

Recv   Send    Send                          Utilization       Service Demand

Socket Socket  Message  Elapsed              Send     Recv     Send    Recv

Size   Size    Size     Time     Throughput  local    remote   local   remote

bytes  bytes   bytes    secs.    MBytes  /s  % T      % T      us/KB   us/KB

 87380  16384  16384    10.00        32.78   65.19    19.59    19.421  5.837 

```

the "slower" 1200mhz processor greatly outperforms the xp2000+ in terms of cpu utilizaiont. how ist this possible? how comes that a simple network benchmark requiers so much cpu power? this is rediculous!!!

here a testfile is being copied over my network from the tb to xp and vice versa via scp:

```

nt@eNTi $ scp hydra:/usr/portage/distfiles/nsco_beta19.zip .

nsco_beta19.zip                               100%  300MB   9.1MB/s   00:33 

```

```

nt@hydralisk $ scp enti:/tmp/nsco_beta19.zip .

nsco_beta19.zip                               100%  300MB   7.9MB/s   00:38 

```

both at about 86-99% cpu utilization. is there something wrong with 2.6.5 kernel sources? a friend of mine laughed about me today, because he's got a cpu utilization of maybe 5-20% under winxp! of course he's using 100mbit and get's "only" about 4-5MB/s. what am i to do???

----------

## Malakin

For me transfering files from linux->winxp the cpu utilization on winxp is 50-60% and linux is 12%. Transfering from winxp->Linux windows cpu utilization is 18-30% and Linux is about 10%.

----------

## eNTi

so... what do you think i can do about this?

----------

