# Gigabit Ethernet slow/unstable - recommended gigabit cards?

## Ge(nto)org

Hi!

The performance of my Gigabit Ethernet is really bad: When I try to scp a file from one host to the other, it first starts with about 15MB/s but immediately decreases step by step to 3MB/s. A NFS transfer has similar problems except that it is also stalled every Minute for a few seconds, then the transfer again has about 15MB/s which decrease again until it is stalled and again...

With 100MBit cards on the same hosts I get constant 11MB/s with scp and NFS!

I have a onboard  Marvell 88E8001 using the skge module, RTL-8169 cards using r8169, a Gigabit Ethernet Switch and Cat6 cables. I have tried the following setups:

skge <-> switch <-> r8169

r8169 <-> switch <-> r8169

skge <-> r8169

r8169 <-> r8169

Also the RTL-8169 cards have a LED indicating the speed and it says that they are connected at gigabit speed.

I'm really stuck now, and don't know what to check next   :Crying or Very sad: 

Any help greatly appreciated!Last edited by Ge(nto)org on Wed Jul 04, 2007 10:03 am; edited 1 time in total

----------

## alex.blackbit

is maybe the cable crap?

i had the problem with the nfs stalls before. i never solved it. if you find any hints, please let me know.

----------

## agent_jdh

If it's any consolation, I've got 2x r8169 based cards, netgear gigabit switch, cat6 cables, and it's pretty flakey as well.  A lot of the time on booting it doesn't even use gigabit speed, the switch light shows 100Mbit.  Often requires _several_ reboots to get 1Gb.  The transfer speeds varies wildly as well - sometimes it screams, sometimes it crawls, and on any large transfers it wobbles between the two.  I can't get the driver to work with large frames (mtu should be able to go up to about 7k) either, which is a shame.

To summarise (!), I don't think rtl8169 gigabit ethernet cards are very good in linux - be it a hardware or software problem.

----------

## Ge(nto)org

 *Quote:*   

> is maybe the cable crap? 

 

I tried different ones, 2 patchcables with the switch and a crossover without...

 *Quote:*   

> I don't think rtl8169 gigabit ethernet cards are very good in linux

 

Hmm, I have tested one of the cards together with a 100MBit card, and it worked as expected. So these cards are slower in gigabit mode than in 100MBit mode? weird!

----------

## Ge(nto)org

Now, I compiled in the "Use Rx and Tx Polling (NAPI) (EXPERIMENTAL)" option into the r8169 module and the following happened:

In the 

skge <-> switch <-> r8169 and

skge <-> r8169

setups I get a constant scp transfer rate of 19MB/s now, the NFS issue still exists (maybe it's only NFS related).

(The other setups

r8169 <-> switch <-> r8169

r8169 <-> r8169

still don't work.)

However, I didn't buy a Gigabit Ethernet card to get only 8MB/s more...

Could anyone recommend a Gigabit card that works smoothly under Linux?

Thx, georg

----------

## ljubo

I can recommend Intel PRO/1000 GT desktop adapter, or any other Intel Gigabit Ethernet Adapter.

Regards, Ljubo

----------

## alex.blackbit

that's right. intel cards have good driver support and work as expected.

----------

## godish

I 3rd that, Never had any issues with intel gig adaptors

----------

## Ge(nto)org

K, now I bought a Intel PRO/1000 GT desktop adapter which uses the e1000 module.

With scp I get about 17MB/s when using

e1000 <-> skge

e1000 <-> switch <-> skge.   :Mad: 

Grml, I really thought the Realtek chipset caused the problems, but since I don't think that Realtek and Marvell cards are crap the problem has to be somewhere else...

----------

## alex.blackbit

i am sorry to hear that. but... hmm, the problem MUST be somewhere.

----------

## Cyker

The RealTek Gbe chips have been reported to underperform, apparently due to IRQ spamming. Jumbo frames and enabling any off-loading options can help.

I have a Yukon2 and it's pretty quick, but has a tendency to randomly die during heavy transfers.

The built-in nVidia Forcdeth NIC is the fastest performer here.

Out of curiosity, how are you measuring the transfer?

I used to use BWM, but found that it was reporting speeds much lower than were really happening...

----------

## Ge(nto)org

 *Quote:*   

> Jumbo frames and enabling any off-loading options can help.

 

 :Shocked:  How would I do that?

 *Quote:*   

> Out of curiosity, how are you measuring the transfer? 

 

scp shows the transfer rate while copying the file - I have checked it, and it seems to be quite accurate.

----------

## pdr

I just scp'd a 5.5GB file from my server to my workstation. Pretty consistent 22-24MB/s, but had one "stall" where it dipped to 18MB/s.

The file is also on an NFS mount, so I timed a cp of it from the NFS mount to local drive. Now maybe caching is mucking with this (didn't run "sync"), but it took 2m31.359s real time, which comes out to 37MB/s. As a test (disk speed) I also timed copying it into the same directory on the server - time was 4m26.491s (which implies I am disk-bound on timing).

This file is on an encrypted, lvm, raid-5 partition on the server, but I'm happy with the speed, given the other benefits   :Twisted Evil: 

SETUP:

Server nic: I think I'm using the onboard skge, but could be the onboard forcedeth. Its an Asus A8N SLI Deluxe (love this motherboard for socket 939!)

Switch: netgear Gb switch. Running cat-6.

Workstation nic: PCIe card (Koutech PEN120 - USD 30 at Newegg.com) - is a Marvell 88E8053, using sky2 driver. I originally got this card because the previous motherboard (Asus P5V-LM or something - the microatx P5 board) had an attansic which I don't want to use (not in 2.6.20-r8?); new board (P5B Deluxe) has a Marvell chip so I could use it now, but I think I'll stick with the PCIe card for now.

----------

## flybynite

A common misconception is forgetting that gigabit ethernet is faster than the (shared) PCI bus its attached to, and faster than the hard disk thats sending it data.  Transfer rates peak due to initial transfer from cache to cache, then slow to disk/ PCI bus speeds.

PCI express mounted gigabit is faster, but is still limited by the disk or whatever your source/sink can deliver.

In disk to disk transfer, 100mb ethernet is only slightly slower than the real disk capability, so gigabit only transfers a bit faster because the disk is now the limit.

----------

## Cyker

 *Ge(nto)org wrote:*   

>  *Quote:*   Jumbo frames and enabling any off-loading options can help. 
> 
>  How would I do that?
> 
>  *Quote:*   Out of curiosity, how are you measuring the transfer?  
> ...

 

Jumbo frames basically just means having a bigger MTU than 1500, but any cards it's talking to must also be able to cope/support with the extra large frames.

I'm not sure if/which Linux NIC drivers have stuff like TCP Off-load and hardware checksum, but there is the experimental NAPI stuff which is supposed to reduce IRQ load...

The jumbo frames usually helps the most, as long as you have a very reliable network.

----------

## LinuxLoader

when u try to copy with scp u use small packets ..  thats a pain for the NIC . What are da CPU on the machines , try to use smp_affinity if they are smp . other thing is ... did u on the dma on the harddrive ? some times it`s didn`t workign and it`s a deam hell ... scp is not prefer for tests ... try ftp or http

----------

## Galahad

What do you expect from scp? It fully encrypts the whole connection through a ssh tunnel!

use ftp if you want speed.

----------

## Ge(nto)org

 *Quote:*   

> What do you expect from scp? It fully encrypts the whole connection through a ssh tunnel! 

 

Well, that is a good point. I should have concentrated on NFS from the beginning.

Also I got a bit confused by the earlier problems with the r8169...  

Now, I took a deep breath and gave my current setup (e1000 <-> switch <-> skge) another try:

I ignored that NFS stalls and did some tests with dd and /dev/zero. So, though the transfer rate goes up and down and transfer is even stalled every now and then I got pretty good values here: ~50MB/s while reading and ~15MB/s for writing. I still don't understand why speed is so low for writing because I tested the disk locally and got 55MB/s, however, I can live with that.

Thx for all your help!

Regards, georg

----------

## alex.blackbit

it's of course true that scp causes overhead because of the encryption. but isn't that done in the host cpu and not in the nic?

but nevertheless nfs can be a big problemchild. one more thing you can try is samba. it may sound strange... but my experience was good using it between unix-like machines.

----------

## chovy

 *Quote:*   

> 00:0a.0 Ethernet controller: Marvell Technology Group Ltd. 88E8001 Gigabit Ethernet Controller (rev 13)

 

I have the same onboard card on two linux machines, connected via 1000Mbit switch. I am only seeing about 4-5Mb/sec rate when I transfer a large file between the two.

The driver skge says it is "DEPRECATED" for Marvell Yukon chipset (no idea if this is what I should use or not).

 *Quote:*   

> Marvell Yukon Chipset / SysKonnect SK-98xx Support (DEPRECATED) 

 

----------

## chovy

I am definitely capped at 10Mbs -- I copied a fat file (700mb) and it started at 15Mbps, and quickly settled at 8.9 - 9.5Mbps.

Any idea how to get my speed up from 10Mbps to 1000Mbps (let alone 100Mbps would be an improvement at this point).

I feel like a config file somewhere is limiting my throughput.

----------

## xbmodder

Best performance with Intel gigabit (server) + Netgear Switch + Intel Gigabit (Server+)

Worst: ForceDeth + Cisco

Filesystem is cifs

----------

## chovy

Maybe, but i'm not even seeing 100Mbps let alone 1Gigabit/sec

I want to know why I am only transferring at 10Mbps first. I have the kernel config params set for my Gigabit card. Is there anything else I need to do?

I also read about Cat5e cables instead of Cat5....would that be limiting me to 10mbps?

----------

## eccerr0r

I've a bunch of gigabit cards, and had mixed results with them:

RTL8169 x 3 - 2 of them DIED or stopped working at Gbit speeds.  One of them still works fine.

RTL8110S - On-Motherboard (MSI) - OK

Syskonnect/3Com 3c940 (SKGE) - On-Motherboard (Abit) - OK

Broadcom Tigon3 - On-motherboard (Dell Inspiron) - OK

Marvell (Sky2) On-Motherboard (Foxconn) - OK

From past experience I can't recommend the RTL8169 unless you look carefully at the boards.  I think I notice a pattern of the failing boards, but not certain yet (only 3 boards!).  The on-board ethernets have worked OK for me though.

I haven't been able to get anywhere near Gbit saturation speeds.  I typically send my data unencrypted on my personal LAN, and so far on a 10/100/1000 shared cat5 system (meaning, no packet size hacking) I can only get 40MB/sec.  Many of my machines are restricted by PCI as well, but I'm happy if I can get 20MB/sec one way through the Gbit link... at least it's 2x what I can get over Fast Ethernet.

----------

