# Gigabit network performance

## number_nine

I've got a fileserver (running gentoo) where I house all my media.  Transfers from the fileserver to the media PC (also running gentoo) are extremely slow.  However, transfers from the fileserver to other computers on the home LAN are at least 2x as fast.

Example: transferring a 2.4 GB file via NFS from the fileserver to the media PC takes 2:49 (about 15 MB/sec).  The same transfer on other machines took 1:20 or less.  I tried to do the same transfer via CIFS (samba mount), and it only transferred 17 MB in 5 minutes!

Any hint as to what could cause the severe network performance problem on the media PC?

Also, all computers on the LAN have their MTU set to 1500 (the default).  For an experiment, I tried using "jumbo frames", ala:

```
ifconfig eth0 mtu 9000
```

All my machines seem to support this, except the media PC, which says:

```
SIOCSIFMTU: Invalid argument
```

The machine is using the Abit  NF-M2 nView (alt link) motherboard.  This has the nVidia nForce 430 south bridge.  Does anyone know if this integrated NIC supports jumbo frames?  I have another machine using an integrated nvidia ethernet device, and it does support jumbo frames.

The only other thing that's unusual about the media PC is that it's connected by the longest cable.  But it's a purchased (not homemade) cable, and it's only 100 ft long.

Relevant info:

```

$ ifconfig eth0

eth0      Link encap:Ethernet  HWaddr 00:50:8D:98:D0:3C  

          inet addr:192.168.1.34  Bcast:192.168.255.255  Mask:255.255.0.0

          UP BROADCAST NOTRAILERS RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:13886363 errors:121 dropped:0 overruns:0 frame:0

          TX packets:9391786 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000 

          RX bytes:19843540531 (18924.2 Mb)  TX bytes:11549050130 (11014.0 Mb)

          Interrupt:23 Base address:0x2000

$ ethtool eth0

Settings for eth0:

        Supported ports: [ MII ]

        Supported link modes:   10baseT/Half 10baseT/Full 

                                100baseT/Half 100baseT/Full 

                                1000baseT/Full 

        Supports auto-negotiation: Yes

        Advertised link modes:  10baseT/Half 10baseT/Full 

                                100baseT/Half 100baseT/Full 

                                1000baseT/Full 

        Advertised auto-negotiation: Yes

        Speed: 1000Mb/s

        Duplex: Full

        Port: MII

        PHYAD: 1

        Transceiver: external

        Auto-negotiation: on

        Supports Wake-on: g

        Wake-on: d

        Link detected: yes

$ dmesg

...

forcedeth.c: Reverse Engineered nForce ethernet driver. Version 0.57.

ACPI: PCI Interrupt Link [APCH] enabled at IRQ 23

ACPI: PCI Interrupt 0000:00:14.0[A] -> Link [APCH] -> GSI 23 (level, low) -> IRQ 23

PCI: Setting latency timer of device 0000:00:14.0 to 64

forcedeth: using HIGHDMA

eth0: forcedeth.c: subsystem: 0147b:1c26 bound to 0000:00:14.0

dmfe: Davicom DM9xxx net driver, version 1.36.4 (2002-01-17)

...

$ lspci -vx

...

00:14.0 Bridge: nVidia Corporation MCP51 Ethernet Controller (rev a1)

        Subsystem: ABIT Computer Corp. Unknown device 1c26

        Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 23

        Memory at fe02b000 (32-bit, non-prefetchable) [size=4K]

        I/O ports at c800 [size=8]

        Capabilities: [44] Power Management version 2

00: de 10 69 02 07 00 b0 00 a1 00 80 06 00 00 00 00

10: 00 b0 02 fe 01 c8 00 00 00 00 00 00 00 00 00 00

20: 00 00 00 00 00 00 00 00 00 00 00 00 7b 14 26 1c

30: 00 00 00 00 44 00 00 00 00 00 00 00 05 01 01 14

...

$ cat /proc/interrupts

           CPU0       CPU1       

  0:  544743919  406279734    XT-PIC-XT        timer

  1:          0          2   IO-APIC-edge      i8042

  9:          0         26   IO-APIC-fasteoi   acpi

 12:          0          4   IO-APIC-edge      i8042

 14:        644      66411   IO-APIC-edge      ide0

 16:      49468   26872505   IO-APIC-fasteoi   nvidia

 18:       6364    2585281   IO-APIC-fasteoi   cx88[0], cx88[0], cx88[0]

 19:          0          3   IO-APIC-fasteoi   ohci1394

 20:          0          3   IO-APIC-fasteoi   ehci_hcd:usb1

 21:          0          0   IO-APIC-fasteoi   libata

 22:      17200    4233341   IO-APIC-fasteoi   libata, HDA Intel

 23:     143928  159405842   IO-APIC-fasteoi   ohci_hcd:usb2, eth0

NMI:     949175     949078 

LOC:  950874179  950874175 

ERR:          0

```

Any hints or ideas would be much appreciated.

Thanks!

----------

## poly_poly-man

What's the cpu/memory load on the media PC?

It's unlikely that it is network-related, as everything appears okay. 100ft is well under the 384ft (or is it m? - either way...) limit. 

hih,

poly-p man

----------

## pdr

I use cat-6 on the Gb part of my network..

Do your ports have a light indicator that tells if they are operating in 100Mb vs 1Gb mode? Mine have a light that is green for Gb, yellow for 100Mb, off for 10Mb..

If going through a router/switch - is the port used by the media pc Gb? Some have a mix of Gb and 100Mb ports..

----------

## number_nine

 *poly_poly-man wrote:*   

> What's the cpu/memory load on the media PC?
> 
> It's unlikely that it is network-related, as everything appears okay. 100ft is well under the 384ft (or is it m? - either way...) limit.

 

The machine is an X2 3800, running two instances of seti@home.  But the seti process is running at nice 19.  Also, all my other machines (file server included) are running seti@home, and it doesn't hurt network performance.  Machine is otherwise unloaded.

Here's something that did catch my eye though:

```

$ cat /proc/cpuinfo

processor   : 0

vendor_id   : AuthenticAMD

cpu family   : 15

model      : 75

model name   : AMD Athlon(tm) 64 X2 Dual Core Processor 3800+

stepping   : 2

cpu MHz      : 2200.020

cache size   : 512 KB

physical id   : 0

siblings   : 2

core id      : 0

cpu cores   : 2

fpu      : yes

fpu_exception   : yes

cpuid level   : 1

wp      : yes

flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm cr8_legacy

bogomips   : 4402.23

TLB size   : 1024 4K pages

clflush size   : 64

cache_alignment   : 64

address sizes   : 40 bits physical, 48 bits virtual

power management: ts fid vid ttp tm stc

processor   : 1

vendor_id   : AuthenticAMD

cpu family   : 15

model      : 75

model name   : AMD Athlon(tm) 64 X2 Dual Core Processor 3800+

stepping   : 2

cpu MHz      : 2200.020

cache size   : 512 KB

physical id   : 0

siblings   : 2

core id      : 1

cpu cores   : 2

fpu      : yes

fpu_exception   : yes

cpuid level   : 1

wp      : yes

flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm cr8_legacy

bogomips   : 972.38

TLB size   : 1024 4K pages

clflush size   : 64

cache_alignment   : 64

address sizes   : 40 bits physical, 48 bits virtual

power management: ts fid vid ttp tm stc

```

Notice that the bogomips value is different for the two processors?  By a factor of four!  I know that bogomips are "bogus", but I'd expect the value to be exactly the same for each core.  Also, if you look above at the output of "cat /proc/interrupts", you can see that CPU1 services the majority of interrupts.  In the case of eth0, it's by a factor of 1000.  But going by cpuinfo, the kernel detects CPU1 as having the lower BogoMIPS value.

Is it possible that I have a bum CPU?

----------

## number_nine

 *pdr wrote:*   

> I use cat-6 on the Gb part of my network..
> 
> Do your ports have a light indicator that tells if they are operating in 100Mb vs 1Gb mode? Mine have a light that is green for Gb, yellow for 100Mb, off for 10Mb..
> 
> If going through a router/switch - is the port used by the media pc Gb? Some have a mix of Gb and 100Mb ports..

 

Good idea.  I'll double check when I get home, but I'm 99% sure that the media PC is plugged into a gigabit port (this is my switch, FWIW).  Also, I did check the lights on the switch last night (all ports are running at Gb speed).

I suppose I should still check the port on the media PC itself.

Thanks!

----------

## markp2000

I ran this around for awhile myself.  I found that I had a bad NIC.  I was using the onboard NIC, but once I got a GB nic everything went back to normal. Do you have a spare nic to through in there?

Mark

----------

## number_nine

 *markp2000 wrote:*   

> I ran this around for awhile myself.  I found that I had a bad NIC.  I was using the onboard NIC, but once I got a GB nic everything went back to normal. Do you have a spare nic to through in there?

 

You had that weird bogomips problem as well, or just the crummy transfer speeds?

I actually just ordered a nice Intel PCIe gigabit NIC, but I was going to put it in the fileserver.  Looks like I'll be using it for the media PC now.

----------

## number_nine

Update: I used the net-analyzer/nttcp program to do some more testing.  This program sends data from memory (rather than disk), hopefully eliminating disk I/O bottlenecks (i.e. allows you to isolate what is actually being tested, which is just the network).

Using this tool, raw TCP and UDP throughput were fairly consistent between the file server and all other machines (including the media PC).

So it appears that the problem actually lies with NFS and/or disk I/O on the media PC.

Thanks!

By the way, that nttcp program is really easy to use.

----------

## markp2000

It is the slow transfer speed of files.  I have the same setup of a gentoo fileserver and mythbox media pc.  It was a bad integrated NIC.

----------

## number_nine

 *markp2000 wrote:*   

> It is the slow transfer speed of files.  I have the same setup of a gentoo fileserver and mythbox media pc.  It was a bad integrated NIC.

 

I don't think it's the NIC.  See my follow-up post to this one.  I went into more detail about using the nttcp program, as well as other info.  At this point, I'm pretty sure the problem has to do with slow disk I/O on the media PC.  But whether that is due to the hardware, filesystem, or some other factor, I'm not sure.

----------

