# Multiple NIC problems

## DeathCount

Hello everybody,

Having a few issues with my multi NIC server. I have three gigabit network ports (1x onboard and 2x on a dual port PCIe) and for some reason all three initialize like they should but only one ever gets used.

For example, my onboard nic always has all traffic go through it regardless of intended ip address. So no matter what ip I ping (192.168.1.25, 192.168.1.26, 192.168.1.27) it all goes through the onboard port.

I know this because when I unplug the cable from it, I am unable to ping any address even though the other two still have cables and lights on the router. I am using gentoo-sources-2.6.36-r5 with 

udev-151-r4. Thanks for your help!

My onboard NIC: (r8169)

03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)

My PCIe NICs: (e1000e)

02:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)

02:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)

ifconfig

```
eth0      Link encap:Ethernet  HWaddr 00:22:xx:xx:9a:7e  

          inet addr:192.168.1.25  Bcast:192.168.1.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:881 errors:0 dropped:0 overruns:0 frame:0

          TX packets:518 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000 

          RX bytes:315062 (307.6 KiB)  TX bytes:54882 (53.5 KiB)

          Interrupt:42 Base address:0xa000 

eth1      Link encap:Ethernet  HWaddr 00:15:xx:xx:10:22  

          inet addr:192.168.1.26  Bcast:192.168.1.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000 

          RX bytes:0 (0.0 KiB)  TX bytes:0 (0.0 KiB)

          Interrupt:18 Memory:fdfe0000-fe000000 

eth2      Link encap:Ethernet  HWaddr 00:15:xx:xx:10:23  

          inet addr:192.168.1.27  Bcast:192.168.1.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000 

          RX bytes:0 (0.0 KiB)  TX bytes:0 (0.0 B)

          Interrupt:19 Memory:fdfa0000-fdfc0000 
```

/etc/conf.d/net

```
config_eth0=( "192.168.1.25 netmask 255.255.255.0" )

routes_eth0=( "default via 192.168.1.1" )

config_eth1=( "192.168.1.26 netmask 255.255.255.0" )

routes_eth1=( "default via 192.168.1.1" )

config_eth2=( "192.168.1.27 netmask 255.255.255.0" )

routes_eth2=( "default via 192.168.1.1" )
```

70-persistent-net.rules

```
# PCI device 0x10ec:0x8168 (r8169)

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:22:xx:xx:9a:7e", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x105e (e1000e)

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:15:xx:xx:10:22", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:0x105e (e1000e)

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:15:xx:xx:10:23", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"
```

Kernel Config - Network Section

```
#

# Networking options

# 

CONFIG_PACKET=y

CONFIG_UNIX=y

# CONFIG_NET_KEY is not set

CONFIG_INET=y

CONFIG_IP_MULTICAST=y

CONFIG_IP_ADVANCED_ROUTER=y

CONFIG_ASK_IP_FIB_HASH=y

# CONFIG_IP_FIB_TRIE is not set

CONFIG_IP_FIB_HASH=y

CONFIG_IP_MULTIPLE_TABLES=y

CONFIG_IP_ROUTE_MULTIPATH=y

CONFIG_IP_ROUTE_VERBOSE=y

CONFIG_IP_PNP=y

CONFIG_IP_PNP_DHCP=y

CONFIG_IP_PNP_BOOTP=y

CONFIG_IP_PNP_RARP=y

# CONFIG_NET_IPIP is not set

# CONFIG_NET_IPGRE is not set

CONFIG_IP_MROUTE=y

# CONFIG_IP_MROUTE_MULTIPLE_TABLES is not set

CONFIG_IP_PIMSM_V1=y

CONFIG_IP_PIMSM_V2=y

# CONFIG_ARPD is not set

CONFIG_SYN_COOKIES=y

# CONFIG_INET_AH is not set

# CONFIG_INET_ESP is not set

# CONFIG_INET_IPCOMP is not set

# CONFIG_INET_XFRM_TUNNEL is not set

# CONFIG_INET_TUNNEL is not set

# CONFIG_INET_XFRM_MODE_TRANSPORT is not set

# CONFIG_INET_XFRM_MODE_TUNNEL is not set

# CONFIG_INET_XFRM_MODE_BEET is not set

CONFIG_INET_LRO=y

# CONFIG_INET_DIAG is not set

CONFIG_TCP_CONG_ADVANCED=y

CONFIG_TCP_CONG_BIC=m

CONFIG_TCP_CONG_CUBIC=y

CONFIG_TCP_CONG_WESTWOOD=m

CONFIG_TCP_CONG_HTCP=m

# CONFIG_TCP_CONG_HSTCP is not set

# CONFIG_TCP_CONG_HYBLA is not set

# CONFIG_TCP_CONG_VEGAS is not set

# CONFIG_TCP_CONG_SCALABLE is not set

# CONFIG_TCP_CONG_LP is not set

# CONFIG_TCP_CONG_VENO is not set

# CONFIG_TCP_CONG_YEAH is not set

# CONFIG_TCP_CONG_ILLINOIS is not set

CONFIG_DEFAULT_CUBIC=y

# CONFIG_DEFAULT_RENO is not set

CONFIG_DEFAULT_TCP_CONG="cubic"

CONFIG_TCP_MD5SIG=y

# CONFIG_IPV6 is not set

CONFIG_NETWORK_SECMARK=y

# CONFIG_NETWORK_PHY_TIMESTAMPING is not set

CONFIG_NETFILTER=y

# CONFIG_NETFILTER_DEBUG is not set

CONFIG_NETFILTER_ADVANCED=y
```

----------

## Jaglover

You can't have multiple NICs on same subnet just like that.

http://www.google.com/linux?hl=en&safe=off&q=nic+bonding&btnG=Search

----------

## your_WooDness

Hi DeathCount,

this is because of the so called "arp flux". If you have several NICs in one subnet, the first NIC will answer on the MAC layer address for all addresses on the subnet. So all arp tables of switches and other servers will have only one MAC address for, in your case three IP addresses. You can check this by using arping from a host pingin the host with the three NICs. You will get one and the same MAC address for each IP address.

There are different ways to work around this. Not sure, but I think there is a option somewhere in /proc you can alter. Search for "arp flux". Best would be to configure a bonding on the NICs if you only use one subnet.

WooD

----------

## TJNII

That config will never work the way you expect, regardless of ARP.  Ask yourself these questions: What is my IP?  What are my routes?  What ethernet device is used for which subnet?  What IP do other machines use to connect to me?

Ethernet bonding isn't going to work unless you have something on the other end which supports bonding.  If you don't have a bonding capable switch, you can't share traffic across all 3 nics transparently.  If you have a home switch, you're SOL.  Buy a managed switch.

You can hack around this with routing, by having all local traffic go through eth0 and all the outside traffic go through eth1.  That is just a suboptimal hack using 2 Nics, though.

Buy a bridge capable switch and set up link bonding.  That will do what you want.

----------

## DeathCount

Thanks for the replies everybody. What about setting up bonding with mode 6? I hear that has the benefits of 802.3ad without special hardware. Am I incorrect?

----------

## TJNII

AFAIK that isn't true bonding, then.  With true bonding all the ports function as one with one MAC address.  That is why you need a bonding capable switch, as a regular switch will be very confused by having 3 pots share a MAC address.

However, that is just semantics.  If it will do what you want, go for it.  Keep in mind that that won't load balance across multiple NICs on a per connection basis using that methodology, though.  Traffic to one endpoint will go through one NIC, not all 3.  However, traffic to 3 endpoints should use all 3 NICs.

----------

## TJNII

Also, out of curiosity, what are you doing that requires so much chutzpah from your network?  I've never been able to choke a 1 gig connection, I've always hit other throttles first.

----------

## DeathCount

I'm not actually hitting a bottleneck really but it's more for multi-tasking. I originally set this up like this for LAN parties. The onboard nic would dedicate to serving all game server traffic while the other 2 nics would serve patches, files, isos, etc. full speed to multiple users without affecting game server traffic. When I'm not having a LAN party, I still would want the onboard to continue to serve game server traffic for my online gaming community and other "server" stuff such as SSH while the other two would do file sharing and 1080p video streams to multiple devices throughout the house at the same time. So in reality, no real reason besides multi-tasking. 

Same reason this server has a quad-core processor and 2 hard drives setup in raid0 for sharing.... No real reason besides multi-tasking, and the fact I can do it.  :Wink: 

Thanks again for all your help everybody.

----------

