# Simple multiple NICs setup - does not work + weirdness

## albanard

I simply have 3 network cards that I would like to assign 3 IPs to. I'm not sure what I'm doing wrong and any help would be much appreciated as I'm a noob.

BACKGROUND:

I have a box with 3 network cards. I would like to assign each card a seperate IP. All cards go through the same gateway.

PROBLEM:

I've followed the install instructions, but when I reboot only 1 network card works, even though they all seem to be detected and I believe I have configured them properly.

WEIRDNESS: 

Here's the weird part. If I only plug the one network card that is working into the network I can still ping all three ip addresses.

QUESTIONS:

a) I thought an IP address was bound through the conf files to a specific device. Why do all 3 ip addresses respond to ping when only one network card is plugged into the network?

b) Why aren't my other 2 network cards working?

SYSTEM INFO AND CONF FILES:

(replaced mac numbers with A,B,C etc. , but left unique tail)

-----------

Seems to detect cards OK

-----------

www root # dmesg 

....

eepro100.c:v1.09j-t 9/29/99 Donald Becker http://www.scyld.com/network/eepro100.html

eepro100.c: $Revision: 1.36 $ 2000/11/17 Modified by Andrey V. Savochkin <saw@saw.sw.com.sg> and others

eth0: Intel Corp. 82557/8/9 [Ethernet Pro 100], A:B:C:D:11:B7, IRQ 17.

  Board assembly 567812-052, Physical connectors present: RJ45

  Primary interface chip i82555 PHY #1.

  General self-test: passed.

  Serial sub-system self-test: passed.

  Internal registers self-test: passed.

  ROM checksum self-test: passed (0xd0a6c714).

Intel(R) PRO/1000 Network Driver - version 5.2.16-k2

Copyright (c) 1999-2003 Intel Corporation.

eth1: Intel(R) PRO/1000 Network Connection

eth2: Intel(R) PRO/1000 Network Connection

...

eth0: no IPv6 routers present

eth1: no IPv6 routers present

eth2: no IPv6 routers present

-----------

each eth seems to have a different "HWaddr"

-----------

www root # ifconfig

eth0      Link encap:Ethernet  HWaddr A:B:C:D:11:B7

          inet addr:192.168.0.4  Bcast:192.168.0.255  Mask:255.255.255.0

          inet6 addr: a::b:c:d:11b7/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:328 errors:0 dropped:0 overruns:0 frame:0

          TX packets:208 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:31457 (30.7 Kb)  TX bytes:44159 (43.1 Kb)

          Interrupt:17 Base address:0x2000

eth1      Link encap:Ethernet  HWaddr A:B:C:D:10:61

          inet addr:192.168.0.5  Bcast:192.168.0.255  Mask:255.255.255.0

          inet6 addr: a::b:c:d:1061/64 Scope:Link

          UP BROADCAST MULTICAST  MTU:1500  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

          Interrupt:53 Base address:0xc400 Memory:fe9c0000-fe9e0000

eth2      Link encap:Ethernet  HWaddr A:B:C:D:10:60

          inet addr:192.168.0.6  Bcast:192.168.0.255  Mask:255.255.255.0

          inet6 addr: a::b:c:d:1060/64 Scope:Link

          UP BROADCAST MULTICAST  MTU:1500  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

          Interrupt:48 Base address:0xc000 Memory:fe9a0000-fe9c0000

...

-----------

My conf settings

-----------

www root # cat /etc/conf.d/net | grep -v "#"

iface_eth0="192.168.0.4 broadcast 192.168.0.255 netmask 255.255.255.0"

iface_eth1="192.168.0.5 broadcast 192.168.0.255 netmask 255.255.255.0"

iface_eth2="192.168.0.6 broadcast 192.168.0.255 netmask 255.255.255.0"

gateway="eth0/192.168.0.1"

-----------

My init.d directory

-----------

www init.d # cd /etc/init.d

www init.d # ls -l | grep eth

-rwxr-xr-x    1 root     root         4208 Sep 12 16:02 net.eth0

lrwxrwxrwx    1 root     root            8 Dec  5 03:07 net.eth1 -> net.eth0

lrwxrwxrwx    1 root     root            8 Dec  5 04:51 net.eth2 -> net.eth0

-----------

My default startup directoy

-----------

www default # cd /etc/runlevels/default

www default # ls -l | grep eth

lrwxrwxrwx    1 root     root           20 Nov 30 10:06 net.eth0 -> /etc/init.d/net.eth0

lrwxrwxrwx    1 root     root           20 Dec  5 03:08 net.eth1 -> /etc/init.d/net.eth1

lrwxrwxrwx    1 root     root           20 Dec  5 04:51 net.eth2 -> /etc/init.d/net.eth2Last edited by albanard on Fri Dec 05, 2003 12:34 am; edited 1 time in total

----------

## FuzzeX

Looking at the your output from ifconfig I can say that all of our cards are being detected and initalized properly. The reason that only the first card is being used is a routing problem.

To answer your questions:

a) Ip's do not necissarily have to be specific to a device (you can alias ips so that one physical device will look like multiple ips), but in your case they are specific to each card.

b) Your two other network cards would appear to be working (you can ping them, they appear correcty configured in ifconfig), but they see no traffic because your default route probably is set to use eth0 and not eth1 or eth2.

Perhaps if you could give an explination of what kind of networking setup you're trying to acheive we can give you some ideas of how to accomplish it. I would guess you want to let traffic be shared across all three cards?

----------

## albanard

Hi FuzzeX,

Eventually what I want is 2 seperate public IP addresses that face the internet and a 3rd internal network address. The internal network does NOT need access to the internet and would be purely for communicating with other computers on the closed off internal network.

The computer should therefore NOT be acting as a router of any kind. I'm not sure if I need to set this somewhere explicitly in Gentoo.

At the moment of course I'm just testing with 3 local IPs on my LAN.

If all cards are connected to the internet(or LAN) I want to be able to physically unplug the cable from one of the cards and therefore disconnect the IP associated with that card from the internet (or LAN).

 *Quote:*   

> 
> 
> b) Your two other network cards would appear to be working (you can ping them ...
> 
> 

 

Well in a very round about sort of way maybe which may have something to do with the default route you were talking about. Just to be absolutely sure about the results of my ping tests.. I'm testing the computer with a single network cable. When I plug the cable into eth0 then I can ping all three IP addresses from another computer. However if I plug the cable into eth1 or eth2 I can't ping any of the 3 IPs. Thats why I thought the other two cards weren't working.

I don't know anything about default routes so any more help would be greatly appreciated.

----------

## FuzzeX

From the pinging behavior you describe it sounds like something is kind of weird.

Try this:

bring all of the interfaces down (/etc/init.d/net.eth* stop)

have the cable plugged into eth0 and manualy start it (/etc/init.d/net.eth0 start)

unplug the cable from eth0 and plug it into eth1 and start it (/etc/init.d/net.eth1 start)

see if you can then ping the eth1 address.

As far as the routing goes, your default setup should give you the behavior you want. You would have to setup iptables to get any kind of routing behavior.

As far as checking the routes try this command:

```
netstat -rn
```

Some good info on routing is here, you might not need everything they describe, but it could come in handy later (http://lartc.org/howto/lartc.rpdb.html).

----------

## albanard

 *Quote:*   

> 
> 
> Try this: 
> 
> bring all of the interfaces down (/etc/init.d/net.eth* stop) 
> ...

 

After this step can ping all ips  :Shocked: 

 *Quote:*   

> 
> 
> unplug the cable from eth0 and plug it into eth1 and start it (/etc/init.d/net.eth1 start)  
> 
> 

 

After this step can't ping any ips.

www root # netstat -rn

Kernel IP routing table

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface

192.168.0.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0

192.168.0.0     0.0.0.0         255.255.255.0   U         0 0          0 eth1

192.168.0.0     0.0.0.0         255.255.255.0   U         0 0          0 eth2

127.0.0.0       127.0.0.1       255.0.0.0       UG        0 0          0 lo

0.0.0.0         192.168.0.1     0.0.0.0         UG        0 0          0 eth0

I've read in another post that you can't have two ips in the same subnet. I'll keep researching but if anyone has any more input they'd like to share feel free ...

----------

## albanard

OK, I think I'm starting to understand this behaviour. Lets say I'm pinging from the computer out. If I ping 192.168.0.20 (another computer on the LAN)  when the cable is connected to eth0 then my routing table says "192.168.0.20 matches the first entry, therefore send this to eth0". I can therefore ping away to my hearts content.

Now if I plug the cable into eth1 instead the routing table will still say "192.168.0.20 matches the first entry, therefore send this to eth0" even though eth0 is down. It never gets to the next entry in the routing table because the first entry was matched. What I'd need is something that says "go to eth0 if its up, otherwise go to eth1".  Is this possible?

Similarly my theory is that when I ping the computer from 192.168.0.20 the ping message arrives, but the return packet never makes it back because it is redirected to eth0 when eth0 has in fact been unplugged and only eth1 is plugged in. So its essentially the same problem.

----------

## FuzzeX

That does sound like the routing problem I was thinking of.

As far as dynamic routing, there are ways to do dynamic routing like what you would want in this case. They describe a bit here: http://lartc.org/howto/lartc.dynamic-routing.html and there is a good project here: http://www.zebra.org/.

I'd be interested to hear how this works out for you.

----------

## creese

You're also running into a well known bug/feature of the linux kernel. The kernel will answer ARPs for any configured IP address on any interface with an IP assigned. IPs are not tied to interfaces, but pooled at a higher level. Discussions appear on this topic frequently on the kernel mailing list. This is one of the very few things I truely hate about linux.

----------

## madmango

I think also you were right in saying you can't have each card on the same subnet, mostly because you broadcast addresses are the same for all three card.

In my box, i've got one card on the 10.152.2.0 subnet, bcast 10.152.2.255, another one on the 10.152.3.0 subnet, bcast 10.152.3.255, and another that gets a dhcp address of 68.82.x.x, bcast 68.82.x.255.

----------

## fleed

I think it shouldn't be a problem having as many cards on the same subnet as you want. I've done it in the past and had no problems with it. Unfortunately, I don't have any setup like that right now so I can't test it to see if I get the same problem.

----------

## albanard

My forage into the two cards one subnet arena has now ended. Thanks for the help everyone and especially for the links FuzzeX.  Below is a summary of my findings / decision. I'm not a network guru so if you have any other tips / corrections feel free to make them. (Also I may refer to "computer" when technically I should be writing "OS" or "kernel" but I'm trying to simplify).

What I originally wanted was to have two public IP addresses for a computer, each "attached" to a different card. That way I would have two ways of reaching that computer. One (IP1)  which was mapped to a domain name, and another IP (IP2) with which I could access the machine even if I physically disconnected the cable from IP1.

Why would I want to do this you might ask. Well the plan was to have two computers set up in this way, a main computer and a backup. Both would have one of the cards mapped to a public IP(IP1), but only one of the computers would have this card "plugged in" at any one time.  That way if the main server goes down I could quickly switch over to the backup server by simply unplugging from the main and plugging into the card with IP1 on the backup.

The problem lies in not being able to update the  backup while the main server is up (and visa versa). This is why I wanted a second public IP  connected to the internet so that I could still access the backup server. Unfortunatly I haven't been able to find a way to do this seemingly simple task, at least not in a simple way.

I think I've narrowed the problem down to this:

I'm thinking in terms of accessing the computer via two "channels", but the computer's logic is simply how to best get data out, regardless of which channel communication began with.

I assumed that the computer could treat communication in and out of the different ethernet cards seperately. Then I realised that the computer doesn't specifically tag incomming and matching outgoing data on an application level. e.g. if someone requests a html page from IP1/eth1 then when Apache goes to answer that request, the operating system doesn't know that it's an answer to a request comming in on IP1/eth1. For all it knows the request may have come in on IP2/eth2. So the operating system just thinks in terms of "How do I best get this message out of the computer" which is when it references the routing table.

The alternative I thought was to have the routing table say "Send the message via eth1, but if eth1 is down (e.g. unplugged), send it through eth2". That way only one "channel" would be active at any one time. Unfortunately there doesn't seem to be a (easy/reliable) way to do this. The closest I've come, if you have intel cards/ports is intel's ANS software which has failover support. However there was a bit about needing ARP_filter active in the kernel and something about only being compatible with kernel >= 2.4.5 and well, overall seemed like too much effort (with questionable outcome) to go to without knowing the end result. I don't want to use something like Zebra for similar reasons, it just doesn't seem to be designed for what I want so I don't want to risk using it. (Seems more to be designed for best routes betrween different computers, not different network cards on the same computer).

What I will do now is set up a LAN between the two computers and access the backup computer via the main computer to update it. This has several drawbacks however, namely:

- The backup computer won't have access to the internet so emerging will be a pain the the butt. (I would have thought that the backup could access the internet via the main server but I'm not sure how to do this. I'll be looking into this though).

- Because I'll be accessing the backup via the main server (i.e. ssh-ing into the main then ssh-ing from the main to the backup) modes of access won't be as flexible.

But of course it has the big advantage of being able to set things up the way they were intended and not having to worry about peculiarities that may arise from playing with stuff that, lets face it   :Smile:  , I know very little about.

----------

## albanard

Setting:

/bin/echo "0" > /proc/sys/net/ipv4/conf/all/accept_source_route

seems to disable the strange ping behaviour. i.e I can now only ping the IP of the ethernet card that is physically connected to the network and not the IP of the disconnected card(s).

----------

## fleed

That's good information! Thx!!!

----------

