# Routing and ARP Issues

## karl001

Hallo!

We encounter atm strange problems with our gentoo linux system and need help.

We have a system with 3 nics:

2 Port internal mobo nic (tg3 module)

4 port intel pro 1000 (2 times)

so we have 10 different eth interfaces with different ip's on thats nics.

All interfaces are in the same network (10.72.0.0/21) with rising ip's:

eth0: 10.72.0.135  ...  eth9: 10.72.0.144

what we want is that all nics's are autonomical. Means every traffic issued by eth1 should go over eth1 (in/out).

To reach this we tried to implement iproute2 since we need to set a default gateway per nic.

Our problem atm is that eth0 asnwers on all arp requests for any of the server ip's.

So the arp table on a remote machine (same network) points to the mac of eth0 for any ip from 135 to 144.

I tip on a kernel option (we use 2.6.8.1 actually) who causes that, could anyone plz point me to the right direction to solve this issue?

Greetings & Thanks in advance,

Karl

----------

## adaptr

 *karl001 wrote:*   

> All interfaces are in the same network (10.72.0.0/21) with rising ip's:
> 
> eth0: 10.72.0.135  ...  eth9: 10.72.0.144

 

You probably want to bridge that.

Nothing else will produce a functioning network.

----------

## karl001

@adaptr

Mhh, ich don't see any reason to bridge. The communication outside the local net goes by router (gw), no way to go around that.

Could you explain a bit more what you mean?

Karl

----------

## adaptr

What I mean is that, according to well-published ethernet / IP behaviour, assigning IP addresses in the same subnet to two different NICs on the same host will inevitably result in routing problems.

That's just how TCP/IP is designed.

When you want to connect NICs on the same host together (i.e. move packets from one to the other) there are basically two mechanisms to accomplish this: routing from one subnet to another, or bridging the NICs together to form a bigger subnet.

Without a clearer picture of what it is you want to achieve - but assuming you know what you're doing when you say that all 10 NICs have to lie in the same subnet - I suggest bridging as the only sane solution.

The problem, of course, is with the have to lie in the same subnet - why do they have to ?

----------

## karl001

well, lets give your some more details why we do that:

eth0 in used by nagios, a network monitoring system

on the same machine we use also nessus to gain security reports about our whole network.

The basic problem is, that nessus causes nagios to report failures on several machines while it is running.

I guess this is cause by too many open sockets resulting in timeouts while nagios scans maschines. (e.g. ping probes)

Our idea was to seperate the nessus communication on a different interface.

The problem is, that our current design will for sure not work this way since we are in same subnet and i can only set one default gateway over one nic.

So the traffic will not be splittet truely, thats clear.

since our admin demands the machine to be with all interfaces in same subnet i tried to solve this with iproute2.

Now with this i am still sitting on that mac issue i explained above.

the current issue with MAC adresses is IMHO no issue of the ip  protocol itself but of the kernel, correct me if i am wrong.

Karl

----------

## jklmnop

karl001,

you want to tweak the arp settings.

/proc/sys/net/{ipv4,ipv6}/conf/{all,default,eth0,eth1,..}/arp_{announce,filter,ignore}

the documentation is with the kernel source:

/usr/src/linux/Documentation/networking/ip-sysctl.txt

you can make it so that an interface only answers an ARP if the address

is assigned to that interface.

adaptr,

it's quite common in the clustering world to have multiple interfaces on multiple

cards into the same network.  you just have to setup your routing the way you

want it instead of relying on default route.

----------

## adaptr

I'll gladly take your word for it.

Some mention by the OP of terms like "clustering", "nagios" and "yes I really really want to do this" might have curbed my righteousness.

Just a hint.

----------

## karl001

@jklmnop

thnx alot for your reply. This kernel settings you suggested seems to be the exact answer.

Poorly it seems that they have no effect. 

I did an echo "1" to default/arp_ignore, filter and announce, but my other machine on the same subnet resolves after a arp -d still to the same mac of eth0.

This is realy wierd, is something special in kernel needed to get this to work?

Karl

----------

## jklmnop

you might have to configure the settings before you bring up the interfaces.

you might also want to check http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.arp_problem.html#2.6_arp

and/or http://www.linuxvirtualserver.org/mailing.html

it seems that most people are putting these settings in /etc/sysctl.conf so they

are configured at boot (by /etc/init.d/bootmisc in gentoo)

```

net.ipv4.conf.all.arp_ignore = 1

net.ipv4.conf.all.arp_announce = 2

```

seems to be what most people are using...

i don't know much more.  the same day i found out about this i realized that i

didn't need the other interface that was giving me grief, so i just unconfigured

it.  i'm also really using a single physical interface with 802.1q vlans with

spoofed MAC addresses so i don't know if they would work for me anyway.

----------

## karl001

well, finaly it works, at least eth0 is no more responding to arp requests for the other ip's.

but i still have a problem now, since a communication to that other ip's is still impossible.

so it seems we go round 2 atm.

i tried several options without any luck, it seems that eth0 is now only responding to its own ip arp requests (correct) but eth1 and the rest do not respond to any other arp request to its ip's...

Mhhhh....

the real wierd is that with tcpdump i even don't see any arp request from the remote machine to the interfaces.

and again a huge question mark materializes...... :>

Any further hints?

Karl

----------

