# Multi-homed Server - Routing Problems - Solved

## Utoxin

I'm trying to set up a server that is multihomed. It has (currently) two NICs, each of which is used by a different IP block. The first IP block works great. The second block can ping it's gateway, but pinging from outside to the IPs on the second NIC doesn't work.

```

chlorine ~ # ip route

209.90.90.32/27 dev eth1  proto kernel  scope link  src 209.90.90.34

209.90.77.64/26 dev eth0  proto kernel  scope link  src 209.90.77.96

127.0.0.0/8 dev lo  scope link

default via 209.90.77.65 dev eth0

default via 209.90.90.33 dev eth1  metric 10

chlorine ~ # ip link

1: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000

    link/ether 00:00:1a:19:2f:76 brd ff:ff:ff:ff:ff:ff

2: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000

    link/ether 00:00:1a:19:2f:77 brd ff:ff:ff:ff:ff:ff

3: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

chlorine ~ # ip addr

1: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000

    link/ether 00:00:1a:19:2f:76 brd ff:ff:ff:ff:ff:ff

    inet 209.90.77.96/26 brd 209.90.77.127 scope global eth0

       valid_lft forever preferred_lft forever

2: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000

    link/ether 00:00:1a:19:2f:77 brd ff:ff:ff:ff:ff:ff

    inet 209.90.90.34/27 brd 209.90.90.63 scope global eth1

       valid_lft forever preferred_lft forever

3: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

```

I've tried every combination on the default routes I can think of, but none of them allow eth1 to work, and some of them break both interfaces. These don't need to be load balanced, but they need to serve seperate IP blocks.

----------

## jmbsvicetto

Hi.

Did you remember to activate ip forwarding with the following?

```
echo "1" > /proc/sys/net/ipv4/ip_forward
```

----------

## Utoxin

Just tried that. No change.

----------

## jmbsvicetto

OK. If I understand your setup correctly, everything works on the 209.90.77.96/26 network block, right? The 209.90.90.34/27 block is capable of reaching the gateway, what about the outside?, but isn't reachable from the first block, right?

If so, what is the default gateway of your first network block? Is it your router or your router default gateway? If the latter, did you remember to add a rule to that gateway specifying that the second block is available through your router?

You can debug this with traceroute. Try to reach an address of the second block from a system on the first block. Is your router used?

----------

## Utoxin

Some more details.

The server is sitting at a colocation facility. The colo company has assigned us the two blocks you see here. (As well as others, but I'm not adding those into the mix until I have two working).

The blocks are each distinct public IP blocks, and aren't related except for pointing at the same server.

Each block has it's own gateway router.

eth0 (209.90.77.*) is working correctly, and has been for some time. The gateway is 209.90.77.65.

eth1 (209.90.90.*) is the new interface we are trying to set up. The gateways is 209.90.90.33.

I just tried pinging, using the -I parameter to bind to eth1, and I can ping the gateway for that interface, but can't get beyond that gateway. Is this an issue I may need to take up with the colo company?

----------

## jmbsvicetto

Try using traceroute to help debugging this You should look at traceroute -i and or traceroute -s output.

Have you tried pinging the second ip from the outside?

----------

## Utoxin

Tried pinging from the outside, and it just times out.

As for the traceroute suggestions, I can't get anything to work there. Not sure if I'm doing it right or not though.

```
chlorine ~ # traceroute -i eth1 166.70.166.210

traceroute to kydance.net (166.70.166.210), 30 hops max, 46 byte packets

chlorine ~ # man traceroute

chlorine ~ # traceroute -i eth1 -r 166.70.166.210

traceroute to kydance.net (166.70.166.210), 30 hops max, 46 byte packets

traceroute: sendto: Network is unreachable

 1 traceroute: wrote 166.70.166.210 46 chars, ret=-1

 *traceroute: sendto: Network is unreachable

traceroute: wrote 166.70.166.210 46 chars, ret=-1

chlorine ~ # traceroute -i eth1 -r 209.90.90.33

traceroute to ip-209-90-90-33.forgeglobal.com (209.90.90.33), 30 hops max, 46 byte packets

 1  *

```

----------

## jmbsvicetto

 *Utoxin wrote:*   

> Tried pinging from the outside, and it just times out.

 

Try using traceroute. Where does it end?

 *Utoxin wrote:*   

> 
> 
> As for the traceroute suggestions, I can't get anything to work there. Not sure if I'm doing it right or not though.
> 
> ```
> ...

 

Try using the following

```
# traceroute -i eth1 166.70.166.210

# traceroute -s 209.90.90.* 166.70.166.210

# traceroute -i eth1  140.211.166.170

# traceroute -s 209.90.90.* 140.211.166.170
```

----------

## Utoxin

From the server:

```

chlorine ~ # traceroute -i eth1 166.70.166.210

traceroute to kydance.net (166.70.166.210), 30 hops max, 46 byte packets

 1  * * *

chlorine ~ # traceroute -s 209.90.90.34 166.70.166.210

traceroute to kydance.net (166.70.166.210) from 209.90.90.34, 30 hops max, 46 byte packets

 1  * * *

 2  *

chlorine ~ # traceroute -i eth1 140.211.166.170

traceroute to dove.gentoo.osuosl.org (140.211.166.170), 30 hops max, 46 byte packets

 1  * * *

chlorine ~ # traceroute -s 209.90.90.34 140.211.166.170

traceroute to dove.gentoo.osuosl.org (140.211.166.170) from 209.90.90.34, 30 hops max, 46 byte packets

 1  * * *

```

From external system:

```

localhost ~ # traceroute 209.90.90.34

traceroute to ip-209-90-90-34.forgeglobal.com (209.90.90.34), 30 hops max, 38 byte packets

 1  192.168.1.1 (192.168.1.1)  1.485 ms  1.137 ms  1.098 ms

 2  router.kydance.net (166.70.166.209)  2.577 ms  2.142 ms  2.080 ms

 3  atm1-nrp4.slc.xmission.net (166.70.4.82)  42.326 ms  42.860 ms  42.066 ms

 4  vl-75.asr1.slc.xmission.net (166.70.4.76)  44.102 ms  43.093 ms  189.545 ms

 5  ge-1-3-0.cr1.slc.xmission.net (166.70.5.129)  42.244 ms  42.675 ms  42.080 ms

 6  gi-1-0.gbr3.slc.xmission.net (166.70.5.100)  42.749 ms  42.652 ms  42.325 ms

 7  12.127.106.97 (12.127.106.97)  42.718 ms  42.273 ms  43.811 ms

 8  12.127.106.106 (12.127.106.106)  46.541 ms  44.223 ms  44.250 ms

 9  0-0-0.bdr2.fbp.ore.fiber.net (216.83.133.31)  44.693 ms  43.374 ms  44.341 ms

10  * * *

localhost ~ # traceroute 209.90.77.96

traceroute to forgeglobal.com (209.90.77.96), 30 hops max, 38 byte packets

 1  192.168.1.1 (192.168.1.1)  1.428 ms  1.136 ms  1.095 ms

 2  router.kydance.net (166.70.166.209)  2.247 ms  2.172 ms  2.258 ms

 3  atm1-nrp4.slc.xmission.net (166.70.4.82)  88.554 ms  114.860 ms  97.092 ms

 4  vl-75.asr1.slc.xmission.net (166.70.4.76)  77.431 ms  97.359 ms  116.802 ms

 5  ge-1-3-0.cr2.slc.xmission.net (166.70.5.193)  77.423 ms  97.062 ms  97.767 ms

 6  gi-2-0.gbr3.slc.xmission.net (166.70.5.116)  116.489 ms  58.521 ms  96.279 ms

 7  12.127.106.97 (12.127.106.97)  98.273 ms  97.199 ms  77.577 ms

 8  12.127.106.106 (12.127.106.106)  202.292 ms  157.035 ms  198.960 ms

 9  0-0-0.bdr2.fbp.ore.fiber.net (216.83.133.31)  45.005 ms  44.569 ms  43.824 ms

10  forgeglobal.com (209.90.77.96)  46.573 ms  45.607 ms  43.506 ms

```

----------

## Utoxin

I just checked with the hosting company, and they say everything is set up and operating properly on their end. More help with troubleshooting would be greatly appreciated.

As some more info, here's the relevant section of /etc/conf.d/net

```
config_eth0=(

   "209.90.77.{96..126}/26"

)

routes_eth0=(

   "default via 209.90.77.65"

)

config_eth1=(

   "209.90.90.{34..62}/27"

)

routes_eth1=(

   "default via 209.90.90.33 metric 10"

)

```

I tried providing no metric on the eth1 route, but as I recall that broke routing for the whole box. Is there a better way to set the route for each interface?

----------

## sschlueter

 *Utoxin wrote:*   

> Tried pinging from the outside, and it just times out.
> 
> 

 

The response packets are probably leaving the wrong interface and therefore never reach you. You can use tcpdump to verify this.

This is a common problem when using multiple default gateways.

See http://lartc.org/howto/lartc.rpdb.multiple-links.html for a description how this problem can be solved.

----------

## Utoxin

Trying to figure out how to set this up. How do I completely override the routes Gentoo adds? It's causing conflicts with the routes that the document you pointed me to wants to add.

----------

## jmbsvicetto

You need to look at /etc/conf.d/net. The default network config is defined there.

If you're looking at lartc, you should emerge iproute2. You can create the interfaces and routes by hand using the ip command. When you are able to tweak the config to your liking, you can update the /etc/conf.d/net file. If you want to use iproute2 to configure your interfaces, you should use a config based on the following:

```

# cat /etc/conf.d/net

modules=( "iproute2" )

config_eth0=(

        "AAA.BBB.CCC.DDD/XXX"

)

routes_eth0=(

        "..."

)

config_eth1=(

        "AAA.BBB.CCC.DDD/XXX"

)

routes_eth1=(

        "..."

)

```

----------

## Utoxin

Already using iproute2. So would I just put the routes they reccomend in each of the route blocks for the interfaces, and then put any leftover routes in local.start?

----------

## jmbsvicetto

That should work. However, if you want to try a different method, buried in /etc/conf.d/net.example there's a few tidbits about advanced configurations. I would suggest using something like:

```
modules=( "iproute2" )

config_eth0=( ... }

routes_eth0=(

   $P1_NET dev $IF1 src $IP1 table T1

   default via $P1 table T1

)

config_eth1=( ... )

routes_eth1=(

   $P2_NET dev $IF2 src $IP2 table T2

   default via $P2 table T2

)

routes={

   default scope global nexthop via $P1 dev $IF1 weight 1 \

   nexthop via $P2 dev $IF2 weight 1

}

rules_eth0=(

   from $IP1 table T1

)

rules_eth1=(

   $IP2 table T2

)
```

You might also need to enable the postup() function on the example file. I'm not sure if you can use the routes statement above, but if you can't, try adding the default route to eth0.

----------

## lbrtuk

With your setup as it originally was, it was always going to send packets going to 0.0.0.0 (anywhere in the internet) out of eth0 (even if it came from a daemon that was listening on 209.90.90.34) because it is the first matching entry in the routing table. The routing table is read downwards until a match is found.

What you need is a routing rule saying that packets coming from 209.90.77.96 should exit eth0 and packets coming from 209.90.90.34 should exit eth1. The full process is documented in the document sschlueter pointed you to.

----------

## Utoxin

I just got back from trying the setup reccomended in that document, and it errors out every time I try to add the default route to T2. I emptied the whole routing table before I started, and every time I try to add the default line in T2 it errors with a 'file exists' error.

----------

## Utoxin

*bump*

Still trying to solve this. That LARTC document doesn't help at all, because the commands it recommends don't work.

----------

## sschlueter

Mh, I don't know, maybe your kernel lacks the required features? I guess your kernel needs CONFIG_IP_MULTIPLE_TABLES.

----------

## Utoxin

Ahha. That very well could be it. It wasn't enabled, though it took me forever to find it in menuconfig. I'll be going over later tonight to reboot into the new kernel, and see if I can get the routing working now.

----------

## Utoxin

Well, I just got back from installing the new kernel with the option enabled. After some fiddling, I got it working properly. Then I had to reboot to check on some other hardware. I made NO changes to the config between reboots. But after the reboot, I fought with it for over a half hour, trying everything I had tried before, and couldn't get it working again.

Here's my setup now:

```

chlorine # ip route show table T1

209.90.77.64/26 dev eth0  scope link  src 209.90.77.96

default via 209.90.77.65 dev eth0

chlorine # ip route show table T2

209.90.90.32/27 dev eth1  scope link  src 209.90.90.34

default via 209.90.90.33 dev eth1

chlorine # ip route show

209.90.90.32/27 dev eth1  proto kernel  scope link  src 209.90.90.34

209.90.77.64/26 dev eth0  proto kernel  scope link  src 209.90.77.96

127.0.0.0/8 dev lo  scope link

default via 209.90.77.65 dev eth0

chlorine # ip rule show

0:      from all lookup local

32761:  from 209.90.77.64/26 lookup T1

32762:  from 209.90.90.32/27 lookup T2

32766:  from all lookup main

32767:  from all lookup default

```

I'm completely baffled why it isn't working now. I do plan on calling the tech support for the hosting facility, and doing some network diagnostics with them. But I figure maybe someone here can give some pointers.

----------

## Utoxin

Nevermind! It's working now. Looks like a routing or arp table problem that sorted itself out. So, marking this solved. Thanks for all the help!

----------

