# Subnet router help needed, subnets cannot see each other

## ObsidianBlade

I need configuration assistance for routing between two NICs on a server.  Let me describe the situation to you, because I'm not sure the best way to approach the solution - bridged and one subnet, routed, etc....

cc

I'm building a Distributed Compiling farm for a pair of Gentoo servers I want  to bring on line for our home lan/IT learning environment.  It is a 'production' network in that the network itself is a Windows 2000 Adv Server based network running Active Directory.  The AD server also hosts commond network services such as DNS over AD (LDAP), DHCP, file and printer services.  

We are looking to migrate to Gentoo Linux based servers and as part of a production environment, I don't want to burden the two Gentoo servers with handling their own gcc compiling.  I've built a single box that has 5 computers mounted on backplanes.  They are all fresh installs of Gentoo 2007.0 LiveCD running gcc-4.1.2 and the latest versions of ccache and distcc and all are running exactly the same build environment (cflags, -march, USE flags).  

Here's the 'problem':

All five computers in the Hivemind (name of the cluster) are connected to a 5 port 10baseTX hub, actually an ISA card, installed in one of the computers (Hivemind-1).  Hivemind-1 also has two NICs, one connects to the 10mb hub and the other connects to the 12 port 100baseTX hub that serves the production network.  I also have a 5port 100baseTX switch that I can use if I have to, but it's a bit flakey.  

here's a bit of a diagram:

<----------production subnet-----><---------Router---------------><---distcc subnet----------------------->

.............................................................................................................  ____[eth0--Overmind]        

............................................................................................................. | ___[eth0--Hivemind-2]    

............................................................................................................. |/

[production network hub]----------[eth1--<Hivemind-1>--eth0]---[10mb hub]-

............................................................................................................. |\___[eth0--Hivemind-3]  

............................................................................................................. |____[eth0--Hivemind-4]   

The question is: What is the 'best' way to connect the distcc subnet to the production subnet?  Should I set Hivemind-1 up as a router or as a bridge?  Bear in mind that Hivemind-1 is one of the PC's that will be compiling for the distcc cluster.  

Currently Hivemind-1 can successfully ping both subnets but the rest of the computers on the distcc subnet cannot ping any computers on the production network.  I've read through the Home Router Guide, but it doesn't answer my questions.  I don't need ppoe, nat, DNS, or DHCP , but my configuration won't let the the two subnets communicate with each other, and the Home Router Guide is kind of vague on which service provides that functionality.

/etc/conf.d/net:

```
routes_eth1=( "default via 192.168.1.10") 

routes_eth0=( "192.168.10.0/24 via 192.168.10.211" ) # IPv4 subnet route

         

iface_eth0="192.168.10.211 broadcast 192.168.10.255 netmask 255.255.255.0" #distcc subnet

iface_eth1="192.168.1.211 broadcast 192.168.1.255 netmask 255.255.255.0" #production subnet
```

For the record, I KNOW that this is not right, but I've tried several configurations and I can't figure out what I'm doing wrong.

----------

## Stever

Do you have ip forwarding enabled on Hivemind-1?

```
# cat /proc/sys/net/ipv4/ip_forward

1
```

if not, make sure it is enabled in your kernel, and set in /etc/sysctl.conf

```
net.ipv4.ip_forward = 1
```

EDIT: Is there a reason you can't just put all the machines directly on the production network?  I'm betting the 10Mb/s network is going to take a lot of the punch out of the distcc performance.

----------

## Hu

Depending on the routing tables of the machines in production, you may experience problems if you do not use NAT.  If the production machines do not have a route entry that directs them to use Hivemind-1 as the gateway for traffic intended for Hivemind-[2345], then they will send all hive related traffic to their default gateway, which may not be able to deliver the traffic.  To avoid this, you can add the appropriate routing entry, or enable NAT in such a way that all traffic associated with the hive has a source address of Hivemind-1, which will forward the traffic to its proper destination.

----------

## ObsidianBlade

 *Stever wrote:*   

> Do you have ip forwarding enabled on Hivemind-1?
> 
> ```
> # cat /proc/sys/net/ipv4/ip_forward
> 
> ...

 

I do not have ip forwarding enabled - that was one of the commands I've seen over the last day, but I couldn't remember it or where I'd found it.  As for sticking them on the production network, the production network is on a 100mbit hub, not a switch, so the traffic could affect the network performance.  I'd also like the box to have as few cables as possible - all the computers are in a 19x17x16 inch box, and if I can attach it to the network with one cable instead of five, that would be great.  Of course, all of these "I'd like" and "I want" go out the bleeding window if they have to.  I do have a 100mbit switch that I can use instead of the 10mb hub, and I;ll test it both ways to get an idea of performance diference.

-->more in the morning

----------

## ObsidianBlade

 *Hu wrote:*   

> Depending on the routing tables of the machines in production, you may experience problems if you do not use NAT.  If the production machines do not have a route entry that directs them to use Hivemind-1 as the gateway for traffic intended for Hivemind-[2345], then they will send all hive related traffic to their default gateway, which may not be able to deliver the traffic.  To avoid this, you can add the appropriate routing entry, or enable NAT in such a way that all traffic associated with the hive has a source address of Hivemind-1, which will forward the traffic to its proper destination.

 

I haven't done this yet, I figured I'd address this after I have the distcc subnet connected out to the internet on the far side of the production network.  Second task tomorrow morning.

----------

## ObsidianBlade

I've done the following:

On Hivemind-1, added the following to /etc/sysctl.conf, as suggested by Stever:

```
net.ipv4.ip_forward = 1
```

and on WRT54G (broadband Internet router, between internet and production subnet), added static route to distcc subnet, as recommended by Hu:

```
destination network 192.168.10.0/24 via gateway 192.168.1.211 (eth1 on Hivemind)
```

I'll add static routes to the two Gentoo servers on the production subnet (192.168.1.0/24) once I get the distcc subnet (192.168.10.0/24) communicating all the way through to the internet.

I still cannot ping out of the distcc subnet:

From Overmind (192.168.10.210) I can successfully ping eth0 on Hivemind-1 (192.168.10.211) but not eth1 on Hivemind-1(192.168.1.211), so I'm still not routing through Hivemind-1.  Is there another service that I need to install, or is my /etc/conf.d/net borked? 

I compiled a new kernel last night and made sure to include the settings covered in the Home Router Guide.  I fairly sure that I'm loading that kernel because there were no kernel sources in /usr/src/linux and I had to emerge gentoo-sources, and after running:

```
genkernel --gensplash --install --gconfig --clean --save-config --mountboot all
```

the only kernel listed in /boot has the same version number as the kernel sources in /usr/src/linux and it is the kernel being loaded by Grub

----------

## Hu

Please run this sequence on Overmind and on Hivemind-1 and post the output: ip addr; echo ---; ip route; echo ---; ip link; echo ---; cat /proc/sys/net/ipv4/ip_forward; echo ---; cat /proc/sys/net/ipv4/conf/*/rp_filter; iptables-save -c.  This will list all the addresses, routes, interface information, and IP forwarding data that I can think of.  My guess at this point is that Hivemind-1 is refusing to respond to the pings due to either a routing error or a netfilter rule that is dropping traffic for coming at the "wrong" interface (comes in eth0 headed for eth1).

What happens when Overmind pings Hivemind-1:eth1?  Specifically, what error is returned, and is it being returned by Overmind (e.g. cannot figure out to route traffic to Hivemind-1) or by Hivemind-1 (because it does not like the request)?

----------

## ObsidianBlade

 *Hu wrote:*   

> Please run this sequence on Overmind and on Hivemind-1 and post the output: ip addr; echo ---; ip route; echo ---; ip link; echo ---; cat /proc/sys/net/ipv4/ip_forward; echo ---; cat /proc/sys/net/ipv4/conf/*/rp_filter; iptables-save -c.  This will list all the addresses, routes, interface information, and IP forwarding data that I can think of.  My guess at this point is that Hivemind-1 is refusing to respond to the pings due to either a routing error or a netfilter rule that is dropping traffic for coming at the "wrong" interface (comes in eth0 headed for eth1).
> 
> What happens when Overmind pings Hivemind-1:eth1?  Specifically, what error is returned, and is it being returned by Overmind (e.g. cannot figure out to route traffic to Hivemind-1) or by Hivemind-1 (because it does not like the request)?

 

Well, I get "command not found" for ip addr, ip route, and ip link.  does this help at all?

Here's the output:

```
Hivemind-1~ # ip addr         

bash: ip: command not found

Hivemind-1 ~ # ip route

bash: ip: command not found

Hivemind-1 ~ # ip link

bash: ip: command not found

Hivemind-1 ~ # cat /proc/sys/net/ipv4/ip_forward

1

Hivemind-1 ~ # cat /proc/sys/net/ipv4/conf/*/rp_filter 

1

1

1

1

0

Hivemind-1 ~ # iptables-save -c

bash: iptables-save: command not found

Hivemind-1 ~ # 

```

looks to me like I need to emerge iptables

----------

## xbmodder

emerge iproute2

emerge iptables

Why do you want a router in between the networks? Just for security?

----------

## ObsidianBlade

 *xbmodder wrote:*   

> emerge iproute2
> 
> emerge iptables
> 
> Why do you want a router in between the networks? Just for security?

 

no, actually.  It's mostly a matter of available hardware.  I cannot connect all machines to the production subnet, partly because of port availability, and partly because I don't want to saturate the production subnet with extraneus traffic - it's on a hub, so bandwidth is shared between all connected machines and every machine sees all the traffic. If I stick a distcc farm on the production network and run an emerge world, I suspect that network performance would drop through the floor.  I have a 10mb hub with 5 ports that I can use to connect all the distcc machines on one hub, but I would need 6 ports to chain it to the production hub.  The natural solution to the problem is to all a second NIC to one of the machines and route the traffic.  This keeps unnecessary traffic off the production network and and solves the issue of being short one port on the distcc hub.

From a first glance, it wouldn't appear that I'm saving any traffic at all because the distcc cluster is to support two production servers on the production network; however, every time I run a major update on the production servers, I should be running the same update on the distcc boxes (emerge world/system).  Having the distcc boxes on a seperate subnet will keep the compiling traffick of 5 computers off the production subnet, cutting the distcc traffic from seven to two.

It turns out that I had already installed iptables, but not iproute2.  

Heres the output you requested from Hivemind-1, the router machine:

```
Hivemind-1 ~ # ip addr

1: lo: <LOOPBACK,UP,10000> mtu 16436 qdisc noqueue 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: eth1: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000

    link/ether 00:a0:c9:ec:7e:cd brd ff:ff:ff:ff:ff:ff

    inet 192.168.1.211/24 brd 192.168.1.255 scope global eth1

    inet6 fe80::2a0:c9ff:feec:7ecd/64 scope link 

       valid_lft forever preferred_lft forever

3: eth0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000

    link/ether 00:a0:cc:e6:0f:56 brd ff:ff:ff:ff:ff:ff

    inet 192.168.10.211/24 brd 192.168.10.255 scope global eth0

    inet6 fe80::2a0:ccff:fee6:f56/64 scope link 

       valid_lft forever preferred_lft forever

Hivemind-1 ~ # ip route

192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.211 

192.168.10.0/24 dev eth0  proto kernel  scope link  src 192.168.10.211 

192.168.10.0/24 via 192.168.10.211 dev eth0  scope link  metric 1 

127.0.0.0/8 dev lo  scope link 

default via 192.168.1.10 dev eth1 

Hivemind-1 ~ # ip link

1: lo: <LOOPBACK,UP,10000> mtu 16436 qdisc noqueue 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2: eth1: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000

    link/ether 00:a0:c9:ec:7e:cd brd ff:ff:ff:ff:ff:ff

3: eth0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000

    link/ether 00:a0:cc:e6:0f:56 brd ff:ff:ff:ff:ff:ff

Hivemind-1 ~ # cat /proc/sys/net/ipv4/ip_forward

1

Hivemind-1 ~ # cat /proc/sys/net/ipv4/conf/*/rp_filter

1

1

1

1

0

Hivemind-1 ~ # iptables-save -c

Hivemind-1 ~ # 

```

iptables is a nice tool, reminds me a bit of programming a Cisco router - not in its format, but  in the information it gives.

----------

## ObsidianBlade

 *Hu wrote:*   

> Please run this sequence on Overmind and on Hivemind-1 and post the output: ip addr; echo ---; ip route; echo ---; ip link; echo ---; cat /proc/sys/net/ipv4/ip_forward; echo ---; cat /proc/sys/net/ipv4/conf/*/rp_filter; iptables-save -c.  This will list all the addresses, routes, interface information, and IP forwarding data that I can think of.  My guess at this point is that Hivemind-1 is refusing to respond to the pings due to either a routing error or a netfilter rule that is dropping traffic for coming at the "wrong" interface (comes in eth0 headed for eth1).
> 
> What happens when Overmind pings Hivemind-1:eth1?  Specifically, what error is returned, and is it being returned by Overmind (e.g. cannot figure out to route traffic to Hivemind-1) or by Hivemind-1 (because it does not like the request)?

 

Oh, and the error returned by Overmind when trying to ping the 'far side' of Hivemind-1 is:

```
Overmind ~ # ping 192.168.1.211

PING 192.168.1.211 (192.168.1.211) 56(84) bytes of data.

From 192.168.10.210 icmp_seq=1 Destination Host Unreachable

...

From 192.168.10.210 icmp_seq=15 Destination Host Unreachable

--- 192.168.1.211 ping statistics ---

18 packets transmitted, 0 recieved, +12 errors, 100% packet loss, time 17007 ms, pipe 3

Overmind ~ #
```

----------

## Stever

 *ObsidianBlade wrote:*   

> From a first glance, it wouldn't appear that I'm saving any traffic at all because the distcc cluster is to support two production servers on the production network; however, every time I run a major update on the production servers, I should be running the same update on the distcc boxes (emerge world/system).  Having the distcc boxes on a seperate subnet will keep the compiling traffick of 5 computers off the production subnet, cutting the distcc traffic from seven to two.

 

Simplest answer here might be to simply put an 8-port switch between the production network and your distcc machines.  They are cheap, will segment the local distcc traffic from the production network, and will simplify your life immensely by keeping you out of the router management business  :Smile: 

http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&N=2050400030+115010988+1150110238&name=8+x+RJ45

----------

## ObsidianBlade

So I have to ask a realy stupid question.  I see all these articles about how easy it is to configure linux for routing and for building quick-and-dirty internet routers and such, so why is it so hard to just route between two subnets?  Something like this has been incredibly easy to do in every other OS I've ever used. In Windows NT, 2000, and 2003 it takes about 5 minutes to set this stuff up.  We're trying to move on to linux from Windows to get away from the security issues and machevilian licensing agreements - not to mention costs, but I have to say that it takes less time to harden a Windows installation than it does to get a basic linux server off the ground.

That said, I REALLY like Gentoo and ease of package management and system maintenance.  There are so many single-package solutions out there to fill other needs, such as web serving, DNS, DHCP, FTP and mail servers, and so on.  Isn't there a single-package solution for non-gateway routing?  With all the innovation in the linux world, I would think that such a basic need had been filled long ago with a robust, non user-space, no nonsense solution.

----------

## Hu

Part of it is experience.  If you had all the tools installed and you were a Linux network guru, you could put it together in a few minutes.  Since this is your first time, you are incurring all the overhead of learning it as you go.  If you do not have a good document handy, you are further burdened by your reliance on other people to answer your questions.  That can quickly turn minutes into days, as you have found.  On that note, my apologies for the delay in answering.

You did not provide the output of the ip addr/route/link triplet from Overmind, so I will have to guess a bit.  The From address from the Overmind ping output is Overmind's own address.  That usually indicates that the kernel tried to send the packet directly, but failed.  Is Overmind configured to use Hivemind-1 as a default route?

The complete lack of output from iptables-save is a little worrisome.  I suspect that means that Hivemind-1 is missing some or all of the Netfilter code.  Netfilter is an optional component, so you may have left it out of the kernel.  If Hivemind-1 does not have Netfilter support, it cannot do NAT.  It can do straight routing, but that means that systems on production need a route entry that tells them to send responses for Overmind and friends to Hivemind-1, rather than the default gateway.  If you are fine with that, then do not worry further.  If you want NAT, we will need to modify your kernel configuration.

It is typical, but not required, to run a DHCP server on the NAT-facing card of the router (Hivemind-1) and let it assign both addresses and routing information to the internal machines (Hivemind-[2345]).  If you go with that approach and use dhcpd, you can configure it to assign pseudo-static IP addresses to the internal hosts.  That is, the machine with MAC address 00:00:00:00:00:01 will always be 192.168.10.2, MAC address ::02 will always be 192.168.10.3, etc.  This is useful because it gives the machines static addresses (which simplifies auditing and firewall rules) but keeps them configured via DHCP, so that all their configuration is managed by the dhcpd on Hivemind-1, rather than being scattered in the /etc/conf.d/net of the individual machines.

----------

## ObsidianBlade

 *Hu wrote:*   

> Part of it is experience.  If you had all the tools installed and you were a Linux network guru, you could put it together in a few minutes.  Since this is your first time, you are incurring all the overhead of learning it as you go.  If you do not have a good document handy, you are further burdened by your reliance on other people to answer your questions.  That can quickly turn minutes into days, as you have found.  On that note, my apologies for the delay in answering.
> 
> You did not provide the output of the ip addr/route/link triplet from Overmind, so I will have to guess a bit.  The From address from the Overmind ping output is Overmind's own address.  That usually indicates that the kernel tried to send the packet directly, but failed.  Is Overmind configured to use Hivemind-1 as a default route?
> 
> The complete lack of output from iptables-save is a little worrisome.  I suspect that means that Hivemind-1 is missing some or all of the Netfilter code.  Netfilter is an optional component, so you may have left it out of the kernel.  If Hivemind-1 does not have Netfilter support, it cannot do NAT.  It can do straight routing, but that means that systems on production need a route entry that tells them to send responses for Overmind and friends to Hivemind-1, rather than the default gateway.  If you are fine with that, then do not worry further.  If you want NAT, we will need to modify your kernel configuration.
> ...

 

Sorry for the omissions in the outputs and the reply time - my whole family has been sick this past week.  I sat back and gave the whole thing a good evaluation and took Stever's advice.  I reconfigured the LAN and moved some hardware around.  It's all on one subnet now, and unrouted.  I took the 100mb hub and bolted it to the Hivemind's case ( a roughly 17" cube of plywood) and moved the 5 port 100mb switch to be the main network switch - basically swapped their positions.  I trimmed a few cables out of the production subnet and eliminated a couple of dead drops, and everything is connected as it should be.  The cluster's hub is connected to the switch, effectively killing the collision domain issue and the routing problem.

I apologize for the tirade, everyone - especially to you, stevers.  It was uncalled for.  You have ALL been a tremendous help - thank you.

Hopefully, I'll be posting some pictures of this thing later tonight over in Portage & Programming

----------

## xbmodder

Its not difficult. It should seriously be about 30-50 commands and a few hours are the terminal/hardware room.

----------

