# Routing problem

## petrjanda

There are 4 nics in the server, eth0 being connection to ISP, eth3 connection to 2nd ISP, but normally not used, eth1 being dmz and eth2 internal. The server runs gentoo x86 with hardened-sources. Normally all packets to external destinations are routed out thru eth0, but today for some weird reason the server stopped being able to routh through eth0(so currently eth3 is used a fall back). I double checked all packet filtering configuration, even disabled the firewall altogether but it doesnt work. The routing table looks ok, default route is via eth0/172.16.10.1, i am able to ping 172.16.10.1, the line seems to be normally working, even connected windows workstation to it and its working(can get internet), but not on the server. I checked ifconfig and theres no errors, overruns or dropped packets, i even restarted the server but still no luck afterwards. dmesg doesnt report anything unusual too. Could this be a kernel bug? This problem occured before, it fixed itself somehow. I cant get to the root of this problem. What do you guys suggest?

----------

## rojaro

Hi,

first provide us the output of the following commands:

```
ifconfig -a

route -N
```

That should make it easier for us to analyze your problem and to help you with it.

- rojaro -

----------

## petrjanda

```

root@a216server /boot # ifconfig -a                                     <14:14>

eth0      Link encap:Ethernet  HWaddr 00:0F:B5:44:94:80

          inet addr:172.16.10.2  Bcast:172.16.10.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:15 errors:0 dropped:0 overruns:0 frame:0

          TX packets:229 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:900 (900.0 b)  TX bytes:18591 (18.1 Kb)

          Interrupt:11 Base address:0xc000

eth1      Link encap:Ethernet  HWaddr 00:0F:B5:46:65:61

          inet addr:192.168.1.1  Bcast:192.168.1.3  Mask:255.255.255.252

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:2652 errors:0 dropped:0 overruns:0 frame:0

          TX packets:2550 errors:2 dropped:0 overruns:2 carrier:2

          collisions:0 txqueuelen:1000

          RX bytes:2023870 (1.9 Mb)  TX bytes:362321 (353.8 Kb)

          Interrupt:5 Base address:0xe000

eth2      Link encap:Ethernet  HWaddr 00:0F:B5:46:68:93

          inet addr:192.168.1.65  Bcast:192.168.1.127  Mask:255.255.255.192

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:585621 errors:0 dropped:0 overruns:0 frame:0

          TX packets:935708 errors:15 dropped:0 overruns:15 carrier:15

          collisions:0 txqueuelen:1000

          RX bytes:104003928 (99.1 Mb)  TX bytes:1046438538 (997.9 Mb)

          Interrupt:10

eth3      Link encap:Ethernet  HWaddr 00:0F:B5:46:65:04

          inet addr:10.81.102.16  Bcast:10.81.103.255  Mask:255.255.254.0

          UP BROADCAST NOTRAILERS RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:848737 errors:0 dropped:0 overruns:0 frame:0

          TX packets:599167 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:1005234226 (958.6 Mb)  TX bytes:48790338 (46.5 Mb)

          Interrupt:11 Base address:0x2000

lo        Link encap:Local Loopback

          inet addr:127.0.0.1  Mask:255.0.0.0

          UP LOOPBACK RUNNING  MTU:16436  Metric:1

          RX packets:27015 errors:0 dropped:0 overruns:0 frame:0

          TX packets:27015 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:3936124 (3.7 Mb)  TX bytes:3936124 (3.7 Mb)

tunl0     Link encap:IPIP Tunnel  HWaddr

          NOARP  MTU:1480  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

```

```

```

```

root@a216server /boot # ip route show                                   <14:28>

192.168.1.0/30 dev eth1  proto kernel  scope link  src 192.168.1.1

192.168.1.64/26 dev eth2  proto kernel  scope link  src 192.168.1.65

172.16.10.0/24 dev eth0  proto kernel  scope link  src 172.16.10.2

10.81.102.0/23 dev eth3  scope link

127.0.0.0/8 dev lo  scope link

```

Any clues? accorcding to tcpdump (tcpdump -i eth0) shows that traffic gets routed out of the interface.

----------

## petrjanda

tcpdump -i eth0

```

root@a216server /home/admin # tcpdump -i eth0                           <14:30>

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes

14:33:29.997145 00:10:7b:cd:6c:e2 > 00:10:7b:cd:6c:e2, ethertype Loopback (0x9000), length 60:

        0x0000:  0000 0100 0000 0000 0000 0000 0000 0000  ................

        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............

14:33:30.197830 IP a216server.a216network.edu.58733 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 863302875:863302875(0) win 5840 <mss 1460,sackOK,timestamp 81422473 0,nop,wscale 2>

14:33:39.997443 00:10:7b:cd:6c:e2 > 00:10:7b:cd:6c:e2, ethertype Loopback (0x9000), length 60:

        0x0000:  0000 0100 0000 0000 0000 0000 0000 0000  ................

        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............

14:33:42.196027 IP a216server.a216network.edu.58733 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 863302875:863302875(0) win 5840 <mss 1460,sackOK,timestamp 81434473 0,nop,wscale 2>

14:33:44.473467 IP a216server.a216network.edu.56472 > dandns1.chisholm.vic.edu.au.domain:  13168+ A? www.l.google.com. (34)

14:33:44.487977 IP a216server.a216network.edu.12539 > dandns1.chisholm.vic.edu.au.domain:  65250+ A? fxfeeds.mozilla.org. (37)

14:33:47.493319 IP a216server.a216network.edu.50370 > dandns1.chisholm.vic.edu.au.domain:  11675+ A? www.l.google.com. (34)

14:33:47.493350 IP a216server.a216network.edu.36107 > dandns1.chisholm.vic.edu.au.domain:  14825+ A? fxfeeds.mozilla.org. (37)

14:33:49.763555 IP a216server.a216network.edu.44633 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 900882454:900882454(0) win 5840 <mss 1460,sackOK,timestamp 81442041 0,nop,wscale 2>

14:33:49.997746 00:10:7b:cd:6c:e2 > 00:10:7b:cd:6c:e2, ethertype Loopback (0x9000), length 60:

        0x0000:  0000 0100 0000 0000 0000 0000 0000 0000  ................

        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............

14:33:50.545953 IP a216server.a216network.edu.44067 > dandns1.chisholm.vic.edu.au.domain:  22412+ A? www.l.google.com. (34)

14:33:50.546220 IP a216server.a216network.edu.1313 > dandns1.chisholm.vic.edu.au.domain:  54904+ A? fxfeeds.mozilla.org. (37)

14:33:51.445719 IP a216server.a216network.edu.49253 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 893475647:893475647(0) win 5840 <mss 1460,sackOK,timestamp 81443724 0,nop,wscale 2>

14:33:52.762404 IP a216server.a216network.edu.44633 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 900882454:900882454(0) win 5840 <mss 1460,sackOK,timestamp 81445041 0,nop,wscale 2>

14:33:53.565385 IP a216server.a216network.edu.27002 > dandns1.chisholm.vic.edu.au.domain:  44327+ A? www.l.google.com. (34)

14:33:53.565409 IP a216server.a216network.edu.22926 > dandns1.chisholm.vic.edu.au.domain:  33801+ A? fxfeeds.mozilla.org. (37)

14:33:54.445146 IP a216server.a216network.edu.49253 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 893475647:893475647(0) win 5840 <mss 1460,sackOK,timestamp 81446724 0,nop,wscale 2>

14:33:58.512649 IP a216server.a216network.edu.43733 > dandns1.chisholm.vic.edu.au.domain:  52782+ A? www.l.google.com. (34)

14:33:58.512680 IP a216server.a216network.edu.48903 > dandns1.chisholm.vic.edu.au.domain:  22692+ A? fxfeeds.mozilla.org. (37)

14:33:58.761495 IP a216server.a216network.edu.44633 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 900882454:900882454(0) win 5840 <mss 1460,sackOK,timestamp 81451041 0,nop,wscale 2>

14:33:59.444381 arp who-has 172.16.10.1 tell a216server.a216network.edu

14:33:59.444762 IP a216server.a216network.edu.46080 > dandns1.chisholm.vic.edu.au.domain:  58837+ PTR? 1.10.16.172.in-addr.arpa. (42)

14:33:59.445790 arp reply 172.16.10.1 is-at 00:10:7b:cd:6c:e2

14:33:59.998059 00:10:7b:cd:6c:e2 > 00:10:7b:cd:6c:e2, ethertype Loopback (0x9000), length 60:

        0x0000:  0000 0100 0000 0000 0000 0000 0000 0000  ................

        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............

14:34:00.444239 IP a216server.a216network.edu.49253 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 893475647:893475647(0) win 5840 <mss 1460,sackOK,timestamp 81452724 0,nop,wscale 2>

14:34:01.099311 IP a216server.a216network.edu.3291 > dandns1.chisholm.vic.edu.au.domain:  427+ A? www.l.google.com. (34)

14:34:01.099383 IP a216server.a216network.edu.43200 > dandns1.chisholm.vic.edu.au.domain:  35918+ A? fxfeeds.mozilla.org. (37)

14:34:02.465054 IP a216server.a216network.edu.51739 > dandns1.chisholm.vic.edu.au.domain:  36250+ PTR? 1.10.16.172.in-addr.arpa. (42)

14:34:04.119719 IP a216server.a216network.edu.54992 > dandns1.chisholm.vic.edu.au.domain:  12190+ A? www.l.google.com. (34)

14:34:04.119743 IP a216server.a216network.edu.28357 > dandns1.chisholm.vic.edu.au.domain:  25962+ A? fxfeeds.mozilla.org. (37)

14:34:04.444766 IP a216server.a216network.edu.29245 > dandns1.chisholm.vic.edu.au.domain:  28211+ PTR? 1.10.16.172.in-addr.arpa. (42)

14:34:04.585675 IP a216server.a216network.edu.62432 > dandns1.chisholm.vic.edu.au.domain:  28426+ A? www.l.google.com. (34)

14:34:04.585700 IP a216server.a216network.edu.7143 > dandns1.chisholm.vic.edu.au.domain:  47547+ A? fxfeeds.mozilla.org. (37)

14:34:07.465291 IP a216server.a216network.edu.19025 > dandns1.chisholm.vic.edu.au.domain:  63044+ PTR? 1.10.16.172.in-addr.arpa. (42)

14:34:09.998367 00:10:7b:cd:6c:e2 > 00:10:7b:cd:6c:e2, ethertype Loopback (0x9000), length 60:

        0x0000:  0000 0100 0000 0000 0000 0000 0000 0000  ................

        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............

14:34:10.759664 IP a216server.a216network.edu.44633 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 900882454:900882454(0) win 5840 <mss 1460,sackOK,timestamp 81463041 0,nop,wscale 2>

14:34:11.322926 IP a216server.a216network.edu.48066 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 922009834:922009834(0) win 5840 <mss 1460,sackOK,timestamp 81463604 0,nop,wscale 2>

14:34:13.484365 IP a216server.a216network.edu.36943 > dandns1.chisholm.vic.edu.au.domain:  42369+ PTR? 1.10.16.172.in-addr.arpa. (42)

14:34:14.322122 IP a216server.a216network.edu.48066 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 922009834:922009834(0) win 5840 <mss 1460,sackOK,timestamp 81466604 0,nop,wscale 2>

14:34:15.140108 IP a216server.a216network.edu.5638 > dandns1.chisholm.vic.edu.au.domain:  15118+ A? www.l.google.com. (34)

14:34:15.140134 IP a216server.a216network.edu.31564 > dandns1.chisholm.vic.edu.au.domain:  37613+ A? fxfeeds.mozilla.org. (37)

14:34:18.485613 IP a216server.a216network.edu.2806 > dandns1.chisholm.vic.edu.au.domain:  51330+ PTR? 1.10.16.172.in-addr.arpa. (42)

14:34:19.222478 IP a216server.a216network.edu.41756 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 936303680:936303680(0) win 5840 <mss 1460,sackOK,timestamp 81471505 0,nop,wscale 2>

14:34:20.026910 00:10:7b:cd:6c:e2 > 00:10:7b:cd:6c:e2, ethertype Loopback (0x9000), length 60:

        0x0000:  0000 0100 0000 0000 0000 0000 0000 0000  ................

        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............

14:34:20.321209 IP a216server.a216network.edu.48066 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 922009834:922009834(0) win 5840 <mss 1460,sackOK,timestamp 81472604 0,nop,wscale 2>

14:34:21.650281 IP a216server.a216network.edu.46463 > dandns1.chisholm.vic.edu.au.domain:  14505+ A? www.l.google.com. (34)

14:34:21.650355 IP a216server.a216network.edu.29100 > dandns1.chisholm.vic.edu.au.domain:  7450+ A? fxfeeds.mozilla.org. (37)

14:34:22.221921 IP a216server.a216network.edu.41756 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 936303680:936303680(0) win 5840 <mss 1460,sackOK,timestamp 81474505 0,nop,wscale 2>

14:34:24.669660 IP a216server.a216network.edu.5264 > dandns1.chisholm.vic.edu.au.domain:  51383+ A? www.l.google.com. (34)

14:34:24.669685 IP a216server.a216network.edu.28532 > dandns1.chisholm.vic.edu.au.domain:  54684+ A? fxfeeds.mozilla.org. (37)

14:34:26.915808 IP a216server.a216network.edu.33241 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 940525655:940525655(0) win 5840 <mss 1460,sackOK,timestamp 81479199 0,nop,wscale 2>

14:34:28.221021 IP a216server.a216network.edu.41756 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 936303680:936303680(0) win 5840 <mss 1460,sackOK,timestamp 81480505 0,nop,wscale 2>

14:34:29.914753 IP a216server.a216network.edu.33241 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 940525655:940525655(0) win 5840 <mss 1460,sackOK,timestamp 81482199 0,nop,wscale 2>

14:34:30.026998 00:10:7b:cd:6c:e2 > 00:10:7b:cd:6c:e2, ethertype Loopback (0x9000), length 60:

        0x0000:  0000 0100 0000 0000 0000 0000 0000 0000  ................

        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............

14:34:32.319409 IP a216server.a216network.edu.48066 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 922009834:922009834(0) win 5840 <mss 1460,sackOK,timestamp 81484604 0,nop,wscale 2>

14:34:35.687992 IP a216server.a216network.edu.23067 > dandns1.chisholm.vic.edu.au.domain:  7156+ A? www.l.google.com. (34)

14:34:35.688021 IP a216server.a216network.edu.25507 > dandns1.chisholm.vic.edu.au.domain:  18576+ A? fxfeeds.mozilla.org. (37)

14:34:35.913840 IP a216server.a216network.edu.33241 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 940525655:940525655(0) win 5840 <mss 1460,sackOK,timestamp 81488199 0,nop,wscale 2>

14:34:36.950954 IP a216server.a216network.edu.43238 > dandns1.chisholm.vic.edu.au.domain:  39283+ A? lists.gnu.org. (31)

14:34:39.970309 IP a216server.a216network.edu.49609 > dandns1.chisholm.vic.edu.au.domain:  20321+ A? lists.gnu.org. (31)

14:34:40.027309 00:10:7b:cd:6c:e2 > 00:10:7b:cd:6c:e2, ethertype Loopback (0x9000), length 60:

        0x0000:  0000 0100 0000 0000 0000 0000 0000 0000  ................

        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............

14:34:40.219185 IP a216server.a216network.edu.41756 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 936303680:936303680(0) win 5840 <mss 1460,sackOK,timestamp 81492505 0,nop,wscale 2>

14:34:40.687109 arp who-has 172.16.10.1 tell a216server.a216network.edu

14:34:40.688116 arp reply 172.16.10.1 is-at 00:10:7b:cd:6c:e2

14:34:43.533114 IP a216server.a216network.edu.21370 > dandns1.chisholm.vic.edu.au.domain:  8892+ A? www.l.google.com. (34)

14:34:43.533182 IP a216server.a216network.edu.37543 > dandns1.chisholm.vic.edu.au.domain:  58514+ A? fxfeeds.mozilla.org. (37)

14:34:44.533121 IP a216server.a216network.edu.13194 > dandns1.chisholm.vic.edu.au.domain:  48430+ A? lists.gnu.org. (31)

14:34:46.552368 IP a216server.a216network.edu.19619 > dandns1.chisholm.vic.edu.au.domain:  15214+ A? www.l.google.com. (34)

14:34:46.552400 IP a216server.a216network.edu.26023 > dandns1.chisholm.vic.edu.au.domain:  2374+ A? fxfeeds.mozilla.org. (37)

14:34:47.553178 IP a216server.a216network.edu.39737 > dandns1.chisholm.vic.edu.au.domain:  9487+ A? lists.gnu.org. (31)

14:34:47.912016 IP a216server.a216network.edu.33241 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 940525655:940525655(0) win 5840 <mss 1460,sackOK,timestamp 81500199 0,nop,wscale 2>

14:34:49.222909 IP a216server.a216network.edu.59291 > PROXY.CHISHOLM.VIC.EDU.AU.http-alt: S 950566263:950566263(0) win 5840 <mss 1460,sackOK,timestamp 81501510 0,nop,wscale 2>

14:34:50.027635 00:10:7b:cd:6c:e2 > 00:10:7b:cd:6c:e2, ethertype Loopback (0x9000), length 60:

        0x0000:  0000 0100 0000 0000 0000 0000 0000 0000  ................

        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................

        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............

73 packets captured

146 packets received by filter

0 packets dropped by kernel[/quote]

```

----------

