# [FIXED] How to route the router's traffic

## kuteninja

Hard to get title... well this is the case:

I have a linux router (gentoo) with 2 gateways WAN1 and WAN2. 

The same machine have a LAN where all my machines are connected with UPNP, DHCP and such.

As a router it works perfectly, I'm using this method:

LAN user wants to go to 1.2.3.4

iptables mangle marks everything going to 1.2.3.4 as mark1

ip route knows it should go thru WAN1 (using table1)

all further packets to 1.2.3.4 go thru WAN1

LAN user wants to go to 33.33.33.33

iptables mangle marks everything going to 33.33.33.33 as mark2

ip route knows it should go thru WAN2 (using table2)

all further packets to 33.33.33.33 go thru WAN2

But... if i visit 33.33.33.33 from the router itself (ping, wget, etc), iptables doesn't do anything and it uses the default gateway which is WAN1 instead of going out thru WAN2 like it should have. 

How can I force localhost to decide where to go using iptables mangle instead of using the default gateway directly?

----------

## papahuhn

You need to mangle in OUTPUT for that.

----------

## kuteninja

 *papahuhn wrote:*   

> You need to mangle in OUTPUT for that.

 

Oh, so the local traffic from the router goes directly thru mangle->output and the LAN traffic goes thru mangle->prerouting or I should use only output?

----------

## papahuhn

Use both.

----------

## kuteninja

 *papahuhn wrote:*   

> Use both.

 

Nope, still not working.

I've setup the same connmark rules on output and I'm still going out thru the default gateway instead of the iptables rule.

----------

## papahuhn

Do you only use CONNMARK without packet MARK? The latter is actually relevant for policy routing.

----------

## kuteninja

Let's make it easier, here's my iptables mangle setup:

CONNMARK1 = WAN1

CONNMARK2 = WAN2

```
-A PREROUTING -m state --state RELATED,ESTABLISHED -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff

-A PREROUTING -d 192.168.0.0/16 -j RETURN

-A PREROUTING -d 10.0.0.0/24 -j RETURN

-A PREROUTING -d 190.2.0.0/24 -m state --state NEW -j CONNMARK1

-A PREROUTING -d 66.220.158.0/24 -m state --state NEW -j CONNMARK2

-A PREROUTING -m mark ! --mark 0x1 -m state --state NEW -j CONNMARK2

-A PREROUTING -m mark --mark 0x0 -m state --state NEW -j CONNMARK2

-A OUTPUT -m state --state RELATED,ESTABLISHED -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff

-A OUTPUT -d 192.168.0.0/16 -j RETURN

-A OUTPUT -d 10.0.0.0/24 -j RETURN

-A OUTPUT -d 190.2.0.0/24 -m state --state NEW -j CONNMARK1

-A OUTPUT -d 66.220.158.0/24 -m state --state NEW -j CONNMARK2

-A OUTPUT -m mark ! --mark 0x1 -m state --state NEW -j CONNMARK2

-A OUTPUT -m mark --mark 0x0 -m state --state NEW -j CONNMARK2

-A CONNMARK1 -j MARK --set-xmark 0x1/0xffffffff

-A CONNMARK1 -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff

-A CONNMARK2 -j MARK --set-xmark 0x2/0xffffffff

-A CONNMARK2 -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
```

Let's say I visit 190.2.0.1 from my computer on the LAN, I go thru WAN1, then if I visit 66.220.158.1 I go thru WAN2.

I do the same on the router and I allways go thru WAN1 since it's the default gateway on route table main when I should use WAN1 or WAN2 depending on what iptables says.

----------

## kuteninja

Sad bump  :Sad: 

I couldn't find what's wrong.

I tried to remove the default gateway on the main table; the LAN kept having internet access but my router can't ping or browse by itself.

----------

## papahuhn

Try some simplified netfilter rules temporarily; one MARK for WAN1/2 each. It might be also important to do some SNAT, otherwise some packets will have the wrong source IP (i.e. the IP of the other interface). You should make some tcpdumps with your tests.

----------

## kuteninja

 *papahuhn wrote:*   

> Try some simplified netfilter rules temporarily; one MARK for WAN1/2 each. It might be also important to do some SNAT, otherwise some packets will have the wrong source IP (i.e. the IP of the other interface). You should make some tcpdumps with your tests.

 

I have SNAT setup, forgot to mention it, here it is:

```
/sbin/iptables -t nat -A POSTROUTING -s 192.168.0.0/16 -m mark --mark 0 -j SNAT --to-source $WAN2

/sbin/iptables -t nat -A POSTROUTING -s 192.168.0.0/16 -m mark --mark 1 -j SNAT --to-source $WAN1

/sbin/iptables -t nat -A POSTROUTING -s 192.168.0.0/16 -m mark --mark 2 -j SNAT --to-source $WAN2
```

Do you recommend to add more lines for -s 127.0.0.1, or just ! -d 192.168.0.0 to include also localhost?

----------

## b0nafide

On the router, try

```
route add 190.2.0.0/24 gw 190.2.0.foo 

route add 66.220.158.0/24 gw 66.220.158.bar

```

I don't see what you're trying to achieve with all the connmark'ing, but perhaps I'm missing something.

----------

## kuteninja

 *b0nafide wrote:*   

> On the router, try
> 
> ```
> route add 190.2.0.0/24 gw 190.2.0.foo 
> 
> ...

 

I think that the gw there should be my WAN1 or WAN2 addressess, but it would be having to do everything twice -.-

I have 2 ISP connections, but since they don't have the same WAN IP i can't bond them together so I'm marking some networks to go thru one or the other ISP, it works fine for LAN users, except for the router itself, which allways uses the default route instead of the iptables marked one.

Although with the last reply about the SNAT I think he nailed it, since it's only "SNATting" the packets comming from 192.168.0.0 and excluding the ones from localhost, perhaps the packets are properly marked but the source is not being changed...

```
/sbin/iptables -t nat -A POSTROUTING -s 127.0.0.1 -d 0.0.0.0 -m mark --mark 2 -j SNAT --to-source $WAN2 

/sbin/iptables -t nat -A POSTROUTING ! -s 192.168.0.0/16 -d 0.0.0.0 -m mark --mark 0 -j SNAT --to-source $WAN2 

/sbin/iptables -t nat -A POSTROUTING ! -s 0.0.0.0 -m mark --mark 0 -j SNAT --to-source $WAN2 
```

Which one of these would you recommend me to try?

----------

## kuteninja

Here's the tcpdump for visiting icanhazip.com from my router...

```
20:10:33.970170 IP WAN1.10320 > 198.61.150.28.80: SWE 3879243526:3879243526(0) win 5840 <mss 1460>

20:10:34.179445 IP 198.61.150.28.80 > WAN1.10320: SE 147161378:147161378(0) ack 3879243527 win 14600 <mss 1460>

20:10:34.179461 IP WAN1.10320 > 198.61.150.28.80: . ack 1 win 5840

20:10:34.179537 IP WAN1.10320 > 198.61.150.28.80: P 1:602(601) ack 1 win 5840

20:10:34.368430 IP 198.61.150.28.80 > WAN1.10320: P 1:327(326) ack 602 win 15626

20:10:34.368447 IP WAN1.10320 > 198.61.150.28.80: . ack 327 win 6432

20:10:34.397445 IP 198.61.150.28.80 > WAN1.10320: . ack 602 win 15626

20:10:34.397455 IP WAN1.10320 > 198.61.150.28.80: . ack 327 win 6432

20:10:36.015355 IP WAN1.10320 > 198.61.150.28.80: F 602:602(0) ack 327 win 6432

20:10:36.230296 IP 198.61.150.28.80 > WAN1.10320: F 327:327(0) ack 603 win 15626

20:10:36.230311 IP WAN1.10320 > 198.61.150.28.80: . ack 328 win 6432
```

And here's the one from visiting it thru my LAN computer:

```
20:11:21.253430 IP WAN2.53645 > 198.61.150.28.80: S 4136494562:4136494562(0) win 8192 <mss 1460,nop,nop,sackOK>

20:11:21.253540 IP WAN2.53646 > 198.61.150.28.80: S 3314477280:3314477280(0) win 8192 <mss 1460,nop,nop,sackOK>

20:11:21.306034 IP WAN2.53647 > 198.61.150.28.80: S 1287074544:1287074544(0) win 8192 <mss 1460,nop,nop,sackOK>

20:11:21.428756 IP 198.61.150.28.80 > WAN2.53645: S 1151037725:1151037725(0) ack 4136494563 win 14600 <mss 1460,nop,nop,sackOK>

20:11:21.430475 IP WAN2.53645 > 198.61.150.28.80: . ack 1 win 17520

20:11:21.431106 IP 198.61.150.28.80 > WAN2.53646: S 3590466767:3590466767(0) ack 3314477281 win 14600 <mss 1460,nop,nop,sackOK>

20:11:21.432504 IP WAN2.53645 > 198.61.150.28.80: P 1:383(382) ack 1 win 17520

20:11:21.438041 IP WAN2.53646 > 198.61.150.28.80: . ack 1 win 17520

20:11:21.484339 IP 198.61.150.28.80 > WAN2.53647: S 1385340538:1385340538(0) ack 1287074545 win 14600 <mss 1460,nop,nop,sackOK>

20:11:21.486884 IP WAN2.53647 > 198.61.150.28.80: . ack 1 win 17520

20:11:21.612309 IP 198.61.150.28.80 > WAN2.53645: . ack 383 win 15544

20:11:21.613295 IP 198.61.150.28.80 > WAN2.53645: P 1:330(329) ack 383 win 15544

20:11:21.809934 IP WAN2.53645 > 198.61.150.28.80: P 383:716(333) ack 330 win 17191

20:11:21.988099 IP 198.61.150.28.80 > WAN2.53645: P 330:661(331) ack 716 win 16616

20:11:22.189800 IP WAN2.53645 > 198.61.150.28.80: . ack 661 win 16860
```

As you can see one of these has WAN1 (the first one) and the second has WAN2 (my LAN), the first one is wrong, it should go out thru the IP of WAN2.

PS: I've changed the gateway IPs manually to WAN1 and WAN2, tcpdump shows the actual IP, but I don't want to show them publicly.

----------

## b0nafide

 *kuteninja wrote:*   

> I think that the gw there should be my WAN1 or WAN2 addressess

 

Hmmm whatever your gateways are could be determined with something like 

```
grep -i "gateway=" `find /var/lib/ -name "dhcpcd-eth*"`
```

or, something better like

```
route add 190.2.0.0/24 gw `sed "s/"\'"/ /g" /var/lib/dhcpcd-eth0.info | awk '/^GATEWAY=/ { print $2 }'`
```

If you knew that particular gateway was always going to be eth0Last edited by b0nafide on Wed Oct 10, 2012 11:44 pm; edited 2 times in total

----------

## kuteninja

I have static dedicated IPs on both of the ISPs, you're not getting the case, I'd like to avoid routing without passing thru iptables mangle -> nat first (from localhost to wan)

On your example you made me edit the route, I don't want to add a route for each one that's on my mangle iptables, I want localhost to go thru the mangle table.

----------

## b0nafide

Ok, I understand now. My setup does what I need it to at the moment without mangling packets.

----------

## kuteninja

I got it fixed, as I noticed the packets were being marked as mark2, but the SNAT wasn't catching them, so I did this:

- If the packet is trying to go out "from" WAN1 (using the default gateway) but with a mark2, then it should be SNAT to WAN2

- If the packet is trying to go out "from" WAN2 (if the default gateway gets changed) but with a mark1, then it should be SNAT to WAN1

That did the trick.

----------

