# FWMARK directed routing for localhost originated traffic

## dbishop

This seems to be a difficult problem.  I have a single host with three NICs, each connected to a different ISP . This host is *not* routing/forwarding for any other hosts.

I am trying to get all traffic for certain ports (22, 25, and 123) to route via eth0.  All other traffic should go via eth1 unless I set something to change to eth0 or eth2  (like changing an iptables rule or a routing table or a routing rule, etc.)

eth0's IP is 172.16.1.100 and its default gateway is 172.16.1.129 and its network is 172.16.1.128/25  (eth1 is 192.168.0.2, 192.168.0.0/24, 192.168.0.1 ; eth2 is 192.168.100.2, 192.168.100.0/24, 192.168.100.1)

I have tried all sorts of combinations of fwmark, ip route tables/rules, and snat'ing. All traffic stubbornly obeys the main table's default gateway (surprise, surprise...)

Is there a way to do this?  It seems that fwmark won't work with localhost because the routing decision is made before marking.  I've been thinking about the 'route' target in iptables, but that doesn't seem like it will work either.

I have searched this forum (and countless others) and none address this issue (or worked it to a successful resolution.  the one workaround I found won't work for me.

Anyone?

----------

## Bones McCracker

You should be marking in the "prerouting" chain of the "mangle" table, and then using SNAT in the "postrouting" chain of the "nat" table.

Edit: correction, per below: you should using the "output" chain of the "mangle" table.

I don't know what sources you have consulted, but this is a good one:

http://linux-ip.net/html/adv-multi-internet.html

----------

## truc

 *Quote:*   

>  mangle:
> 
>                   This table is used for specialized packet alteration.  [...] and OUTPUT (for altering locally-generated packets before routing)

 

----------

## Bones McCracker

 *truc wrote:*   

>  *Quote:*    mangle:
> 
>                   This table is used for specialized packet alteration.  [...] and OUTPUT (for altering locally-generated packets before routing) 

 

Thank you, truc.

My mistake.  The "prerouting" chain is for inbound only.  Use the "output" chain of the mangle table (and the "postrouting" chain of the "nat" table, for the corresponding SNAT rules).

----------

## dbishop

This is now working, thank you all for the help.  For those that may follow this discussion, the the issue that's different is that this is written for fwmark'ing traffice generated by the marking host itself -- most of the howto's written with the expectation that the router doing the firewall marking will be doing so for the packets it is forwarding, that is, not for packets locally generated by the host itself.  The few that discuss the mangle table's OUTPUT chain are a little confusing (and somewhat contradictory at times).

For clarity, I have a single host that has three NICs, each connecting to a different ISP.  I desired some routes to be chosen by UID owner, some by port, and everything else by default route.

There are several kernel options that need to be included.  I have probably switched on a few too many trying to make this work, but there are several you'll need.  My best advice would be to use the ncurses interface and then search for MARK.  You'll see a bunch of tags and their locations and dependencies.   The point is that for Gentoo and 2.6.39 gentoo-sources kernel this definitely works, so if you're having trouble like I did, keep going.  Here are the details of what I ultimately got working.

This snippet sets up the fwmarks in the mangle table's OUTPUT chain, followed by the nat table's POSTROUTING rules:

```

ETH1_IP="192.168.0.223"

ETH2_IP="192.168.20.6"

ETH3_IP="192.168.100.56"

$ISP1_ETH="eth0"

$ISP2_ETH="eth1"

$ISP3_ETH="eth2"

# FWMARK Port 22 (tcp), 25 (tcp) and Port 123 (udp) 

iptables -t mangle -A OUTPUT -p tcp --dport 22  -j MARK --set-mark 1

iptables -t mangle -A OUTPUT -p udp --dport 123 -j MARK --set-mark 2

iptables -t mangle -A OUTPUT -p tcp --dport 25 -j MARK --set-mark 3

# FWMARK base on UID

iptables -t mangle -A OUTPUT -m owner --uid-owner userone -j MARK --set-mark 1

iptables -t mangle -A OUTPUT -m owner --uid-owner usertwo -j MARK --set-mark 2

iptables -t mangle -A OUTPUT -m owner --uid-owner userthree -j MARK --set-mark 3

# SNAT for advanced routing redirects

iptables -t nat -A POSTROUTING -o $ISP1_ETH  -j SNAT --to-source $ETH1_IP

iptables -t nat -A POSTROUTING -o $ISP2_ETH  -j SNAT --to-source $ETH2_IP

iptables -t nat -A POSTROUTING -o $ISP3_ETH  -j SNAT --to-source $ETH3_IP

```

These rules should mark all NTP, SSH and SMTP traffic with 0x1, 0x2, and 0x3, respectively.  The SNAT rules make sure that the source IP is sensible for the interface being used.

There are a few other things that need to get done, and these are pretty close to what the myriad other howto's say.  You'll need some tables defined in /etc/iproute2/rt_tables.  Mine looks like this:

```
255 local

254 main

253 default

102 isp3

101 isp2

100 isp1

0   unspec

```

Then there are the routes and rules.  I am doing this from a script after all the interfaces are up.  I have found, for my purposes anyway, that this is the simplest way to make sure everything is right, without having to second-guess what the Gentoo net scripts may decide do  (i.e., I have everything configured correctly in /etc/conf.d/net but this makes everything just as I like it   :Rolling Eyes: 

```
# Flush the routing tables and rules

ip route flush table main

ip route flush table isp1

ip route flush table isp2

ip route flush table isp3

ip rule flush

# fwmark rules entries

ip rule add lookup default priority 32767

ip rule add lookup main priority 32766

ip rule add fwmark 1 priority 101 table isp1

ip rule add fwmark 2 priority 102 table isp2

ip rule add fwmark 3 priority 103 table isp3

# Add the main routing table entries

ip route add 127.0.0.0/8 via 127.0.0.1 dev lo

ip route add 192.168.0.128/25 dev eth0 src 192.168.0.223

ip route add 192.168.20.0/24 dev eth1 src 192.168.20.6

ip route add 192.168.100.0/24 dev eth2 src 192.168.100.56

ip route add default via 192.168.20.1

# isp1 table entries

ip route add 192.168.0.128/25 dev eth0 table isp1

ip route add default via 192.168.0.129 dev eth0 table isp1

# isp2 table entries

ip route add 192.168.20.0/24 dev eth1 table isp2

ip route add default via 192.168.20.1 dev eth1 table isp2

# isp3 table entries

ip route add 192.168.100.0/24 dev eth2 table isp3

ip route add default via 192.168.100.1 dev eth2 table isp3

# Flush the route cache

ip route flush cache

```

And you'll need to do this somewhere convenient too:

```
echo 1 > /proc/sys/net/ipv4/ip_forward

for f in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo 1 > $f ; done

```

Hopefully someone will find this useful, and equally hopefully, I haven't missed anything or mistyped anything either.  If I have, please correct it.

Thanks again.

----------

## Bones McCracker

Nice write-up.  After a while, when you've let your configuration stabilize, you might want to copy a final version of this over to the "tips and tricks" forum or to the Gentoo Wiki.

 *Quote:*   

> And you'll need to do this somewhere convenient too:
> 
> ```
> echo 1 > /proc/sys/net/ipv4/ip_forward
> 
> ...

 

One good place to set those is in /etc/sysctl.conf.  Another option, if you are scripting it, is to use the sysctl command.  The code you have above is more useful than these two options if your interfaces may change.

On your machine, ip_forward is quite possibly already defaulting to 1, since you may have the kernel config option "IP: Advanced Router" enabled, which I believe changes the default value for that sysctl, along with one or two others.

I'm not sure you actually need or want to enable forwarding, if you are only routing packets originating on the local host, although it's conceivable that this is required for other routing functions.

----------

## dbishop

Okay, will-do on the Gentoo Wiki, etc., good idea.  I've written a few before.  This was written partly to say let you know what I finally got working and partly to help others gain some insight, gentoo-y-gentoo.  And because of that, the writing p.o.v. does seem to annoyingly flip between those two purposes.

Regarding the /etc/sysctrl.conf, that's where I have IP Forwarding set to 1.  Not sure if it's needed for DNAT or SNAT either, but somewhere in the back of my mind I recall that for nat'ing to work this has to be a 1, and that the advanced routing kernel option seems to set it anyway (it is set for me).  At some point I'll try it set to 0, but many readers of this will probably be using such a box as a LAN router too and will need it.

As for the rp_filter,  once I get everything all trimmed up I will probably reduce this to only the interfaces involved, i.e., eth0, eth1 and eth2 in this case.

The /proc/sys/ipv4/... settings are mostly a mystery to me. they're not hard to understand, they're just hard to find out about, and even harder to figure out the side effects.  I only ever look here when someone else tells me to.  Some of these I know from experience, some not so much, and much of the information is hard to understand.  For example, I read more than one site on this very subject that said to set rp_filter to "2", but when I looked up the docs, it reportedly is a boolean.  Maybe "2" is a quantum state for this setting, maybe docs are out of sync, maybe it's a kernel version thing, who knows.  Question: Do you know someplace reliable and straightforward that covers what all these do and their interactions and any collateral/consequential effects?

Thanks again for all your help.  All you guys are great   :Very Happy: 

----------

## Bones McCracker

 *dbishop wrote:*   

> The /proc/sys/ipv4/... settings are mostly a mystery to me. they're not hard to understand, they're just hard to find out about, and even harder to figure out the side effects.  I only ever look here when someone else tells me to.  Some of these I know from experience, some not so much, and much of the information is hard to understand.  For example, I read more than one site on this very subject that said to set rp_filter to "2", but when I looked up the docs, it reportedly is a boolean.  Maybe "2" is a quantum state for this setting, maybe docs are out of sync, maybe it's a kernel version thing, who knows.  Question: Do you know someplace reliable and straightforward that covers what all these do and their interactions and any collateral/consequential effects?

 

I recall something of a dearth of documentation on these too.  I think the best source is probably the kernel docs:

/usr/src/Documentation/networking/ (e.g. ip-sysctl.txt)

/usr/src/linux/Documentation/sysctl (e.g., net.txt)

----------

