# [SOLVED] Load balanced network failing intermitently

## kuteninja

I'm trying to balance our internet connection with 2 different ISP using iptables and iproute. It was pretty hard since I'm using only one ethernet card, with an alias (eth0:1) and the iptables prerouting rules made me try to kill myself more than once since they don't accept ethernet alias cards.

I've made it work, and I can browse the internet and my IP address change between one link and the other on almost every refresh, but sometimes after pressing a few times the F5 key or just randomly, it hangs and starts loosing packets, then, after a few seconds it starts working again. It's really annoying and I believe that it has to be some mistake on the iptables rules, but I can't figure it out yet.

Could you help me? I'm attaching all the rules I'm executing to see if I'm doing something wrong.

Notes:

- IP addresses were censored by security to XXX

- $LAN is my lan eth card (eth1 in this case), and $WAN my wan card (eth0+ to get both eth0 and eth0:1).

- I've already added the tables 10 and 20 on /etc/iproute2/rt_tables (one per each ISP)

```
# Drop all tables

/sbin/ip route del default table main 2>/dev/null

/sbin/ip route del default table 10 2>/dev/null

/sbin/ip route del default table 20 2>/dev/null

/sbin/ip rule del fwmark 10 2>/dev/null

/sbin/ip rule del fwmark 20 2>/dev/null

# Add the gateways and firewall marks

# 10 > isp1-table + 20 > isp2-table

/sbin/ip rule add fwmark 10 table 10

/sbin/ip rule add fwmark 20 table 20

/sbin/ip route add default via 190.2.37.XXX table 10

/sbin/ip route add default via 190.210.43.XXX table 20

/sbin/ip route add default via 190.2.37.XXX table main

# Copy all the internal routes

/sbin/ip route show table main | grep -Ev '^default' | while read ROUTE ; do

     /sbin/ip route add table 10 $ROUTE

     /sbin/ip route add table 20 $ROUTE

done

# Flush cache

/sbin/ip route flush cache

# Cleanup

/sbin/iptables -t mangle -F 2>&1

/sbin/iptables -t mangle -X 2>&1

/sbin/iptables -t mangle -F balance1 >/dev/null 2>&1

/sbin/iptables -t mangle -X balance1 >/dev/null 2>&1

/sbin/iptables -t mangle -N balance1

# Load balancing rules (Split 50/50 between fwmark 10 / 20)

/sbin/iptables -t mangle -A balance1 -d 192.168.0.0/16      -j RETURN

/sbin/iptables -t mangle -A balance1 -m connmark ! --mark 0 -j RETURN

/sbin/iptables -t mangle -A balance1 -m state --state ESTABLISHED,RELATED -j RETURN

/sbin/iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 0 -j CONNMARK --set-mark 10

/sbin/iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 1 -j CONNMARK --set-mark 20

# Check to see if we have already marked a packet

/sbin/iptables -t mangle -A PREROUTING  -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark

/sbin/iptables -t mangle -A OUTPUT      -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark

# Mark incoming connections to return on the interface they came in on

# Check which one of the networks the incoming packets wants to reach, we mark that packet accordly

/sbin/iptables -t mangle -A PREROUTING -i $WAN -d 190.2.37.XXX/29 -m state --state NEW  -j CONNMARK --set-mark 10

/sbin/iptables -t mangle -A PREROUTING -i $WAN -d 190.210.43.XXX/30 -m state --state NEW  -j CONNMARK --set-mark 20

# New outgoing packets

/sbin/iptables -t mangle -A PREROUTING -i $LAN -m state --state NEW  -j balance1

/sbin/iptables -t mangle -A OUTPUT -m state --state NEW -j balance1

# Choose our route and save the mark

/sbin/iptables -t mangle -A PREROUTING -m connmark --mark 10 -j MARK --set-mark 10

/sbin/iptables -t mangle -A PREROUTING -m connmark --mark 20 -j MARK --set-mark 20

/sbin/iptables -t mangle -A PREROUTING -m state --state NEW -m connmark ! --mark 0 -j CONNMARK --save-mark

# Forwarding $LAN > $WAN

/sbin/iptables -I FORWARD -i $LAN -d 192.168.0.0/16 -j DROP # LAN > LAN (DROP)

/sbin/iptables -A FORWARD -i $LAN -s 192.168.0.0/16 -j ACCEPT # LAN > WAN (OK)

/sbin/iptables -A FORWARD -i $WAN -d 192.168.0.0/16 -j ACCEPT # WAN > LAN (OK)

# Masquerade all outgoing packets

/sbin/iptables -t nat -A POSTROUTING -o $WAN -j MASQUERADE
```

I also have some other lines to NAT some ports to my local network and blocking some ports but that's not quite relevant here.Last edited by kuteninja on Mon Jul 18, 2011 10:33 pm; edited 1 time in total

----------

## truc

since you don't really say what's happening when it hangs(see tcpdump, enable logging via iptables and so on), I can only make a few comments:

```
/sbin/iptables -t mangle -A PREROUTING -i $LAN -m state --state NEW  -j balance1

/sbin/iptables -t mangle -A OUTPUT -m state --state NEW -j balance1 
```

why Are you then returning when pakets are ESTABLISHED or RELATED in the balance1 chain? there should only be NEW pakets anyway? 

And, instead of using MASQUERADE, couldn't you SNAT with the right ip source (knowing the MARK)?

----------

## kuteninja

 *truc wrote:*   

> why Are you then returning when pakets are ESTABLISHED or RELATED in the balance1 chain? there should only be NEW pakets anyway? 

 

You're right... anyway that shouldn't do anything if you're right.

I'll try to do some tcpdump and give you a more exact issue to look at.

 *truc wrote:*   

> And, instead of using MASQUERADE, couldn't you SNAT with the right ip source (knowing the MARK)?

 

I'm masquerading because this server is a linux router that gives internet access to a LAN, and so the connections start as 192.168.0.X and when that traffic comes out it should be masked to return with the external IP address, and then "unmasked" to the LAN user.

----------

## truc

 *kuteninja wrote:*   

>  *truc wrote:*   And, instead of using MASQUERADE, couldn't you SNAT with the right ip source (knowing the MARK)? 
> 
> I'm masquerading because this server is a linux router that gives internet access to a LAN, and so the connections start as 192.168.0.X and when that traffic comes out it should be masked to return with the external IP address, and then "unmasked" to the LAN user.

 

search for SNAT in the iptables manual then re-read my comment

----------

## kuteninja

 *truc wrote:*   

> search for SNAT in the iptables manual then re-read my comment

 

I've added the SNAT rules you've asked me but it still fails, on tcpdump it only shows the outgoing traffic not sure why, but it seems to be Ok anyway. Also I've changed to another way to mark packets with a random 0.5 probabilty instead.

Here's my new iptables, the ip routes are the same, as usual I've omitted the NATs and such:

```
iptables -t mangle -F 2>&1

iptables -t mangle -X 2>&1

# Forwarding / Masquerade from LAN > WAN (Router)

iptables -A FORWARD -i ${LANIF} -j ACCEPT # LAN > WAN (OK)

iptables -A FORWARD -i ${WANIF} -j ACCEPT # WAN > LAN (OK)

iptables -t nat -A POSTROUTING -o ${WANIF} -j MASQUERADE

# Load balancing rules (Split 50/50 between fwmark 1/2)

iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark

iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j ACCEPT

iptables -t mangle -A PREROUTING -j MARK --set-mark 10

iptables -t mangle -A PREROUTING -m statistic --mode random --probability 0.5 -j MARK --set-mark 20

iptables -t mangle -A PREROUTING -j CONNMARK --save-mark

# Mark incoming connections to return on the interface they came in on

iptables -t nat -A POSTROUTING -m connmark --mark 10 -j SNAT --to-source 190.2.37.XXX

iptables -t nat -A POSTROUTING -m connmark --mark 20 -j SNAT --to-source 190.210.43.YYY
```

Here's the tcpdump, I'm trying to view a page hosted on 190.210.132.105:443 (https), and my two external IP addressess are 190.210.43.YYY and 190.2.37.XXX

```
22:30:02.165719 IP (tos 0x0, ttl 127, id 23456, offset 0, flags [DF], proto TCP (6), length 52) 190.210.43.YYY.54228 > 190.210.132.105.443: S, cksum 0xe3a6 (correct), 918892913:918892913(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>

22:30:02.166688 IP (tos 0x0, ttl 127, id 23457, offset 0, flags [DF], proto TCP (6), length 52) 190.210.43.YYY.54230 > 190.210.132.105.443: S, cksum 0x5862 (correct), 722192493:722192493(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>

22:30:03.596689 IP (tos 0x0, ttl 127, id 23460, offset 0, flags [DF], proto TCP (6), length 52) 190.210.43.YYY.54231 > 190.210.132.105.443: S, cksum 0x362b (correct), 2952881581:2952881581(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>

22:30:04.086118 IP (tos 0x0, ttl 127, id 23462, offset 0, flags [DF], proto TCP (6), length 52) 190.2.37.XXX.54232 > 190.210.132.105.443: S, cksum 0xb908 (correct), 50697965:50697965(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>

22:30:04.087211 IP (tos 0x0, ttl 127, id 23463, offset 0, flags [DF], proto TCP (6), length 52) 190.210.43.YYY.54233 > 190.210.132.105.443: S, cksum 0x29d6 (correct), 2506198176:2506198176(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>

22:30:04.087699 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto TCP (6), length 52) 190.210.132.105.443 > 190.2.37.XXX.54232: S, cksum 0x48c3 (correct), 1981612872:1981612872(0) ack 50697966 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>

22:30:04.088016 IP (tos 0x0, ttl 127, id 23464, offset 0, flags [DF], proto TCP (6), length 40) 190.2.37.XXX.54232 > 190.210.132.105.443: ., cksum 0x9f65 (correct), 1:1(0) ack 1 win 256

22:30:04.088365 IP (tos 0x0, ttl 127, id 23465, offset 0, flags [DF], proto TCP (6), length 263) 190.2.37.XXX.54232 > 190.210.132.105.443: P 1:224(223) ack 1 win 256

22:30:04.089054 IP (tos 0x0, ttl 127, id 23466, offset 0, flags [DF], proto TCP (6), length 52) 190.2.37.XXX.54234 > 190.210.132.105.443: S, cksum 0x83fd (correct), 4138326097:4138326097(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>

22:30:04.089904 IP (tos 0x0, ttl 62, id 4306, offset 0, flags [DF], proto TCP (6), length 40) 190.210.132.105.443 > 190.2.37.XXX.54232: ., cksum 0x9f50 (correct), 1:1(0) ack 224 win 54

22:30:04.090149 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto TCP (6), length 52) 190.210.132.105.443 > 190.2.37.XXX.54234: S, cksum 0x9423 (correct), 1982694092:1982694092(0) ack 4138326098 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 7>

22:30:04.090276 IP (tos 0x0, ttl 127, id 23468, offset 0, flags [DF], proto TCP (6), length 40) 190.2.37.XXX.54234 > 190.210.132.105.443: ., cksum 0xeac5 (correct), 1:1(0) ack 1 win 256

22:30:04.090289 IP (tos 0x0, ttl 62, id 4307, offset 0, flags [DF], proto TCP (6), length 185) 190.210.132.105.443 > 190.2.37.XXX.54232: P 1:146(145) ack 224 win 54

22:30:04.090598 IP (tos 0x0, ttl 127, id 23469, offset 0, flags [DF], proto TCP (6), length 263) 190.2.37.XXX.54234 > 190.210.132.105.443: P 1:224(223) ack 1 win 256

22:30:04.091154 IP (tos 0x0, ttl 127, id 23470, offset 0, flags [DF], proto TCP (6), length 99) 190.2.37.XXX.54232 > 190.210.132.105.443: P 224:283(59) ack 146 win 256

22:30:04.092035 IP (tos 0x0, ttl 62, id 37200, offset 0, flags [DF], proto TCP (6), length 40) 190.210.132.105.443 > 190.2.37.XXX.54234: ., cksum 0xeab0 (correct), 1:1(0) ack 224 win 54

22:30:04.092743 IP (tos 0x0, ttl 62, id 37201, offset 0, flags [DF], proto TCP (6), length 185) 190.210.132.105.443 > 190.2.37.XXX.54234: P 1:146(145) ack 224 win 54

22:30:04.093926 IP (tos 0x0, ttl 127, id 23471, offset 0, flags [DF], proto TCP (6), length 99) 190.2.37.XXX.54234 > 190.210.132.105.443: P 224:283(59) ack 146 win 256

22:30:04.132380 IP (tos 0x0, ttl 62, id 4308, offset 0, flags [DF], proto TCP (6), length 40) 190.210.132.105.443 > 190.2.37.XXX.54232: ., cksum 0x9e84 (correct), 146:146(0) ack 283 win 54

22:30:04.135389 IP (tos 0x0, ttl 62, id 37202, offset 0, flags [DF], proto TCP (6), length 40) 190.210.132.105.443 > 190.2.37.XXX.54234: ., cksum 0xe9e4 (correct), 146:146(0) ack 283 win 54

22:30:06.593868 IP (tos 0x0, ttl 127, id 23485, offset 0, flags [DF], proto TCP (6), length 52) 190.210.43.YYY.54231 > 190.210.132.105.443: S, cksum 0x362b (correct), 2952881581:2952881581(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>

22:30:07.086900 IP (tos 0x0, ttl 127, id 23487, offset 0, flags [DF], proto TCP (6), length 52) 190.210.43.YYY.54233 > 190.210.132.105.443: S, cksum 0x29d6 (correct), 2506198176:2506198176(0) win 8192 <mss 1460,nop,wscale 8,nop,nop,sackOK>
```

PS: Commenting this line, everything works, but I have no balacing, adding it, it seems to randomly fail:

```
iptables -t mangle -A PREROUTING -m statistic --mode random --probability 0.5 -j MARK --set-mark 20 
```

----------

## Mad Merlin

iptables is voodoo magic to me, but a thought... does this only happen for HTTPS sites? HTTPS won't work with two different source IPs in the same session. Maybe try making HTTPS sessions sticky to one source IP once they're initiated.

----------

## kuteninja

 *Mad Merlin wrote:*   

> iptables is voodoo magic to me, but a thought... does this only happen for HTTPS sites? HTTPS won't work with two different source IPs in the same session. Maybe try making HTTPS sessions sticky to one source IP once they're initiated.

 

It's possible that only HTTPS sites fail, since that's the only error reported on the network that I was able to reproduce easily.

I'd love to make all the connections "stickied" to the network it choosed on the first access, but I'm not sure how to do that on iptables.

I could also force the 443 port to get out using always the same mark, but that would saturate one network more than the other.

I'm gonna try to remove the masquerade completely, since I've added SNAT but forgot to remove the line that truc was complaying about, so maybe that didn't actually fixed anything, hehe.

----------

## kuteninja

I've been trying to make this work for a while, but I'm still stucked

On every tutorial I've came across, there's allways 2 (or more) wan devices to set the load balancing on, and therefore it's really simple with -o and -i using each interface to mark packets and such, but isn't there any way to set this up with just 2 cards: eth1 for lan and eth0 + eth0:1 for wan balancing?

I'm calling all iptables gurús out there for some help  :Sad: 

I need an iptables / iproute script that allows me to...

1. Accept and masquerade connections from LAN (eth1) to WAN (eth0 and eth0:1)

2. Balance the traffic between two different gateways on eth0 and eth0:1

3. Make it work seamlessly or at least with a sticky / cached mode 

If this is not possible and I need to buy an eth2, please tell me so I can go get one and stop suffering  :Razz: 

----------

## truc

two things:

1) You've probably made some modifications in your iptables rules, could you show them again, as well as what is actually not working as you want.

2) I suppose eth0 is connected on a switch right? does this switch supports the 802.1Q norm? (VLANS), if so, then ou definitely want to use this(you'll have two vlans interface on eth0)

----------

## kuteninja

 *truc wrote:*   

> 1) You've probably made some modifications in your iptables rules, could you show them again, as well as what is actually not working as you want.

 

I've changed it following other tutorials, but I couldn't get it to work, so it's pointless that I upload it here. I'm allways on the same issue, how do I know when a packet has to go thru the second gateway since I only have one interface to that.

The last I've checked was this one for a dd-wrt router, that seems to be applicable to a linux router too:

http://www.dd-wrt.com/wiki/index.php/Dual-WAN_for_simple_round-robin_load_equalization

 *truc wrote:*   

> 2) I suppose eth0 is connected on a switch right? does this switch supports the 802.1Q norm? (VLANS), if so, then ou definitely want to use this(you'll have two vlans interface on eth0)

 

Yes, you're right, but it's not administrable and it doesn't have vlans. 

Is the VLAN mandatory to this?

----------

## kuteninja

 *truc wrote:*   

> 2) I suppose eth0 is connected on a switch right? does this switch supports the 802.1Q norm? (VLANS), if so, then ou definitely want to use this(you'll have two vlans interface on eth0)

 

I LOVE YOU !   :Laughing: 

Making a vlan, I ended up with an interface eth0.2 which I can use to masquerade and nat properly, and the balancing seems to work now  :Very Happy: 

This is my final iptables / iproute script in case it helps anyone else (replace the values like {this}, eth0 is my WAN, eth1 my Lan):

```
# VLAN SETUP

vconfig add eth0 2

ifconfig eth0.2 {ISP2-IP} netmask 255.255.255.252

# Limpia tablas de ruteo

/sbin/ip route del default table main 2>/dev/null

/sbin/ip route del default table 10 2>/dev/null

/sbin/ip route del default table 20 2>/dev/null

/sbin/ip rule del fwmark 10 2>/dev/null

/sbin/ip rule del fwmark 20 2>/dev/null

# Reglas de ruteo

/sbin/ip rule add fwmark 10 table 10

/sbin/ip rule add fwmark 20 table 20

/sbin/ip route add default via {ISP1-GW} table 10

/sbin/ip route add default via {ISP2-GW} table 20

/sbin/ip route add default via {ISP1-GW} table main

# Copia red interna

/sbin/ip route show table main | grep -Ev '^default' | while read ROUTE ; do

     /sbin/ip route add table 10 $ROUTE 2>/dev/null # rutas main > iplan1

     /sbin/ip route add table 20 $ROUTE 2>/dev/null # rutas main > iplan2

done

# Flush cache

/sbin/ip route flush cache

# Cleanup

/sbin/iptables -t mangle -F 2>&1  # limpia la cadena mangle

/sbin/iptables -t mangle -X 2>&1  # limpia la cadena mangle

# Load balancing rules (Split 50/50 between fwmark 1/2)

iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark

iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j ACCEPT

iptables -t mangle -A PREROUTING -j MARK --set-mark 10

iptables -t mangle -A PREROUTING -m statistic --mode random --probability 0.5 -j MARK --set-mark 20

iptables -t mangle -A PREROUTING -j CONNMARK --save-mark

# Masquerade

iptables -t nat -A POSTROUTING -o eth0.2 -j MASQUERADE
```

BTW, this is just the balancing, the gentoo router was made separately, but I've used the Gentoo oficial wiki for that.

The credits for the iptables script goes to this guy: 

http://serverfault.com/questions/93678/load-balancing-nat-ing-multiple-isp-connections-on-linux

----------

## truc

So your switch doesn't do VLANS but you ended up doing VLANs anyway? I suppose a node somewhere is removing the VLAN field in your frames?

I see you still use MASQUERADE, but since you already have the {ISP2-IP} address in your script, you should use SNAT --to-source {ISP2-IP} instead (for performance reason)

So, you public IP address can change while browsing a single web page (several distincts requests for a single page: css, js, images..), how does this affect your browsing? I'm especially thinking about restricted access website like your bank, your webmail and so on? Does this cause any problem at all?

thanks!

----------

## kuteninja

 *truc wrote:*   

> So your switch doesn't do VLANS but you ended up doing VLANs anyway? I suppose a node somewhere is removing the VLAN field in your frames?

 

Yeap, I took another switch that was arount to try it, I'll buy a new one later.

 *truc wrote:*   

> I see you still use MASQUERADE, but since you already have the {ISP2-IP} address in your script, you should use SNAT --to-source {ISP2-IP} instead (for performance reason)

 

SNAT seems to work the same way, but it breaks the PPTP when I try to use it from the LAN, I can use SNAT and put the PPTP address manually, but if it changes I have to change the line, so it's easier to just masquerade. The router is a quad core, so performance is not a issue for now.

 *truc wrote:*   

> So, you public IP address can change while browsing a single web page (several distincts requests for a single page: css, js, images..), how does this affect your browsing? I'm especially thinking about restricted access website like your bank, your webmail and so on? Does this cause any problem at all?

 

Well... some pages with IP+cookie protection are a pain in the ass... but this was for my work and our servers doesn't have that protection, so It's not a big deal actually. It was really the only way to allow downloading stuff using both ISP's at the same time with multiple connections, which was something else my boss asked for.

----------

## truc

 *kuteninja wrote:*   

>  *truc wrote:*   So, you public IP address can change while browsing a single web page (several distincts requests for a single page: css, js, images..), how does this affect your browsing? I'm especially thinking about restricted access website like your bank, your webmail and so on? Does this cause any problem at all? 
> 
> Well... some pages with IP+cookie protection are a pain in the ass... but this was for my work and our servers doesn't have that protection, so It's not a big deal actually. It was really the only way to allow downloading stuff using both ISP's at the same time with multiple connections, which was something else my boss asked for.

 

Ok, thank you for your feed back. I think I'll continue doing the load balancing my way (load balancing is only done for http{,s}: traffic for some domains uses the alternate link while everything else uses the main one).

----------

