# Dual Connection Balanced Routing

## paulh430

The network at my job just got a second T1 to allow for extra bandwidth.  We had a T1->Cisco->Gentoo Box->Private & Public network.  We want to add the second T1 into the mix without redoing our network and BGP protocol is not an option.  The network will is now setup like the diagram shown below:

http://blipz.net/network.jpg

The bridge interface on the Gentoo box has pingable (from the outside world) ips from both ISPs.  What I would like to do next is somehow balance the traffic over the 2 T1's as evenly as possible.  I built the box from scratch, so I have decent knowledge of bridging and firewalls.  I also know that BGP is not an option for us, which is why I am trying to handle it with our Linux router/firewall.

Anyone have any suggestions?

----------

## kashani

To be honest I'd drop the Gentoo box altogether, make sure cef and ip route caching is turned on, set two default routes, and NAT at each serial interface.

You need cef, cisco express forwarding, because you want route caching or you'll get packets round robining down your two pipes. With route caching and whatnot the entire stream will be locked into a particular pipe. That'll keep strange out of packet stuff from happening.

Two default routes is pretty straightforward, but I'd point them to an interface rather than an IP. Should tend to drop the routes if the interface goes down.

ip route 0.0.0.0 0.0.0.0 serial0

ip route 0.0.0.0 0.0.0.0 serial1

Now the NAT. I don't have a config for this off hand, but the idea as that you need to source all your connections from an IP on the connection you're using. If you don't do that you get asymetrical routing and you're back to having strange things happen with out of order packets or any local firewall dropping the packets.

If you've got DMZ type stuff that might make this more complicated, but I'm still of the opinion that it's easier to deal with all the routing and NATing on the router. YMMV.

kashani

----------

## paulh430

 *kashani wrote:*   

> To be honest I'd drop the Gentoo box altogether, make sure cef and ip route caching is turned on, set two default routes, and NAT at each serial interface.
> 
> You need cef, cisco express forwarding, because you want route caching or you'll get packets round robining down your two pipes. With route caching and whatnot the entire stream will be locked into a particular pipe. That'll keep strange out of packet stuff from happening.
> 
> Two default routes is pretty straightforward, but I'd point them to an interface rather than an IP. Should tend to drop the routes if the interface goes down.
> ...

 

Thats not really an option tho, unless the router can do more than I am understanding (I, admittedly, don't know Cisco at all, I can log in and reboot the thing, thats it).  We have public servers on the public (bridged) connection.  There is the eth2 that privides our private network, 10.x.x.x for all of our workstations, providing dhcp, dns, etc.  The gentoo box also provides VPN (OpenVPN) access to mobile/late night workers.  Also, with the Gentoo box being a bridge, I can monitor all traffic and find bandwidth hogs, etc.  All of this is required and can't be lost (changed if necessary, but not lost).

I mean, if the Cisco can do all this, please tell me and Ill do whatever research necessary to do that, but it would be easier if I could keep the network the way that it is because it's working and why fix it if it isn't broken?  :Smile: 

----------

## kashani

Ah with servers we've got other issues. Let's assume the following.

ISP1 gives you 20.0.0.0/28 as your routable space

ISP2 gives you 30.0.0.0/28 as your routable space

All your servers are on 20.0.0.0/28 as you haven't turned up ISP2 yet. Let's say you add ISP2, but don't change any IP's. If ISP1 were to go down, all your servers become unreachable. However, internal NAT users on private IP space behind the Gentoo box would be able to route out... but only if they are being sourced from 30.0.0.0/28. Unfortunately that's hard to control. Your Gentoo box hands packets off the router right now and just assumes that things work. The Cisco is smart enough to drop the route to ISP1 when it goes down, but that still leaves the problem of the source IP.

You might be able to solve that with some policy routing maps or double NATing, but it's going to be brittle and still cause you issues. 

With me so far?

If both ISP's are always working, just adding a route to the Cisco to use both connections as default routes will work just fine. If ISP2 goes down things still continue to work just fine. However if ISP1 goes down you're screwed because all your source IP's are ISP1's and that is the only route the rest of the internet knows to use to get to those IP's. If incoming traffic becomes larger than a 1.5mb/s you're also screwed. That's because all traffic is going to come in over ISP1 because all your source IP's are ISP1 IP's. You're going to have no incoming traffic over ISP2 unless connections are tryning to get to ISP2 IP's.

I'm not sure how much of this you knew so the question is what sort of problem are you trying to solve? Just more bandwidth? More bandwidth and reliablility? More bandwidth outgoing? More bandwidth incoming?

If BGP isn't an option and it likely isn't because you don't have a full class C of addresses then I really suggest having two T1's to the same ISP. You might even be able to talk them into terminating them on different routers for a bit more reliability though two T1's to the same gear allow multilink bonding where will give you better performance. If you do have a class C then BGP may actually be an option and is vastly simplier and a ton more bulletproof than anything I've mentioned above. Don't discount BGP just because you don't understand it. 

kashani

----------

## msutton

I agree go with the same ISP and get the T1's bonded.  Much simpler for you by letting the ISP handle all the configuration.

Then you just have to plug into the IAD or Router on your end and its up.

----------

## paulh430

 *kashani wrote:*   

> Ah with servers we've got other issues. Let's assume the following.
> 
> ISP1 gives you 20.0.0.0/28 as your routable space
> 
> ISP2 gives you 30.0.0.0/28 as your routable space
> ...

 

Not really worried if 1 T1 goes down.  We are just trying to get more download bandwidth for the LAN.  All of our servers here are not mission critical.  The 2nd T1 is purely for bandwidth.

 *kashani wrote:*   

> If both ISP's are always working, just adding a route to the Cisco to use both connections as default routes will work just fine. If ISP2 goes down things still continue to work just fine. However if ISP1 goes down you're screwed because all your source IP's are ISP1's and that is the only route the rest of the internet knows to use to get to those IP's. If incoming traffic becomes larger than a 1.5mb/s you're also screwed. That's because all traffic is going to come in over ISP1 because all your source IP's are ISP1 IP's. You're going to have no incoming traffic over ISP2 unless connections are tryning to get to ISP2 IP's.
> 
> I'm not sure how much of this you knew so the question is what sort of problem are you trying to solve? Just more bandwidth? More bandwidth and reliablility? More bandwidth outgoing? More bandwidth incoming?
> 
> 

 

More download bandwidth for the LAN.

 *kashani wrote:*   

> If BGP isn't an option and it likely isn't because you don't have a full class C of addresses then I really suggest having two T1's to the same ISP. You might even be able to talk them into terminating them on different routers for a bit more reliability though two T1's to the same gear allow multilink bonding where will give you better performance. If you do have a class C then BGP may actually be an option and is vastly simplier and a ton more bulletproof than anything I've mentioned above. Don't discount BGP just because you don't understand it. 
> 
> 

 

I wanted it that way, but the boss made his own decision and decided to get the 2nd T1 thru another provider  :Sad: .  I would have loved to just let the first T1's ISP do BGP or bonding and just be done with it, but alas, I had no control over that decision.

 *msutton wrote:*   

> I agree go with the same ISP and get the T1's bonded.  Much simpler for you by letting the ISP handle all the configuration.
> 
> Then you just have to plug into the IAD or Router on your end and its up.

 

I wish I could do that, however, the boss already ordered from a different company and it has been installed.  As I said above to kashani, it wasn't my choice for seperate ISPs.

Thanks for the help.

----------

## paulh430

*bump*

----------

## splooge

What's your question?  I think kashani summed it up fairly well ... 

If:

1) it's only for ftp and

2) you originate the ftp requests and

3) you don't expect this to make your connection to the net 'highly available' or 'redundant'

then you can do what kashani said and simply add a 2nd default gateway on your router.  Outbound connections will be "balanced" out each T1.  It's certainly not a very pretty solution, but you get what you pay for.

----------

## Chewwit

I ran into a similar problem when I had a Devil Linux router connected to 2 ISPs, and did a load of reading about it. Admittedly this isn't your cisco jobby, but you can solve this problem to a limited extent by following this guide

http://www.tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.rpdb.multiple-links.html

ie, run the two ISPs into seperate ethernet devices (could be 2 on one card or two seperate cards) and use you gentoo box to do the routing/masquerading. (I don't know how acceptable this is from a security/practical point of view, so apologies If this is unacceptable.)

Using this code the linux router will send each outgoing packet to the least used interface at that point in time, roughly load balancing the isps (this is as I understand it.) There are some ways of doing more refined versions, but this is simple and quick. As Linux wasn't really built to do this it is actually quite a complex problem (as you're no doubt realising...). The catch is you MUST have a different gateway to route for each outgoing interface, otherwise this doesn't work. as you have 2 different ISPS this should actually be in your favour.

If your worried about security on your gentoo server you can replace the cisco router with a linux one (again don't know how acceptable this is with your companes policy). There are many Linux versions dedicated to this the only one I have experiance is Devil Linux, which worked well for me. These are pre built hardened distributions that run off a Live CD. Basically, get an old computer box (an i486 will happilly route 64+ machines) fill with RAM, remove the hard disk and install an optical one. Run the isp connections into this, and branch off what you want to your network. You could even run your public network off this router in a DMA, actually enhancing security if this suits. Set up the router as per instructions, settings go on a floppy or a pen drive. I prefer pendrives as they are MUCH more reliable, but do require a usb port. This must be hard write protectable. The computer then boots to RAM from the cd, runs in RAM, loading firewall settings etc from the pen drive and handles routing/1st firewall. If it is compromised there is no permanent writable media to write to, just reboot, any rootkit is wiped from ram, and your router is back up and running, clean with its old settings. Some commercial routers are actually much modified versions of this system. Using IpTables it is quite possible to set up complex rules determining which data type, etc goes where, and who can approach which part of the network through which interface. e.g. access to web servers comes through isp1 only, so if you have a dos attack, you retain half your bandwidth through isp2.

Also using 2 linux firewalls enhances security as a flaw in one distro is quite likely not to be in the second (differnt kernels, settings etc) so an intruder has to compromise 2 systems using differnent vunerabilities, and runs the risk of you rebooting leaving him to start frrom scratch in the first place.

I had this system running for a year, once setup it is very reliable, I had no downtime, even powercuts didn't effect it, set your bios to resume in last power state, and when power is back, so is your router, immeadiately protecting your network.

Hope this helps

Chewwit

----------

## baboon

Does the Cisco device support OSPF?

----------

## kashani

Just about everything from a Cisco 1601 on up supports OSPF. However running OSPF to your upstream is considered dangerous, a bad idea, and doesn't really help you in this situation. The problem is still NATing connections so that both interfaces can be used.

kashani

----------

