# Two NIC on same network issue[SOLVED]

## TBKDan

Ok.. I'm going to be up front and let you know that I am not the most experienced with Linux, let alone Gentoo.  But I can usually learn quick.  Now.. onto the problem I am having.  I'm a college student running some gameservers for my clan on my college network (Dual OC-3's and a T3), so I like to use the bandwidth.  The problem is that even though they have assloads of bandwidth, they only give students about 100kb/s upload, not enough for what I want to do.  So I put in a few NIC and it seemed to work fine on Windows.  I just migrated to Linux, since after testing on another machine it seemed to run the gameservers MUCH better.  The issue is that, since it is on the same network, they are trying to go to the same gateway, and I'm getting conflicts.  Route can't decide which address to send on or something... I'm not sure what exactly.  When I first boot, the damn thing wont even see ping outside the machine.  If I remove one of the routes, it only works for that remaining route (say if I remove the route entry for eth1, eth0 will still ping but eth1 will not).  This pretty much defeats the purpose, as I need to be able to send on individual IP addresses/interfaces to utilize the multiple cards.  Is there any way of doing this?  I've been trying for quite a while, but have still not had success.  I have iproute2 installed, and it is loaded via modules in /etc/conf.d/net, but no difference.  Running route takes forever to show the tables (if both routes are still in place; once I remove one default route it comes right up), but ip route pops right up.  I'm really at a loss of ideas... I'll gladly answer any questions/clarify anything that doesn't make sense.  Thanks for any ideas, Dan.Last edited by TBKDan on Wed Jan 11, 2006 11:27 pm; edited 1 time in total

----------

## jcampbell

RIT, huh? Mind you... you only have one jack to use, so you would need to be on a switch. The moment you do that, you are competing with yourself. A better idea would be to build a dedicated router and use your roommate's jack to double your bandwidth. You and your roommate could share a common gateway... and you each can register a NIC.

I would read this: http://linux.com.lb/wiki/index.pl?node=Load%20Balancing%20Across%20Multiple%20Links

----------

## TBKDan

Each IP gets 100kb/s, not jack.  Each NIC I have gets a diff IP, therefore more bandwidth.  Yes, I am going through a 12 port 3Com switch (gotta love eBay  :Very Happy: ), but I'm using nowhere near the max bandwidth for the switch itself.  Each IP in my dorm goes to the same gateway, there's no way around it, hence the troubles that I'm having  :Sad: 

*EDIT* Oh, and btw, I've thought about load balancing but I don't think that will suit my tastes, as since it's a gameserver I want just each server to have it's own, dedicated IP.

----------

## jcampbell

um... you're missing the point. RIT's intranet is on a gigabit backbone. This means that the trunk (core1 and core2 routers) and all branches are gigabit. So... life of a packet... leave your computer... hop to your floor's switch... hop to your dorm's switch... hop to dorm1 router... hop to core2 (usually)... hop to the inet1 gateway router... leave RIT. The bandwidth is capped on the hop from core2 to inet1 (depending on the subnet the packet originated from). Wireless subnets (residential side would be east1) are exempt from this cap.

Why does any of this matter? Because by teaming your NICs, the packets are switched on a 1 gbit enterprise switch versus your $50 100 mbit switch.

 *Quote:*   

> *EDIT* Oh, and btw, I've thought about load balancing but I don't think that will suit my tastes, as since it's a gameserver I want just each server to have it's own, dedicated IP.

 

That's fine... build a dedicated router to handle your load balancing.

----------

## TBKDan

Umm... I'm not entirely sure what point you're trying to make.  Yes, I know RIT has better hardware than I do, but that doesn't mean that if I get a better switch I'll get more bandwidth (what's kinda scary is how much you know about RIT... do you work here? o.O)  Each IP is restricted to 100kb/s.  But if I have multiple IP's, I'll get more bandwidth (ex. 2 IP's * 100kb/s = 200kb/s :p).  The switch is way more than capable of handling that kind of traffic, so it's just a matter of getting linux to cooperate with what I want it to do... Win had no problem with it  :Sad:   As for building a dedicated router... I'm a college student, do you think I'm really that rich? lol.  And besides, the router would just have one IP to transfer on as well, wouldn't it?  Therefore defeating the purpose of having multiple IP's.

Summing up my main thoughts: Download is fine, it's upload that I need more bandwidth with.  Each new IP will get me 100kb/s extra tacked onto my possible bandwidth quota.  I have two IP's on my machine, but due to iproute configuration, it is only sending on one of those IP's, therefore utilizing just 100kb/s of the possible bandwidth.  I need to get route to send on both IP's, with the main problem being that since they're trying to send on the same gateway they try to fight each other  :Sad: 

*EDIT* And yeah... I found out that the wireless was uncapped when I did a speed test and almost crapped myself lol.  Bad part of wifi: latency.  Gameserver+latency=nuh uh.  And if you do work at RIT... think you can come and help me?  :Very Happy:  lol

*EDIT2* Another btw, all of the load balancing stuff that I've looked up relies on the two ifaces sending to two different gateways, which is not my case, hence the pain in the ass this has been.  If I could just send to a different gateway I could stop pulling my hair out with this thing lol.  I really don't want to go back to Windows just so that I can utilize all the possible bandwidth on one machine  :Sad: 

----------

## jcampbell

I'm sorry... I didn't explain properly. You have two jacks, with a common gateway switching at a gigabit. If you have two network cards, you can double your bandwidth if they each have their own IP. Teaming the two NICs still allows you to do this because you're only configuring iptables to balance the load over the two NICs. By switching at 100 mbits at your jack, you end-up competing with your own traffic. RIT's switches can do this faster than yours. Also, depending on what dorm/floor you're in, it is possible for you and your roommate to be on two separate subnets (thus having two separate gateways).

 What I was suggesting was to build a common-gateway for you and your roommate. Allow that machine to perform the load balancing... that way it's transparent to your other machines. That article I linked to above, gives some ideas on how to accomplish this.

And yes, I do work for RIT.

----------

## TBKDan

I was considering something along those lines... if I build my own gateway, would the original packets from IP1 to my gateway then to the RIT's switch still retain their original IP source, therefore allowing me to keep my possible bandwidth?  And maybe I could give you a call, come visit you at your office, or something quicker than just posting on a forum? lol.  Oh, and my roomie and I are on the same gateway  :Sad:   I tried even routing to the wifi gateway, got destination unreachable when I tried to add it to the routing table.

*EDIT* (so many afterthoughts... lol)  I think I realize what you're saying.  By having two 100mbit NIC's going to one jack that can only hold 100mbit, I'm not going to be able to get 200mbit.  But that's fine, because I'm only talking 100kb per card, not mbit.  So by having those two cards each uploading at 100kb/s, I'm still nowhere near the 100mbit capacity of the wall jack.

----------

## jcampbell

Remember... the cap is at the last hop on the network. You are operating at 100 mbit (megabit != megabyte) at your jack. The routers perform traffic shaping long after the packets have left your machine.

 *Quote:*   

> I tried even routing to the wifi gateway, got destination unreachable when I tried to add it to the routing table.

 

Silly question... is your NIC registerd on the network... and do you have a signal?

----------

## TBKDan

Yeah, all the NIC's are registered (some of them with MAC cloning to simplify my life as I have had to change a few cards around).  I was just attempting to add the wifi gateway as another possible gateway for the second wired card so that I could finally lay this issue to rest.  I was able to ping it, but I wasn't able to actually add it as a gateway... kind of strange o.O  And I know, mbit!=mbyte, but I'm still not even using 100mbit, hell, I'm barely using 1mbit with one card for the server.  I was watching the bandwidth monitor last night with a completely full server, and it was peaking at about 110kbyte/s, and I have modified most of my server settings so that it will never really exceed my allotted bandwidth.  Since you work here... are there any other gateways that I could use so that I could solve this problem, or am I just out of luck at this point?

----------

## d_m

OK, forget about the argument about not using your own switch. If you had anything NEAR 100MBps going to the internet, you would have no worries. If all of my friends and I are on a single 100 MBps full-duplex switch, the game is gonna run pretty damn well. I would be *shocked* if latency between your two NICs from the switch was anything close to the normal latency of going across the college network/internet. Anyway...

I assume that about half the people are connecting to/communicating with your server using ipA and the other half using ipB (or your game has some way of saying "this server has multiple IP addresses, so use both"). Otherwise, if all your players connect to ipA, your server will probably respond to all of them on ipA and you won't see a benefit from having two NICs (i.e. your traffic will be rate-limited).

I assume your games use UDP, so they aren't connection-based per se, but still, any "response" from your server needs to probably come from the IP that the player sent their data to, so you need some way to distribute the load.

This set up should work for you, using "ip route":

```
# ip rule add from 192.168.0.5 table 5

# ip rule add from 192.168.0.6 table 6

# ip route add default via 192.168.0.1 dev eth0 table 5

# ip route add default via 192.168.0.1 dev eth1 table 6

# ip route flush cache
```

Where eth0 and eth1 are your NICs, and your IP addresses are 192.168.0.5 and 192.168.0.6 and your gateway is 192.168.0.1.

Good luck.

EDIT: of course, if the shaping is by network port and not by MAC address or IP address, then the other poster is right and you should borrow some extra ethernet jacks.

----------

## TBKDan

I tried just one first: ip rule add from MYIP table 5

I got: RTNETLINK answers: Invalid argument

 :Sad: 

Any other ideas?

BTW, I can tell gameserver to listen on a certain IP, but I can't get linux to transmit back on that IP because of route being a royal pain in my ass  :Sad: 

*EDIT* The shaping is via IP, as on windows I was able to get the bandwidth from each card.

----------

## jcampbell

 *TBKDan wrote:*   

> are there any other gateways that I could use so that I could solve this problem, or am I just out of luck at this point?

 

I'm afraid not. All clients connected to the network are switched onto a VLAN (except CSH). This allows ITS to move subnets around without changing physical connections. It is possible to move your jack onto a different VLAN and change your IP registration, but it requires access that you do not have.

um... don't clone your MAC. You will run into ARP issues and your MAC might end-up quarantined. There is an option in IPEdit to request additional registrations.

----------

## TBKDan

 :Sad:  So basically you're saying I'm screwed and if I want this to work, I'll have to go back to windows since linux wont handle routing correctly?  :Sad: 

If I request a new registration, I'll lose my old IP and then people who have my server IP saved won't know where it went.  I've been cloning for a while and I haven't had any issues  :Smile: 

----------

## d_m

 *TBKDan wrote:*   

> I tried just one first: ip rule add from MYIP table 5
> 
> I got: RTNETLINK answers: Invalid argument

 

What networking options did you compile your kernel with?

Can you post the "CONFIG_IP_*" portion of you kernel configuration?

----------

## TBKDan

OMG... I love you. lol.  After realizing what you were looking for, I went in, enabled a few options, and recompiled.  Rebooted, tried the ip rule command.. and sat there in amazement for a second that it actually worked.  I had a few problems at first, but after rebooting completely, adding those commands and making sure there was only one default route in route, it works! I can ping from each individual device and it actually transmits on that device!! *tears well up in my eyes* I can't believe it! lol.  Thank you sooooo much  :Very Happy:   Now... one other thing.  Will these rules stay in route cache, or do I have to add them each time the machine boots?  I made a script that works, but I'm not entirely sure of how I would add it to get it booted.  I was thinking something along the lines of put it in init.d then rc-update it to default runlevel... but what happens if it gets run before the cards are brought up? o.O  The main issue is solved though, thank you again, all of you for your help  :Smile: 

----------

## d_m

Well, you could just do something like this:

```
#!/sbin/runscript

depend() {

        need net

}

start() {

        ebegin "Setting up custom routes"

        <your script here>

        eend $?

}

stop() {

        ebegin "Stopping custom routes"

        <script to disable routes, or whatever>

        eend $?

}
```

That's the way Gentoo uses to know not to start the thing until after "net" has successfully started.

----------

## TBKDan

Man... I cannot tell you how much relief that is.  Everything works PERFECTLY.  Seriously, PERFECTLY.  I just added another NIC to my other system and it's working wonders.  Thank you soooooo much  :Very Happy: 

*EDIT* On a side note... does anybody know of any utilities that will show both bandwidth and CPU usage on the same window in a terminal?  I've been using bwmon and top, but it's kind of annoying to have to open two terminals...

----------

