# [SOLVED]sanity check: KVM - host OS with pub IP, guests priv

## cach0rr0

'lo gents

Just need to think out loud a bit and hopefully have someone confirm my thinking is correct, or, that it's broken

Gentoo box located in a datacenter, hooked up directly to one of their switches, with a public IP on eth0

Goal is to give br0 the public IP, add eth0 and a bunch of tap interfaces (tap0,tap1,etc) to the bridge as well. 

The guests will all have ONLY a private IP, likely in the 172.16.0.0/12 range. 

Actually as I type this, I've just set up the bridge, br0, and have eth0 and tap0,1,2,3 added to it. 

To meet the desired goal of having a private network for the guests - one wherein they can communicate with each other - what are the remaining moving pieces?

Do I need to assign a second, private address to br0? Or to eth0?

Or do I simply need to configure the guests with the aforementioned (static) private IP's, and they should automagically be able to talk to each other assuming i have ip_forward enabled in sysctl, and have iptables set to accept FORWARD?

Now comes the sharing time - current /etc/conf.d/net with my public IP obfuscated using ${PUBLICIP}, broadcast using ${BROADCAST} and so forth (should be obvious)

```

bridge_add_eth0="br0"

bridge_add_tap0="br0"

bridge_add_tap1="br0"

bridge_add_tap2="br0"

bridge_add_tap3="br0"

bridge_add_tap4="br0"

brctl_br0="setfd 0

stp off

sethello 0"

rc_net_br0_need="net.eth0"

config_br0="${PUBLICIP} netmask 255.255.255.224 broadcast ${BROADCAST}"

routes_br0="default via ${GATEWAY}"

dns_servers_br0="8.8.8.8 8.8.4.4"

tuntap_tap0="tap"

config_tap0="null"

mac_tap0="52:54:00:12:34:56"

tuntap_tap1="tap"

config_tap1="null"

mac_tap1="52:54:00:12:34:57"

tuntap_tap2="tap"

config_tap2="null"

mac_tap2="52:54:00:12:34:58"

tuntap_tap3="tap"

config_tap3="null"

mac_tap3="52:54:00:12:34:59"

config_eth0="null"

```

My current conundrum, I've done this before where the Host OS has one public IP, and the guests all have a public IP as well

I've also done this where both Host OS and guests have private IP's, and I've just done 1:1 NAT on my router to functionall give them all public IP's

But this sort of mix? I'm certain it's doable, I just need to know the general direction to go. Specifically on the host OS (Gentoo), the guests (CentOS) *should* be a simple matter of just setting a static IP, although, having said that I'm not sure what to set the gateway to be within the guest machines. 

So, I'll use my own IP at the house for an example of what I'm driving at:

-host machine connected directly to cable modem, has IP 75.148.243.89, gateway 75.148.243.94

-guest machines running on host machine, need to have 172.16.1.5, 172.16.1.10, 172.16.1.15

I'm assuming I would assign 75.148.243.89/29 to br0, add eth0 to that bridge

Then add tap0, tap1, tap2, to that same bridge, with a mostly "null" configuration for each interface in /etc/conf.d/net on the host side. 

And that's more or less where my brain is hitting a stumbling block. Is that all I need on the host side? If yes, do I have to do anything funky within the guest OS as far as its routing/gateway go? If not, then what am I missing on the host side?

I've just stopped net.eth0 and started up net.br0 and that's all kosher, I can still access the box (always a heart attack doing such a thing remotely!) and get out to the intertubes. 

Just need to get these suckers (the guests) network access, then go through my hoops to do the port forwarding on the iptables side, and I'm good. 

But yeah, I've hit a wall - thanks in advance for any shoves!

----------

## salahx

You don't want to add eth0 and the tap driver to the same bridge. You want a separate eth0 (which is configured normally), and then all the tap devices on their own bridge. Assign the bridge a private IP (like 172.16.0.1). For dynamic configuration of the hosts, use something like dnsmasq, but make sure it only binds to tap-only bridge, not to eth0. If you want the virtual machine to have Internet connectivity, enable ip forwarding and add iptables rules to masquerade.

If you use libvirt, it can do this all for you with the virt-network USE flag.

----------

## cach0rr0

 *salahx wrote:*   

> You don't want to add eth0 and the tap driver to the same bridge. You want a separate eth0 (which is configured normally), and then all the tap devices on their own bridge. Assign the bridge a private IP (like 172.16.0.1). For dynamic configuration of the hosts, use something like dnsmasq, but make sure it only binds to tap-only bridge, not to eth0. If you want the virtual machine to have Internet connectivity, enable ip forwarding and add iptables rules to masquerade.
> 
> If you use libvirt, it can do this all for you with the virt-network USE flag.

 

cheers! what you've said above is precisely the route I'm going. The old unofficial wiki outlines doing exactly this

we'll see if this works. It makes sense

thanks again

EDIT

I should add, this is the new /etc/conf.d/net for any on google that hits this and is pissed that a more detailed answer other than "thanks" is posted. 

```

bridge_add_tap0="br0"

bridge_add_tap1="br0"

bridge_add_tap2="br0"

bridge_add_tap3="br0"

bridge_add_tap4="br0"

brctl_br0="setfd 0

stp off

sethello 10"

#rc_net_br0_need="net.eth0 net.tap0 net.tap1 net.tap2 net.tap3"

rc_net_br0_need="net.tap0 net.tap1 net.tap2 net.tap3 net.tap4"

config_eth0="${PUBLICIP netmask 255.255.255.224 broadcast ${BROADCAST}"

routes_eth0="default via ${PUBGATEWAY}"

dns_servers_eth0="8.8.8.8 8.8.4.4"

config_br0="10.0.1.2/24"

tuntap_tap0="tap"

config_tap0="null"

mac_tap0="52:54:00:12:34:56"

tuntap_tap1="tap"

config_tap1="null"

mac_tap1="52:54:00:12:34:57"

tuntap_tap2="tap"

config_tap2="null"

mac_tap2="52:54:00:12:34:58"

tuntap_tap3="tap"

config_tap3="null"

mac_tap3="52:54:00:12:34:59"

tuntap_tap4="tap"

config_tap4="null"

mac_tap4="52:54:00:12:34:59"

```

i dont yet have network connectivity, but I'm still investigating, and this seems to be *closer*

aint the masquerade rule, already did that one - again, we'll see.

EDIT 2

so this actually DOES work now. Somehow. Interestingly, I shut down br0,tap{0,1,2,3,4} because i wanted to dick around with a different way of doing things. 

out of desperation, i start br0 back up about 5 minutes later, start up the guest (which is configured with a static IP, 10.0.1.5/24, gateway 10.0.1.2) and it works

so my key piece *may* simply be that br0 and eth0 try to start up at incompatible times. 

Hope that helps some other googler.

----------

## Mad Merlin

FYI, the MAC address on the tap interfaces cannot be the same as the MAC address for the VM's actual interface (this is a very common error). Also, the MAC addresses you picked for your taps conflict with the default MAC address in KVM.

Just delete the mac_tap* lines from your network config, the kernel will generate random ones for you.

----------

## cach0rr0

 *Mad Merlin wrote:*   

> FYI, the MAC address on the tap interfaces cannot be the same as the MAC address for the VM's actual interface (this is a very common error). Also, the MAC addresses you picked for your taps conflict with the default MAC address in KVM.

 

yip. I did precisely that way back when, invoked kvm with the same mac, b0rked things nicely. 

didnt catch me this go-round though!

 *Mad Merlin wrote:*   

> 
> 
> Just delete the mac_tap* lines from your network config, the kernel will generate random ones for you.

 

Down the road I'm going to have a need for predictable mac addresses for the tap nodes on this box (yes, on the host side) - no immediate need, but I'll have it in the not terribly distant future. 

Got 'er all working as of Edit 2 - seems to just be the timing I guess.

----------

