# Virtual NIC for Virtualbox, InfiniBand on host [SOLVED]

## Zucca

I need to run a virtual server on VirtualBox as a part of a course I'm currently going.

Then virtual server needs:

ip of 10.20.85.20

gateway 10.20.0.1

subnet mask 255.255.0.0

Everything would be easy just by giving another ip to ib0 (IP-over-InfiniBand interface on the host) and then choosing bridged connection from VirtualBox settings. The problem is that VirtualBox doesn't have support for interfaces of type "Infiniband" (I assume). And I would like to avoid (physically) routing, yet another cable to my PC.

So now I'd (probably) need to create some kind of VLAN, maybe, for the 10.20.0.0 network? Then to get out of the VLAN to the internet the gateway address should probably be the address of host computer's ib0 ip address?

Am I in the right tracks? This is more advanced networking than what I'm used to. I'm learning here.

Please, do point out if there's some other simple way to get this working. The server addresses must not change.

----------

## NeddySeagoon

Zucca,

Am I missing something here?

Create a new bridge with no hardware.

On the bare metal, route 10.20.85.20 to the new bridge.

In Virtualbox, connect the network card to the new bridge.

You can route 10.20/16 to the bridge if you want and assign 10.20.85.20/16  statically to the VM.

----------

## szatox

Yes, another bridge (without any physical interfaces) seems to be the easiest way to go.

You will need IP forwarding enabled in kernel and (most likely) masquerade in iptables, just like any LAN router.

Unfortunately IPoIB does not support VLANS. I'm looking for a better way to handle this. E.g. Etherner over IB may or may not  be a viable option. Or - maybe - eIPoIB.

They all have some drawbacks compared to each other, but each of them should be a workable solution.

----------

## Zucca

 *NeddySeagoon wrote:*   

> Zucca,
> 
> Am I missing something here?

  Nothing really. I'm just missing my skills. :) I've totally forgotten how to manage ip networking. :\ This is what happens when you "quicklearn" something just for one-time purpose.

Luckily I get to learn more ip(v4) networking on the course.

 *NeddySeagoon wrote:*   

> Create a new bridge with no hardware.
> 
> On the bare metal, route 10.20.85.20 to the new bridge.
> 
> In Virtualbox, connect the network card to the new bridge.
> ...

 I tried this last night, but only got to the point where I could ping my host machine's ip. I tried to route the traffic, but with no success. Now (next day) after reading szatox's post I remembered that I need to "sysctl -w net.ipv4.ip_forward=1".

Now I'm not sure if I need to set some nftables rules too or will just normal routing configuration do it.

Quick details:Host PC network 10.0.11.0/28 (InfiniBand)host ip 10.0.11.2 (dev ib0)gateway 10.0.11.1Guest PC network 10.20.0.0/16guest ip 10.20.85.20another guest in the future 10.20.85.25gateway 10.20.0.1

@szatox: You could circumvent VLAN incapabilities etc., which aren't present (the incapabilities I mean) on normal Ethernet by switching your ib interfaces to Ethernet mode, but I guess by doing that you'll lose RDMA features of those cards. :(

----------

## szatox

Zucca, as I mentioned you will probably need NAT like here:

```
-A POSTROUTING -s 10.20.0.0/16 -o ib0 -j MASQUERADE
```

 (or -i vmbridge0 -o ib0)

 *Quote:*   

>  @szatox: You could circumvent VLAN incapabilities etc., which aren't present (the incapabilities I mean) on normal Ethernet by switching your ib interfaces to Ethernet mode, but I guess by doing that you'll lose RDMA features of those cards. 

 Of course I could. I suppose you could do that too, since direct support for ethernet seems to be a common trait for Mellanox HCA, but there are drawbacks to this approach too. Losing RDMA is one of them, degrading wire speed to 10G is another, then comes the advertised reliability and congestion control/path switching (I haven't stressed either of those networks nearly enough to draw any conclusions on the latter aspects).

----------

## NeddySeagoon

Zucca,

As 10.20.0.0/16 is outside the Host PC network 10.0.11.0/28

How does the gateway at 10.20.0.1/16 get anywhere?

I have a horrible hack for that but it doesn't work with iproute2, so I have an interest in the answer.

As 10.20.0.0/16 is not routable over the internet, you will need NAT somewhere to get to the outside world.

----------

## Zucca

Thanks guys. It works now.

I needed compile all the missing nft modules, since this PC doesn't normally do any special networking things.

```
brvbox0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 10.20.0.1  netmask 255.255.0.0  broadcast 0.0.0.0

        inet6 fe80::6c8d:36ff:fe29:13b3  prefixlen 64  scopeid 0x20<link>

        ether 6e:8d:36:29:13:b3  txqueuelen 1000  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 122  bytes 9804 (9.5 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
```

```
Kernel IP routing table

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

default         10.0.11.1       0.0.0.0         UG    6      0        0 ib0

10.0.11.0       0.0.0.0         255.255.255.240 U     0      0        0 ib0

10.20.0.0       0.0.0.0         255.255.0.0     U     0      0        0 brvbox0
```

```
table ip nat {

        chain prerouting {

                type nat hook prerouting priority 0; policy accept;

        }

        chain postrouting {

                type nat hook postrouting priority 100; policy accept;

                oifname "ib0" masquerade

        }

}
```

 ... and of course ipv4 forwarding is enabled. I may need to create some init script or similar to automate all these when needed.

Oh. Btw, if you see anything wrong here, please say so. I may need to specify the broadcast address later...

----------

