# one way communication with kvm+macvlan

## _______0

hi,

host macvlan

guest macvtap

I can ssh from guest to host but no the other way around.

Anyone know the trick for this?

thanks.

----------

## John R. Graham

Did you set up both of these (virtual?) machines?

- John

----------

## _______0

macvlan (host is not virtual)

mavtap (guest is vm)

I don't understand your question. But the answer is of course it's set up. How else would I be able to ssh from guest to host?

To clarify my confusion, what exactly and which part do you mean by 'set up'?

----------

## John R. Graham

Sorry; wn a little bit more detail, what I meant was, did you install Gentoo on both of these machines and do you have control over both of them?

- John

----------

## papahuhn

 *_______0 wrote:*   

> hi,
> 
> host macvlan
> 
> guest macvtap
> ...

 

If you have established an ssh session, then this is already two way communication, and macvlan/macvtap is irrelevant. How about iptables?

----------

## _______0

[quote="papahuhn"] *_______0 wrote:*   

> ... How about iptables?

 

I am all ears.

There was a mistake on my post. The connection was established over a router so the bridge mod in macvlan wasn't doing what's supposed to do.

Is it possible to have two nics have a conversation with each other on the same computer with iptables? What part gets involved in this, routing, iptables?

To be honest the popular bridging method I don't see very elegant as it kills flexibility.

I've been reading some documentation and this huge thread, http://markmail.org/message/vudd7mvq3c5ze3j6 , but the they are quite incloncusive.

I need VM's to be able to talk to each other as well to host without a router being involved. ssh'ing is better than bloated desktop, sometimes the kvm hangs and my mouse and keyboard get stuck inside and I have to reboot!!!

macvlan has three modes one which is bridge which is independent from external router/switch.

One guy claims it works, http://lists.gnu.org/archive/html/qemu-devel/2011-02/msg02687.html ,

 *Quote:*   

> Right, but if I set IP(eth0) == IP(macvlan0), I'm able to communicate
> 
> between macvlan0 and mactapX, thus between guest and host. Just
> 
> re-checked here, still works (after resolving the usual MAC address mess
> ...

 

Actually even with traditional method using brctl there's gotta be an ip in common where both interface are able to talk. The routing part perhaps?

I think I am not doing this properly, just assigning an ip to macvlans and setting them in bridge mode is not enough? Thinking about it makes sense an extra framework for communication.

thanks

----------

## papahuhn

 *Quote:*   

> There was a mistake on my post. The connection was established over a router so the bridge mod in macvlan wasn't doing what's supposed to do.

 

Ok, this changes things. I just googled that your problem is "defined behaviour": http://wiki.libvirt.org/page/Guest_can_reach_outside_network,_but_can't_reach_host_(macvtap)

I can't help you with this by other means than googling, sorry.

 *Quote:*   

> Actually even with traditional method using brctl there's gotta be an ip in common where both interface are able to talk. The routing part perhaps?

 

With a traditional bridge, the host needs a route on that bridge towards the networks that are configured inside the VMs - typically by assigning br0 an IP address from those networks.

----------

## _______0

 *papahuhn wrote:*   

> With a traditional bridge, the host needs a route on that bridge towards the networks that are configured inside the VMs - typically by assigning br0 an IP address from those networks.

 

can you show an example or some steps for this?

----------

## papahuhn

```
root@host:~ ip a sh virbr1 | grep "inet "

    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr1

root@host:~ brctl show

bridge name   bridge id      STP enabled   interfaces

virbr0      8000.047d7b9219be   no      eth0

virbr1      8000.fe54008b8897   no      vnet0

root@host:~ # vnet0 is the VMs host-side interface

root@host:~ # in this case, I don't bridge the VM with my physical network

root@host:~ # but with potential other VMs (currently not running)

root@host:~ route -n | grep virbr1

192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr1

root@host:~ ssh 192.168.122.124

root@192.168.122.124's password: 

Last login: Wed Jun 19 18:39:04 2013

[root@centos64 ~]# ip a sh eth0 | grep "inet "

    inet 192.168.122.124/24 brd 255.255.255.255 scope global eth0

[root@centos64 ~]# route -n

Kernel IP Routentabelle

Ziel            Router          Genmask         Flags Metric Ref    Use Iface

192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0

169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 eth0

0.0.0.0         192.168.122.1   0.0.0.0         UG    0      0        0 eth0

[root@centos64 ~]# 
```

----------

## _______0

what about this claim that I found on two places?

 *Quote:*   

> Right, but if I set IP(eth0) == IP(macvlan0), I'm able to communicate
> 
> between macvlan0 and mactapX, thus between guest and host. Just
> 
> re-checked here, still works (after resolving the usual MAC address mess
> ...

 

This implies something like this:

eth0           192.168.1.2

macvlan0   192.168.1.2

Is this correct? Even then, a place mentioned eth0 needed to have no routes, manually flushed, and guests have some weird thing about host's ip as route??

/me confused as hell.

----------

## _______0

anyone?

thanks

----------

## papahuhn

Nice, it works as described. You don't even need the IP on eth0, neither to set anything special within the VM.

----------

## _______0

 *papahuhn wrote:*   

> Nice, it works as described. You don't even need the IP on eth0, neither to set anything special within the VM.

 

wait... Are you being sarcastic or something? What's working and how?? Plz tell me more!!

----------

## papahuhn

I'm not sarcastic. I just followed the tip from your link.

- Create a macvlan0@eth0

- Move IP and routes from eth0 to macvlan0

- Use macvtap@eth0 in bridge mode for your VMs (I use virt-manager for configuration).

----------

## _______0

 *papahuhn wrote:*   

> I'm not sarcastic. I just followed the tip from your link.
> 
> - Move IP and routes from eth0 to macvlan0
> 
> 

 

:"((( works for everyone except me.

Can you show the steps for that?

Also, it's important that this works unplugged. Try ssh both ways host and guest without the cable and tell me if it works.

----------

## papahuhn

 *_______0 wrote:*   

> Also, it's important that this works unplugged. Try ssh both ways host and guest without the cable and tell me if it works.

 

Right, that doesn't work for me either. The macvlan and macvtap go into a LOWERLAYERDOWN state. While VMs still can communicate with each other, they cannot communicate with the host.

----------

## _______0

 *papahuhn wrote:*   

>  *_______0 wrote:*   Also, it's important that this works unplugged. Try ssh both ways host and guest without the cable and tell me if it works. 
> 
> Right, that doesn't work for me either. The macvlan and macvtap go into a LOWERLAYERDOWN state. While VMs still can communicate with each other, they cannot communicate with the host.

 

hence the suggestion of:

ip eth0 = macvlan

???

dunno...

----------

## papahuhn

The IPs don't matter here. The problem is with the link state. When the cable is unplugged, eth0's and macvlan's internal queueing discipline changes to noop [1].

In net/core/dev.c:dev_queue_xmit() a frame is then dropped by the noop qdisc, while macvlan transmission via macvlan_start_xmit() would be performed after.

So, the problem is not with receiving a frame from a VM but sending the response back to it on a practically DOWN interface.

Plugged:

```

 2)               |        dev_queue_xmit() {

 2)               |          local_bh_disable() {

 2)   0.137 us    |            __local_bh_disable();

 2)   0.660 us    |          }

 2)   0.121 us    |          netdev_pick_tx();

 2)               |          _raw_spin_lock() {

 2)   0.085 us    |            add_preempt_count();

 2)   0.566 us    |          }

 2)               |          sch_direct_xmit() {

 2)               |            _raw_spin_unlock() {

 2)   0.077 us    |              sub_preempt_count();

 2)   0.612 us    |            }

 2)               |            dev_hard_start_xmit() {

 2)   0.155 us    |              netif_skb_features();

 2)               |              macvlan_start_xmit() {

```

Unplugged:

```

 3)               |        dev_queue_xmit() {

 3)               |          local_bh_disable() {

 3)   0.120 us    |            __local_bh_disable();

 3)   0.638 us    |          }

 3)   0.118 us    |          netdev_pick_tx();

 3)               |          _raw_spin_lock() {

 3)   0.063 us    |            add_preempt_count();

 3)   0.591 us    |          }

 3)               |          noop_enqueue() {

 3)               |            kfree_skb() {

 3)               |              __kfree_skb() {

 3)               |                skb_release_head_state() {

 3)               |                  sock_wfree() {

 3)               |                    sock_def_write_space() {

 3)   0.058 us    |                      __rcu_read_lock();

 3)   0.060 us    |                      __rcu_read_unlock();

 3)   1.248 us    |                    }

 3)   1.944 us    |                  }

 3)   2.540 us    |                }

 3)               |                skb_release_data() {

 3)               |                  skb_free_head() {

 3)   0.190 us    |                    kfree();

 3)   0.679 us    |                  }

 3)   1.235 us    |                }

 3)   0.111 us    |                kmem_cache_free();

 3)   5.570 us    |              }

 3)   6.101 us    |            }

 3)   6.633 us    |          }

```

[1] http://ace-host.stuart.id.au/russell/files/tc/doc/sch_noop.txt

----------

## _______0

mm...   fuuu???

What about deleting OR instructing in net/core/dev.c NOT to change states?

Otherwise the next most simple setup would be to use VDE? To be honest the traditional bridging method somehow doesn't look very aesthetic.

thanks

----------

## papahuhn

There is no OR instruction, and the state-update code is somewhere else. I think you should ask the netdev mailing list.

----------

