# [SOLVED] QEMU ipv6 traffic blocked

## ipic

TLDR: if you go to my last post you will see that this was caused (in my case) by the lack of settings in /etc/qemu/bridge.conf. /TLDR 

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

At least I think that's what is happening - QEMU has an almost vertical learning curve  :Smile: 

I have a gentoo VM that has been running under Virtualbox for many years. I now wish to migrate it to run under KVM/QEMU. I'm really close - only issue is that ipv6 network traffic appears to being blocked by something.

Details. My libvirt XM for the network parts of the VM is as follows:

```
<interface type="bridge">

  <mac address="08:00:27:93:2d:e5"/>

  <source bridge="br0"/>

  <target dev="vnet0"/>

  <model type="e1000e"/>

  <alias name="net0"/>

  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>

</interface>
```

When the VM is run, this gets translated into the /usr/bin/qemu-system-x86_64 command line as follows:

```
-netdev tap,fd=30,id=hostnet0 \

-device e1000e,netdev=hostnet0,id=net0,mac=08:00:27:93:2d:e5,bus=pci.1,addr=0x0
```

(There is a ton of other stuff on the command line of course, left out to focus on networking.)

On the host (also a gentoo system), the bridge looks like this:

```
ian2 ~ # brctl show

bridge name   bridge id      STP enabled   interfaces

br0      8000.1c6f65d226f5   yes      eth0

                     vnet0

br1      8000.0050b6083341   no      eth1

ian2 ~ # 
```

In the guest VM, the network looks like this:

```
dns2 ~ # ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    link/ether 08:00:27:93:2d:e5 brd ff:ff:ff:ff:ff:ff

    inet 192.168.1.32/24 brd 192.168.1.255 scope global eth0

       valid_lft forever preferred_lft forever

    inet6 2001:8b0:fb5e:0:289e:16ed:3ff:e4fd/64 scope global temporary dynamic 

       valid_lft 600995sec preferred_lft 82197sec

    inet6 2001:8b0:fb5e:0:a00:27ff:fe93:2de5/64 scope global dynamic mngtmpaddr 

       valid_lft forever preferred_lft forever

    inet6 2001:8b0:fb5e::32/64 scope global 

       valid_lft forever preferred_lft forever

    inet6 fe80::a00:27ff:fe93:2de5/64 scope link 

       valid_lft forever preferred_lft forever

3: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000

    link/ipip 0.0.0.0 brd 0.0.0.0

4: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1000

    link/gre 0.0.0.0 brd 0.0.0.0

5: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000

    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff

6: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000

    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff

7: ip_vti0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000

    link/ipip 0.0.0.0 brd 0.0.0.0

8: ip6_vti0@NONE: <NOARP> mtu 1428 qdisc noop state DOWN group default qlen 1000

    link/tunnel6 :: brd ::

9: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000

    link/sit 0.0.0.0 brd 0.0.0.0

10: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000

    link/tunnel6 :: brd ::

dns2 ~ # 
```

From the host, I can ping the guest in both ipv4 and ipv6:

```

ipic@ian2 ~/bin $ ping -4 dns2

PING dns2.pickworth.me.uk (192.168.1.32) 56(84) bytes of data.

64 bytes from dns2.pickworth.me.uk (192.168.1.32): icmp_seq=1 ttl=64 time=0.396 ms

64 bytes from dns2.pickworth.me.uk (192.168.1.32): icmp_seq=2 ttl=64 time=0.715 ms

64 bytes from dns2.pickworth.me.uk (192.168.1.32): icmp_seq=3 ttl=64 time=0.832 ms

^C

--- dns2.pickworth.me.uk ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 11ms

rtt min/avg/max/mdev = 0.396/0.647/0.832/0.186 ms

ipic@ian2 ~/bin $ ping -6 dns2

PING dns2(dns2.pickworth.me.uk (2001:8b0:fb5e::32)) 56 data bytes

64 bytes from dns2.pickworth.me.uk (2001:8b0:fb5e::32): icmp_seq=1 ttl=64 time=0.786 ms

64 bytes from dns2.pickworth.me.uk (2001:8b0:fb5e::32): icmp_seq=2 ttl=64 time=0.742 ms

^C

--- dns2 ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 3ms

rtt min/avg/max/mdev = 0.742/0.764/0.786/0.022 ms

ipic@ian2 ~/bin $
```

However, a connection to ipv6 ssh hangs, whilst an ipv4 sshd connection works:

```

pic@ian2 ~/bin $ ssh -Y -t -4 -p 4899 root@dns2

X11 forwarding request failed on channel 0

dns2 ~ # hostname

dns2

dns2 ~ # logout

Connection to dns2 closed.

ipic@ian2 ~/bin $ ssh -Y -t -6 -p 4899 root@dns2

^C

ipic@ian2 ~/bin $ 
```

I have spent the day trying to find documentation on this, but failed. Only things I can find involve setting up the QEMU shared network, which isn't want I want to do.

I can see that there is a nwfilter rule for ipv4 in virsh:

```
virsh # nwfilter-list

 UUID                                   Name

-----------------------------------------------------------------

 17f8df51-4f61-499e-9d88-fc9177c7b834   allow-arp

 fdb7d1a9-3b1a-43d5-afd3-1515f0228039   allow-dhcp

 88bc7ab0-9006-42da-9561-9d5d2c1410de   allow-dhcp-server

 4a461fb8-1703-4fef-9953-098002f3a8c4   allow-incoming-ipv4

 86d9e00c-1a44-4fc8-926c-2ee54b489713   allow-ipv4

 76630340-576a-43e8-b716-b833d94679c2   clean-traffic

 8094dec2-4f63-45d0-8c8d-b74b7eefec5c   clean-traffic-gateway

 4010010e-be0d-450d-b1e4-e5e5f8cc7076   no-arp-ip-spoofing

 e3e8319a-9336-4427-b179-b505522b84ae   no-arp-mac-spoofing

 4cc1ebfb-2640-48c4-a5f7-5ec4ef7653ef   no-arp-spoofing

 04908ecd-c0b4-4aeb-aebb-75c77471eded   no-ip-multicast

 ad6a1c85-ccf8-4f1f-8bf7-cc830cac54a9   no-ip-spoofing

 98d4c7a9-9e3c-4047-ac6f-182cedcc4aa6   no-mac-broadcast

 1d90c2d9-b02d-4561-91c6-6b209520548c   no-mac-spoofing

 001a8f78-cc2b-4892-8e13-f8b7ef1ada58   no-other-l2-traffic

 213ce201-07f1-4428-9a8e-759dc68b7e59   no-other-rarp-traffic

 1dc26217-d085-41c6-8e30-f9e33f4dba42   qemu-announce-self

 9d956941-7e32-478c-8964-afbb0b899de3   qemu-announce-self-rarp

virsh # 
```

..but I don't know if this is being applied.

I can also see in QEMU documentation that there is a ipv6=on|off option, but it states "If neither is specified both protocols are enabled".

So I am at a bit of a dead end on this. Any help appreciated.

Thanks

(PS: Sorry for half cocked post half way through - pressed "submit" rather than "preview" Doh.)Last edited by ipic on Mon Feb 17, 2020 6:05 pm; edited 1 time in total

----------

## mike155

```
However, a connection to ipv6 ssh hangs, whilst an ipv4 sshd connection works:
```

Does you SSH daemon listen on an IPv6 socket? Or does it listen on a IPv4 socket only? Look at the output of 'netstat -l -n' and check your sshd config file (ListenAddress option).

run 'tcpdump -v -n -i eth0' in your VM and look at the packets. Do you see the expected packets when your run ping -4, ping -6, ssh -4, ssh -6 from your host?

----------

## ipic

I am using the same disk image (it's an LVM partition) under both Virtualbox and QEUM. On Virtualbox it works OK, so I anticipated seeing the same output for netstat, which I did:

```

dns2 ~ # netstat -l -n 

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address           Foreign Address         State      

tcp        0      0 192.168.1.32:53         0.0.0.0:*               LISTEN     

tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN     

tcp        0      0 0.0.0.0:953             0.0.0.0:*               LISTEN     

tcp        0      0 0.0.0.0:4899            0.0.0.0:*               LISTEN     

tcp6       0      0 ::1:53                  :::*                    LISTEN     

tcp6       0      0 2001:8b0:fb5e:0:1c00:53 :::*                    LISTEN     

tcp6       0      0 2001:8b0:fb5e:0:a00::53 :::*                    LISTEN     

tcp6       0      0 2001:8b0:fb5e::32:53    :::*                    LISTEN     

tcp6       0      0 :::4899                 :::*                    LISTEN     

udp        0      0 192.168.1.32:53         0.0.0.0:*                          

udp        0      0 127.0.0.1:53            0.0.0.0:*                          

udp6       0      0 ::1:53                  :::*                               

udp6       0      0 2001:8b0:fb5e:0:1c00:53 :::*                               

udp6       0      0 2001:8b0:fb5e:0:a00::53 :::*                               

udp6       0      0 2001:8b0:fb5e::32:53    :::*                               

Active UNIX domain sockets (only servers)

Proto RefCnt Flags       Type       State         I-Node   Path

unix  2      [ ACC ]     SEQPACKET  LISTENING     4926     /run/udev/control

unix  2      [ ACC ]     STREAM     LISTENING     5760     /var/run/acpid.socket

unix  2      [ ACC ]     STREAM     LISTENING     6617     /run/syslog-ng.ctl

dns2 ~ #

```

The routes look good as well:

```

dns2 ~ # ip route

default via 192.168.1.5 dev eth0 metric 2 

127.0.0.0/8 via 127.0.0.1 dev lo 

192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.32 

dns2 ~ #

dns2 ~ # ip -6 route

2001:8b0:fb5e::/64 dev eth0 proto kernel metric 256 pref medium

fe80::/64 dev eth0 proto kernel metric 256 pref medium

default via 2001:8b0:fb5e::2 dev eth0 metric 2 pref medium

default via fe80::a2e4:cbff:fe26:5d0b dev eth0 proto ra metric 1024 expires 29sec hoplimit 64 pref medium

dns2 ~ # 

```

The sshd config is as follows (snippet with relevant bits):

```

#Port 224

Port 4899

#AddressFamily any

#ListenAddress 0.0.0.0

#ListenAddress ::

```

Commented lines mean they are the defaults, as explained a bit higher up:

```

# The strategy used for options in the default sshd_config shipped with

# OpenSSH is to specify options with their default value where

# possible, but leave them commented.  Uncommented options override the

# default value.

```

During a ping, I can see the following traffic: (leaving out lots of "spurious" bits) 

```

....

18:13:42.526977 IP6 (flowlabel 0xce182, hlim 64, next-header ICMPv6 (58) payload length: 64) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3 > 2001:8b0:fb5e::32: [icmp6 sum ok] ICMP6, echo request, seq 1

18:13:42.527020 IP6 (flowlabel 0x4fa98, hlim 64, next-header ICMPv6 (58) payload length: 64) 2001:8b0:fb5e::32 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3: [icmp6 sum ok] ICMP6, echo reply, seq 1

......

18:13:43.527562 IP6 (flowlabel 0xce182, hlim 64, next-header ICMPv6 (58) payload length: 64) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3 > 2001:8b0:fb5e::32: [icmp6 sum ok] ICMP6, echo request, seq 2

18:13:43.527632 IP6 (flowlabel 0x4fa98, hlim 64, next-header ICMPv6 (58) payload length: 64) 2001:8b0:fb5e::32 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3: [icmp6 sum ok] ICMP6, echo reply, seq 2

.....

18:13:44.554650 IP6 (flowlabel 0xce182, hlim 64, next-header ICMPv6 (58) payload length: 64) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3 > 2001:8b0:fb5e::32: [icmp6 sum ok] ICMP6, echo request, seq 3

18:13:44.554683 IP6 (flowlabel 0x4fa98, hlim 64, next-header ICMPv6 (58) payload length: 64) 2001:8b0:fb5e::32 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3: [icmp6 sum ok] ICMP6, echo reply, seq 3

...

```

I think that means that icmp6 is making the round trip OK.

Getting on to the attempted ipv6 ssh connection I see this:

```

18:14:11.497785 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 96) fe80::a2e4:cbff:fe26:5d0b > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 96

   hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 30s, reachable time 0ms, retrans timer 0ms

     prefix info option (3), length 32 (4): 2001:8b0:fb5e::2/64, Flags [onlink, auto], valid time infinity, pref. time infinity

     rdnss option (25), length 40 (5):  lifetime 20s, addr: 2001:8b0::2020 addr: 2001:8b0::2021

     source link-address option (1), length 8 (1): a0:e4:cb:26:5d:0b

18:14:14.314389 STP 802.1d, Config, Flags [none], bridge-id 8000.1c:6f:65:d2:26:f5.8002, length 43

   message-age 0.00s, max-age 20.00s, hello-time 10.00s, forwarding-delay 2.00s

   root-id 8000.1c:6f:65:d2:26:f5, root-pathcost 0

18:14:14.745695 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 96) fe80::a2e4:cbff:fe26:5d0b > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 96

   hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 30s, reachable time 0ms, retrans timer 0ms

     prefix info option (3), length 32 (4): 2001:8b0:fb5e::2/64, Flags [onlink, auto], valid time infinity, pref. time infinity

     rdnss option (25), length 40 (5):  lifetime 20s, addr: 2001:8b0::2020 addr: 2001:8b0::2021

     source link-address option (1), length 8 (1): a0:e4:cb:26:5d:0b

18:14:15.616542 IP6 (flowlabel 0x5a0e3, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0x14f7 (correct), seq 2924141216, win 64800, options [mss 1440,sackOK,TS val 5904132 ecr 0,nop,wscale 7], length 0

18:14:15.616667 IP6 (flowlabel 0x7d490, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xd997), seq 2800007329, ack 2924141217, win 64260, options [mss 1440,sackOK,TS val 2824091023 ecr 5904132,nop,wscale 7], length 0

18:14:16.618558 IP6 (flowlabel 0x8b1c8, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0x110d (correct), seq 2924141216, win 64800, options [mss 1440,sackOK,TS val 5905134 ecr 0,nop,wscale 7], length 0

18:14:16.618613 IP6 (flowlabel 0xe2fd2, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xd5ad), seq 2800007329, ack 2924141217, win 64260, options [mss 1440,sackOK,TS val 2824092025 ecr 5904132,nop,wscale 7], length 0

18:14:17.621885 IP6 (flowlabel 0xc5943, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xd1c1), seq 2800007329, ack 2924141217, win 64260, options [mss 1440,sackOK,TS val 2824093029 ecr 5904132,nop,wscale 7], length 0

18:14:18.666553 IP6 (flowlabel 0x41ef2, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0x090d (correct), seq 2924141216, win 64800, options [mss 1440,sackOK,TS val 5907182 ecr 0,nop,wscale 7], length 0

18:14:18.666616 IP6 (flowlabel 0x89bfb, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xcdad), seq 2800007329, ack 2924141217, win 64260, options [mss 1440,sackOK,TS val 2824094073 ecr 5904132,nop,wscale 7], length 0

18:14:19.331646 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 96) fe80::a2e4:cbff:fe26:5d0b > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 96

   hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 30s, reachable time 0ms, retrans timer 0ms

     prefix info option (3), length 32 (4): 2001:8b0:fb5e::2/64, Flags [onlink, auto], valid time infinity, pref. time infinity

     rdnss option (25), length 40 (5):  lifetime 20s, addr: 2001:8b0::2020 addr: 2001:8b0::2021

     source link-address option (1), length 8 (1): a0:e4:cb:26:5d:0b

18:14:20.693952 IP6 (flowlabel 0x96918, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xc5c1), seq 2800007329, ack 2924141217, win 64260, options [mss 1440,sackOK,TS val 2824096101 ecr 5904132,nop,wscale 7], length 0

18:14:22.698550 IP6 (flowlabel 0xf2c0d, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0xf94c (correct), seq 2924141216, win 64800, options [mss 1440,sackOK,TS val 5911214 ecr 0,nop,wscale 7], length 0

18:14:22.698619 IP6 (flowlabel 0x1ac16, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xbded), seq 2800007329, ack 2924141217, win 64260, options [mss 1440,sackOK,TS val 2824098105 ecr 5904132,nop,wscale 7], length 0

18:14:24.474280 IP (tos 0x0, ttl 64, id 4366, offset 0, flags [DF], proto UDP (17), length 159)

    192.168.1.11.17500 > 255.255.255.255.17500: UDP, length 131

18:14:24.475146 IP (tos 0x0, ttl 64, id 21757, offset 0, flags [DF], proto UDP (17), length 159)

    192.168.1.11.17500 > 192.168.1.255.17500: UDP, length 131

18:14:24.554552 STP 802.1d, Config, Flags [none], bridge-id 8000.1c:6f:65:d2:26:f5.8002, length 43

   message-age 0.00s, max-age 20.00s, hello-time 10.00s, forwarding-delay 2.00s

   root-id 8000.1c:6f:65:d2:26:f5, root-pathcost 0

18:14:26.709902 IP6 (flowlabel 0x58102, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xae41), seq 2800007329, ack 2924141217, win 64260, options [mss 1440,sackOK,TS val 2824102117 ecr 5904132,nop,wscale 7], length 0

```

I don't know what to make of this. It seems that the incoming connect requests are arriving from 2001:8b0:fb5e:0:f91b:1d04 but I can't make out what 2001:8b0:fb5e::32 is responding with. Its not what the host was expecting, hence a seemingly endless dialogue.

This is reminding me of https://xkcd.com/2259/  :Crying or Very sad: 

My make.conf has ipv6 set, so everything that has that use option is built with it. I'm still struggling with exactly the same disk image doing something different when run under QEMU. 

Stumped  :Sad: 

----------

## mike155

The output shows that the SSH daemon listens on IPv4 as well as on IPv6 (good!):

```
tcp        0      0 0.0.0.0:4899            0.0.0.0:*               LISTEN     

tcp6       0      0 :::4899                 :::*                    LISTEN     
```

The TCP dump shows that the 3 way TCP handshake doesn't work. It shows only the first 2 packets. The third packet (ACK) is missing:

```
18:14:15.616542 IP6 (flowlabel 0x5a0e3, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0x14f7 (correct), seq 2924141216, win 64800, options [mss 1440,sackOK,TS val 5904132 ecr 0,nop,wscale 7], length 0

18:14:15.616667 IP6 (flowlabel 0x7d490, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xd997), seq 2800007329, ack 2924141217, win 64260, options [mss 1440,sackOK,TS val 2824091023 ecr 5904132,nop,wscale 7], length 0
```

There are several possible reasons for this:

the answer packet from the guest to the host (packet 2, SYN/ACK) doesn't reach the host

The third packet from the host to the guest (ACK) doesn't reach the guest

Something is wrong with the contents of the packets

Something is wrong with the SSH software on the host or on the guest.

It's likely that (1) will help us. The IPv6 default route on your guest is: 

```
default via 2001:8b0:fb5e::2 dev eth0 metric 2 pref medium
```

What is '2001:8b0:fb5e::2'? Does it exist and does it act as a router? Does it route the answer packets from the guest to your host (2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3)?Last edited by mike155 on Sun Feb 16, 2020 9:31 pm; edited 1 time in total

----------

## ipic

I did a trace on the guest when it is running under Virtualbox, when I connected to ssh using ipv6:

```

21:11:07.991306 IP6 (flowlabel 0xc51f8, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35250 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0x43ee (correct), seq 3457914931, win 64800, options [mss 1440,sackOK,TS val 16516659 ecr 0,nop,wscale 7], length 0

21:11:07.991376 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:8b0:fb5e::32 > ff02::1:ff4b:b3f3: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3

     source link-address option (1), length 8 (1): 08:00:27:93:2d:e5

21:11:07.991766 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3 > 2001:8b0:fb5e::32: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3, Flags [solicited, override]

     destination link-address option (2), length 8 (1): 1c:6f:65:d2:26:f5

21:11:07.991783 IP6 (flowlabel 0xd8451, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35250: Flags [S.], cksum 0xbdb0 (correct), seq 331762219, ack 3457914932, win 64260, options [mss 1440,sackOK,TS val 3325977624 ecr 16516659,nop,wscale 7], length 0

21:11:07.992163 IP6 (flowlabel 0xc51f8, hlim 64, next-header TCP (6) payload length: 32) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35250 > 2001:8b0:fb5e::32.4899: Flags [.], cksum 0xe571 (correct), ack 1, win 507, options [nop,nop,TS val 16516660 ecr 3325977624], length 0

21:11:07.993087 IP6 (flowlabel 0xc51f8, hlim 64, next-header TCP (6) payload length: 53) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35250 > 2001:8b0:fb5e::32.4899: Flags [P.], cksum 0x24ac (correct), seq 1:22, ack 1, win 507, options [nop,nop,TS val 16516660 ecr 3325977624], length 21

21:11:07.993124 IP6 (flowlabel 0xd8451, hlim 64, next-header TCP (6) payload length: 32) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35250: Flags [.], cksum 0xe55f (correct), ack 22, win 502, options [nop,nop,TS val 3325977626 ecr 16516660], length 0

.....

```

Now some of the trace in the previous post makes a little bit of sense, the key bit being:

```

....2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0x14f7 (correct).....

followed by

.....2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.34270: Flags [S.], cksum 0xfcdf (incorrect -> 0xd997)....

```

A little knowledge is a dangerous thing, but I'll take a stab. It seems that the initial challenge from the host '[S]' is met with a response saying "I don't agree with your checksum".

OK. Not sure I know what to do with that.

----------

## mike155

ipic, forget about the checksum. tcpdump and Wireshark often show invalid checksums, for various reasons.

I added a paragraph to my last post. Please look at it.

EDIT: I was probably wrong. Guest and host are on the same /64 subnet. There should be no router involved.

Please repeat the SSH test and look at traffic on your host, preferably on the bridge. Do you see the second packet (SYN/ACK) on the bridge? Or does it get lost in the VM software between the guest and the host?Last edited by mike155 on Sun Feb 16, 2020 9:51 pm; edited 2 times in total

----------

## ipic

'2001:8b0:fb5e::2' is my internet router, i.e.the gateway to the outside world.

All my machines are configured to use it as a gateway, and it can reach all the machines on the local network.

On the host, for example, the routes are thus:

```

ian2 ~ # ip -6 route 

2001:8b0:fb5e::/64 dev br0 proto kernel metric 256 pref medium

fe80::/64 dev eth0 proto kernel metric 256 pref medium

fe80::/64 dev br0 proto kernel metric 256 pref medium

fe80::/64 dev eth1 proto kernel metric 256 pref medium

fe80::/64 dev br1 proto kernel metric 256 pref medium

default via 2001:8b0:fb5e::2 dev br0 metric 4 pref medium

default via fe80::a2e4:cbff:fe26:5d0b dev br0 proto ra metric 1024 expires 20sec hoplimit 64 pref medium

ian2 ~ # 

```

I think the fact that the icmp6 ping goes back and forth OK shows basic routing is OK?

And it works when running under Virtualbox.

Actually, looking at the two route outputs, they have both picked up the link level address for the router: ''fe80::a2e4:cbff:fe26:5d0b". Is it a better approach to let ipv6 work out the gateway for itself?

----------

## mike155

OK, good. I was probably wrong. Guest and host are on the same /64 subnet. There should be no router involved. My fault. But it's good to know that the router works!

Let's try to find out where the second packet (SYN/ACK) gets lost. Please repeat the SSH test and look at traffic on your host, preferably on the bridge (try all interfaces). Do you see the second packet (SYN/ACK) on the bridge? Or does it get lost in the VM software between the guest and the host?

----------

## ipic

I took tcpdump traces in the guest, on the host vnet0 and on the host br0. In all three I could see the same packets:

Guest eth0:

```

22:23:29.958342 IP6 (flowlabel 0x1ea7d, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35668 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0xfa4e (correct), seq 3977543678, win 64800, options [mss 1440,sackOK,TS val 20858666 ecr 0,nop,wscale 7], length 0

22:23:29.958408 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:8b0:fb5e::32 > ff02::1:ff4b:b3f3: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3

     source link-address option (1), length 8 (1): 08:00:27:93:2d:e5

22:23:29.958708 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3 > 2001:8b0:fb5e::32: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3, Flags [solicited, override]

     destination link-address option (2), length 8 (1): 1c:6f:65:d2:26:f5

22:23:29.958722 IP6 (flowlabel 0x5dd10, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35668: Flags [S.], cksum 0xfcdf (incorrect -> 0x74af), seq 3614469519, ack 3977543679, win 64260, options [mss 1440,sackOK,TS val 578756651 ecr 20858666,nop,wscale 7], length 0

```

Host vnet0:

```

22:23:30.105910 IP6 (flowlabel 0x1ea7d, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35668 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0xfa4e (correct), seq 3977543678, win 64800, options [mss 1440,sackOK,TS val 20858666 ecr 0,nop,wscale 7], length 0

22:23:30.106308 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:8b0:fb5e::32 > ff02::1:ff4b:b3f3: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3

     source link-address option (1), length 8 (1): 08:00:27:93:2d:e5

22:23:30.106371 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3 > 2001:8b0:fb5e::32: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3, Flags [solicited, override]

     destination link-address option (2), length 8 (1): 1c:6f:65:d2:26:f5

22:23:30.106626 IP6 (flowlabel 0x5dd10, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35668: Flags [S.], cksum 0x7607 (incorrect -> 0x74af), seq 3614469519, ack 3977543679, win 64260, options [mss 1440,sackOK,TS val 578756651 ecr 20858666,nop,wscale 7], length 0

```

Host br0:

```

22:23:30.105901 IP6 (flowlabel 0x1ea7d, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35668 > 2001:8b0:fb5e::32.4899: Flags [S], cksum 0xfcdf (incorrect -> 0xfa4e), seq 3977543678, win 64800, options [mss 1440,sackOK,TS val 20858666 ecr 0,nop,wscale 7], length 0

22:23:30.106308 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:8b0:fb5e::32 > ff02::1:ff4b:b3f3: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3

     source link-address option (1), length 8 (1): 08:00:27:93:2d:e5

22:23:30.106367 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3 > 2001:8b0:fb5e::32: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3, Flags [solicited, override]

     destination link-address option (2), length 8 (1): 1c:6f:65:d2:26:f5

22:23:30.106626 IP6 (flowlabel 0x5dd10, hlim 64, next-header TCP (6) payload length: 40) 2001:8b0:fb5e::32.4899 > 2001:8b0:fb5e:0:f91b:1d04:ea4b:b3f3.35668: Flags [S.], cksum 0x7607 (incorrect -> 0x74af), seq 3614469519, ack 3977543679, win 64260, options [mss 1440,sackOK,TS val 578756651 ecr 20858666,nop,wscale 7], length 0

```

I have the full files - there is a lot of traffic on br0. above is edited down to show the trace through the interfaces.

Following this, the pattern is repeated over and over as before.

What this all has done (and thank you very much for helping me with this), is make me see that the networking approach between QEMU and Virtualbox is different.

 - In Virtualbox the guest connects direct to the bridge (br0 in my case) which is placed in promiscuous mode.

 - In QEMU the guest inserts a virtual NIC into the bridge.

So, it is possible that something in ipv6 land doesn't like this setup. The frustrating thing is that it works with ipv4  :Sad: 

I am going to try some different options on the guest NIC setup, to see if any make a difference.

----------

## mike155

 *Quote:*   

> I have the full files - there is a lot of traffic on br0

 

You could add 'port 4899' to the tcpdump statement

```
tcpdump -v -i <interface> -n port 4899
```

This will show only TCP and UDP packets to and from port 4899. Beware that it won't show ICMP (ping) packets. Look at 'man tcdump' and 'man pcap-filters' for details.

----------

## mike155

 *Quote:*   

> In all three I could see the same packets: 

 So the SYN/ACK packets reach the bridge on the host. Great! 

Please show us the output of 'ip addr' on your host.

I would like to see the MAC addresses. Please run the SSH test again. Start tcpdump on your host with option '-e':

```
tcpdump -v -e -i br0 -n port 4899
```

----------

## ipic

First, I want to say thank you, mike155, for all your help in this. By guiding me through the fault finding, you lead me to the root cause. Reminded me of my old days in  operations!

The problem was permissions in the end. I had started to sniff this at the end of my last post, since besides the different approaches to networking, virtualbox and qemu ran under different users.

Completely by accident I stumbled across this file: /etc/qemu/bridge.conf

Since I am using libvirtd as the manager, it didn't occur to me to look under a qemu folder for settiings (for example, the conf file is here: /etc/libvirt/qemu.conf).

So full marks to libvert/qemu for being inconsistent and obtuse!

Inside the bridge.conf file is this (after I edited it):

```

ian2 ~ # cat /etc/qemu/bridge.conf

# This should have the following permissions: root:qemu 0640

# IRP Changes #

allow br0

allow br1

allow virbr0

# allow br0

# Uncommenting the above would allow users in the 'qemu' group

# to add devices to 'br0'

# allow virbr0

# Uncommenting the above would allow users in the 'qemu' group

# to add devices to 'virbr0'

# include /etc/qemu/bob.conf

# Uncommenting the above would allow users in the 'bob' group

# to have permissions defined in it, iff it has the following

# permissions: root:bob 0640

ian2 ~ #

```

With the permissions set as described, and the entries uncommitted (well added), I started up the VM, and the ipv6 ssh session just connected straight up - like it should.

phew.

I suppose a good question would be - why only ipv6 with the problem, surely permissions would stop ipv4 as well??? At this stage I can't be arsed.

And again many thanks again for the patient help.

----------

