# [SOLVED] linux bridge not forwarding traffic outside

## viacheslavg

I'm trying to setup network namespace which later will be used by dedicated user. The idea is that this user will play with network configurations (e.g. setup VPN) but this changes should not affect network on host (i.e. "default" network namespace).

What I did so far:

1. Setup network namespace along with veth pair:

```

ip netns add vnet0

ip link add veth0 type veth peer name eth0 netns vnet0

```

2. Configure interfaces and routes in namespace:

```

ip netns exec vnet0 ip link set lo up

ip netns exec vnet0 ip addr add 192.168.8.2/24 dev eth0

ip netns exec vnet0 ip link set eth0 up

ip netns exec vnet0 ip route add default via 192.168.8.1 dev eth0

```

3. Create bridge in the host and add veth there:

```

ip link add name br0 type bridge

ip addr add 192.168.8.1/24 brd 192.168.8.255 dev br0

ip link set veth0 master br0

ip link set br0 up

```

As result I have following on host:

```

# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000

    link/ether 2e:9e:3f:99:fc:c9 brd ff:ff:ff:ff:ff:ff

3: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000

    link/sit 0.0.0.0 brd 0.0.0.0

4: enp2s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000

    link/ether e4:7f:b2:17:26:49 brd ff:ff:ff:ff:ff:ff

5: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 80:19:34:c0:e6:8a brd ff:ff:ff:ff:ff:ff

    inet 192.168.1.120/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp3s0

       valid_lft 41981sec preferred_lft 41981sec

    inet6 fd63:61e4:95ff::f97/128 scope global dynamic noprefixroute 

       valid_lft 85184sec preferred_lft 85184sec

    inet6 fd63:61e4:95ff:0:1cdd:1f29:e279:ca55/64 scope global dynamic noprefixroute 

       valid_lft 5984sec preferred_lft 584sec

    inet6 fe80::5957:17e5:842f:3cd8/64 scope link noprefixroute 

       valid_lft forever preferred_lft forever

8: veth0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000

    link/ether ae:de:a7:6d:0d:dd brd ff:ff:ff:ff:ff:ff link-netns vnet0

    inet6 fe80::d04:c570:6e2d:fdc3/64 scope link noprefixroute 

       valid_lft forever preferred_lft forever

9: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether ae:de:a7:6d:0d:dd brd ff:ff:ff:ff:ff:ff

    inet 192.168.8.1/24 brd 192.168.8.255 scope global br0

       valid_lft forever preferred_lft forever

    inet6 fe80::acde:a7ff:fe6d:ddd/64 scope link 

       valid_lft forever preferred_lft forever

# ip route

default via 192.168.1.1 dev wlp3s0 proto dhcp metric 600 

192.168.1.0/24 dev wlp3s0 proto kernel scope link src 192.168.1.120 metric 600 

192.168.8.0/24 dev br0 proto kernel scope link src 192.168.8.1 

```

and following inside network namespace:

```

# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000

    link/sit 0.0.0.0 brd 0.0.0.0

3: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

    link/ether 3a:d2:8c:56:cc:6b brd ff:ff:ff:ff:ff:ff link-netnsid 0

    inet 192.168.8.2/24 scope global eth0

       valid_lft forever preferred_lft forever

    inet6 fe80::38d2:8cff:fe56:cc6b/64 scope link 

       valid_lft forever preferred_lft forever

# ip route

default via 192.168.8.1 dev eth0 

192.168.8.0/24 dev eth0 proto kernel scope link src 192.168.8.2 

```

My problem is that I'm not able to reach any IP from within this namespace outside of host.

e.g.:

```

# ping 8.8.8.8

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

^C

--- 8.8.8.8 ping statistics ---

3 packets transmitted, 0 received, 100% packet loss, time 2082ms

```

What I'm doing wrong? Could it be that packets are somehow filtered in bridge? I have no special rules in iptables:

```

# iptables -t filter -L

Chain INPUT (policy ACCEPT)

target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)

target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)

target     prot opt source               destination         

# iptables -t nat -L

Chain PREROUTING (policy ACCEPT)

target     prot opt source               destination         

Chain INPUT (policy ACCEPT)

target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)

target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)

target     prot opt source               destination         

# ebtables -t filter -L

Bridge table: filter

Bridge chain: INPUT, entries: 0, policy: ACCEPT

Bridge chain: FORWARD, entries: 0, policy: ACCEPT

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT

# ebtables -t nat -L

Bridge table: nat

Bridge chain: PREROUTING, entries: 0, policy: ACCEPT

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT

Bridge chain: POSTROUTING, entries: 0, policy: ACCEPT

```

bridge also looks good:

```

# brctl showstp br0

br0

 bridge id      8000.aedea76d0ddd

 designated root   8000.aedea76d0ddd

 root port         0         path cost         0

 max age        20.00         bridge max age        20.00

 hello time         2.00         bridge hello time      2.00

 forward delay        15.00         bridge forward delay     15.00

 ageing time       300.00

 hello timer         0.00         tcn timer         0.00

 topology change timer      0.00         gc timer       201.75

 flags         

veth0 (1)

 port id      8001         state           forwarding

 designated root   8000.aedea76d0ddd   path cost         2

 designated bridge   8000.aedea76d0ddd   message age timer      0.00

 designated port   8001         forward delay timer      0.00

 designated cost      0         hold timer         0.00

 flags         

```

As an additional observation: if i start docker, which will create docker0 bridge and bunch of rules in iptables and add my veth0 interface to docker0 bridge then everything works fine. I have no glue why docker0 bridge works fine but mine not.

Any idea?Last edited by viacheslavg on Fri Sep 03, 2021 11:19 am; edited 1 time in total

----------

## szatox

Depending on your network, you should either:

* enable ip forwarding (via sysctl) and add MASQUERADE rule to POSTROUTING in nat table - most common.

* bridge your veth with a physical devices (add your wifi to the bridge and move wifi's IP there too).

* (rare - if you don't know this is your use case, it is not) just enable ip forwarding.

----------

## viacheslavg

Thanks for helping.

As I mentioned earlier everything works out of the box if i move my veth to the docker0 bridge (created by docker). I don't need to put wifi to the bridge enable IP forwarding etc.

I replicated all the iptable rules (which docker creates for docker0) for my bridge (including MASQUERADE you mention). so now I have:

```
# iptables -t nat -v -L

Chain PREROUTING (policy ACCEPT 23 packets, 3192 bytes)

 pkts bytes target     prot opt in     out     source               destination         

   26  3130 DOCKER     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 5 packets, 1432 bytes)

 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 424 packets, 28043 bytes)

 pkts bytes target     prot opt in     out     source               destination         

    0     0 DOCKER     all  --  any    any     anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 424 packets, 28043 bytes)

 pkts bytes target     prot opt in     out     source               destination         

    3   252 MASQUERADE  all  --  any    !docker0  192.168.64.0/24      anywhere            

    0     0 MASQUERADE  all  --  any    !br-68c3e5c395af  192.168.70.0/24      anywhere            

    0     0 MASQUERADE  all  --  any    !br0    192.168.8.0/24       anywhere            

Chain DOCKER (2 references)

 pkts bytes target     prot opt in     out     source               destination         

    0     0 RETURN     all  --  docker0 any     anywhere             anywhere            

    0     0 RETURN     all  --  br-68c3e5c395af any     anywhere             anywhere            

    0     0 RETURN     all  --  br0    any     anywhere             anywhere            

# iptables -t filter -v -L

Chain INPUT (policy ACCEPT 1152 packets, 459K bytes)

 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy DROP 4 packets, 336 bytes)

 pkts bytes target     prot opt in     out     source               destination         

   80  6664 DOCKER-USER  all  --  any    any     anywhere             anywhere            

   80  6664 DOCKER-ISOLATION-STAGE-1  all  --  any    any     anywhere             anywhere            

   13  1092 ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED

    9   756 DOCKER     all  --  any    docker0  anywhere             anywhere            

   13  1092 ACCEPT     all  --  docker0 !docker0  anywhere             anywhere            

    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere            

    0     0 ACCEPT     all  --  any    br-68c3e5c395af  anywhere             anywhere             ctstate RELATED,ESTABLISHED

    0     0 DOCKER     all  --  any    br-68c3e5c395af  anywhere             anywhere            

    0     0 ACCEPT     all  --  br-68c3e5c395af !br-68c3e5c395af  anywhere             anywhere            

    0     0 ACCEPT     all  --  br-68c3e5c395af br-68c3e5c395af  anywhere             anywhere            

    0     0 ACCEPT     all  --  any    br0     anywhere             anywhere            

    0     0 DOCKER     all  --  any    br0     anywhere             anywhere            

    0     0 ACCEPT     all  --  br0    !br0    anywhere             anywhere            

    0     0 ACCEPT     all  --  br0    br0     anywhere             anywhere            

Chain OUTPUT (policy ACCEPT 1230 packets, 160K bytes)

 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (3 references)

 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)

 pkts bytes target     prot opt in     out     source               destination         

    0     0 DOCKER-ISOLATION-STAGE-2  all  --  br0    !br0    anywhere             anywhere            

   13  1092 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  anywhere             anywhere            

    0     0 DOCKER-ISOLATION-STAGE-2  all  --  br-68c3e5c395af !br-68c3e5c395af  anywhere             anywhere            

   80  6664 RETURN     all  --  any    any     anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (3 references)

 pkts bytes target     prot opt in     out     source               destination         

    0     0 DROP       all  --  any    br0     anywhere             anywhere            

    0     0 DROP       all  --  any    docker0  anywhere             anywhere            

    0     0 DROP       all  --  any    br-68c3e5c395af  anywhere             anywhere            

   13  1092 RETURN     all  --  any    any     anywhere             anywhere            

Chain DOCKER-USER (1 references)

 pkts bytes target     prot opt in     out     source               destination         

   80  6664 RETURN     all  --  any    any     anywhere             anywhere            

```

still no go. I suspect docker does yet more "magic" with it's docker0 bridge. But what else could it be? no idea....maybe some extra netfilter table I'm not aware of?

----------

## szatox

If you showed us the result of iptables-save instead of iptables -vL, it would be easier to follow.

It provides an unambiguous dump (one that can be imported back). It will also include all tables without any need to manually specify them. And I suppose many people here are more used to its format (yes, including me).

Docker is a really messy piece of software. It alters your firewall rules in weird ways, it can disable your firewall, drop some traffic you actually wanted, bang your milk or drink your cat.

If you want to run it alongside anything else on the same host, you should probably isolate it with another network namespace. It can be done and is not very hard once you know how to use namespaces (AFAIK it breaks swarm mode, but pretty much any other docker manager will work, should you need it) so you're almost there: just create another ns, connect it to the host with a pair of veth and run docker daemon inside. And either bridge or route traffic as needed.

This basically requires you to write your own init scripts; but it's still not very hard to do.

----------

## viacheslavg

Indeed iptables-save output is much better and  it helped me to sort out the problem. Docker was also involved but I can't say it does silly things.

Here is explanation:

I had /etc/sysctl.d/local.conf with

```
net.ipv4.ip_forward=1
```

so, I thought ip forwarding is enabled, but as it appears in /etc/sysctl.conf I had:

```
net.ipv4.ip_forward=0
```

and as result forwarding was disabled. Which is strange, my assumption was that custom configs in sysctl.d/ should override the one from /etc/sysctl.conf

However when docker starts it enables this option and to preserve existing behavior for the rest of system it changes FORWARD policy from ACCEPT to DROP (plus of course adds rules for it's own bridge to make it working).

Thus when I start my system w/o docker enabled forwarding was not working because of net.ipv4.ip_forward=0 setting, but when I start my system with docker forwarding was disabled for my bridge in iptables by docker.

Summarizing, the solution is following:

Possible two options:

1. If you are OK with having net.ipv4.ip_forward=1 system wide then only following rule required in iptables:

```
iptables -t nat -A POSTROUTING -s 192.168.8.0/24 -j MASQUERADE
```

2. If you need to have net.ipv4.ip_forward=0 and docker is installed then two additional rules can fix the problem:

```

iptables -A FORWARD -o br0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

iptables -A FORWARD -i br0 ! -o br0 -j ACCEPT

```

Thanks a lot for helping!

----------

