# MPTCP on Gentoo kernel 4.10 : Dummy questions [SOLVED]

## orion777

I'm running gentoo 64 bit kernel 4.10 on Raspberry PI 3B (SAkakis' image https://forums.gentoo.org/viewtopic-p-8177442.html#8177442 ).

Now I wish to install MPTCP to get robust (redundant) network connection.

So, getting kernel sources and MPTCP patch for 4.10 into linux_mptcp:

```
cd /usr/src/

git clone --depth 1 https://github.com/raspberrypi/linux.git -b rpi-4.10.y

cp linux linux_mptcp

cd linux_mptcp

make distclean

make bcmrpi3_defconfig

wget http://multipath-tcp.org/patches/mptcp-v4.10-d9c9f9273eb1.patch
```

Now applying the patch. It will apply with an error:

```
patch -p1 < mptcp-v4.10-d9c9f9273eb1.patch

1 out of 46 hunks FAILED -- saving rejects to file net/ipv4/tcp_output.c.rej

   pi64 /usr/src/linux_mptcp # cat net/ipv4/tcp_output.c.rej

   --- net/ipv4/tcp_output.c

   +++ net/ipv4/tcp_output.c

   @@ -1282,7 +1307,7 @@ int tcp_fragment(struct sock *sk, struct sk_buff *skb, u32 len,

     * eventually). The difference is that pulled data not copied, but

     * immediately discarded.

     */

   -static void __pskb_trim_head(struct sk_buff *skb, int len)

   +void __pskb_trim_head(struct sk_buff *skb, int len)

    {

           struct skb_shared_info *shinfo;

           int i, k, eat;
```

Patch was prepared for the vanilla 4.10 kernel; however gentoo 4.10 kernel have different string https://github.com/torvalds/linux/commit/7162fb242cb8322beb558828fd26b33c3e9fc805

So now we have to manually change static int __pskb_trim_head(struct sk_buff *skb, int len) to int __pskb_trim_head(struct sk_buff *skb, int len) (changing void to int, because of commit https://github.com/torvalds/linux/commit/7162fb242cb8322beb558828fd26b33c3e9fc805 ))

Now kernel make will fails, as ./include/net/tcp_output.c was changed by the patch to vanilla string. Ok, lets edit it manually for the gentoo:

```
cd ./include/net/

edit by MC tcp_output.c

void __pskb_trim_head(struct sk_buff *skb, int len);

   to

   int __pskb_trim_head(struct sk_buff *skb, int len);
```

Now kernel can be maked without fatal errors. So its time to configure .config https://wiki.gentoo.org/wiki/MultiPath_TCP

However, this how-to seems not full.. I was truing it, but later I was adding more in .config to specify MPTCP "defaults":

```
make menuconfig

¦     -> Networking support (NET [=y])                                    ¦

  ¦       -> Networking options                                             ¦

  ¦         -> TCP/IP networking (INET [=y])                                ¦

  ¦ (1)       -> The IPv6 protocol (IPV6 [=m]) to KERNEL (or DISABLE, but my is in kernel..)

then

Networking support (NET [=y])

  Networking options

    TCP/IP networking (INET [=y])

      [*]   IP: advanced router

        [*]     IP: policy routing

      [*]   MPTCP protocol

      [*]     MPTCP: advanced path-manager control  --->

        <*>   MPTCP Full-Mesh Path-Manager

        <*>   MPTCP Linked Increase

    Default MPTCP Path-Manager (Full mesh)  --->  Full mesh

   MPTCP: advanced scheduler control                            x x

         <*>   MPTCP Round-Robin                                          x x

  x x    <*>   MPTCP Redundant                                            x x

  x x          Default MPTCP Scheduler (Redundant)  --->   

and

Networking support (NET [=y])

  Networking options

    TCP/IP networking (INET [=y])

      [*]   TCP: advanced congestion control  --->

        <*>   MPTCP COUPLED CONGESTION CONTROL (!!Can't be found in my menuconfig!!)

        <*>   MPTCP Opportunistic Linked Increase

           Default TCP congestion control (Lia)  ---> Lia
```

Now it is possible to make the kernel:

```
make

make modules_install

make firmware_install

cp -v arch/arm64/boot/Image /boot/kernel8.img

cp -v arch/arm64/boot/dts/broadcom/bcm{2710,2837}-rpi-3-b.dtb /boot/

sync

reboot
```

Ok, now its time to set MPTCP settings, as is shown here http://multipath-tcp.org/pmwiki.php/Users/ConfigureRouting

We have two network cards (the first one is build-in ethernet, the second one is virtual ethernet 3G/4G dongle (cdc-ethernet, sometimes called "stick")).

```
pi64 ~ # ifconfig eth0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 192.168.1.118  netmask 255.255.255.0  broadcast 192.168.1.255

        inet6 fe80::5102:cac7:da40:234d  prefixlen 64  scopeid 0x20<link>

        ether b8:27:eb:28:f1:db  txqueuelen 1000  (Ethernet)

        RX packets 961  bytes 68443 (66.8 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 470  bytes 63405 (61.9 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

pi64 ~ # ifconfig eth3

eth3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 192.168.8.10  netmask 255.255.255.0  broadcast 192.168.8.255

        inet6 fe80::5d49:fee0:c509:bb11  prefixlen 64  scopeid 0x20<link>

        ether 0c:5b:8f:27:9a:64  txqueuelen 1000  (Ethernet)

        RX packets 60  bytes 25286 (24.6 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 18  bytes 1710 (1.6 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
```

Both obtain its ip by dhcp.

Lets check them:

```
pi64 ~ # ping -I eth0 8.8.8.8

PING 8.8.8.8 (8.8.8.8) from 192.168.1.118 eth0: 56(84) bytes of data.

64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=21.4 ms

64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=23.0 ms

64 bytes from 8.8.8.8: icmp_seq=3 ttl=55 time=18.5 ms

^C

--- 8.8.8.8 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2002ms

rtt min/avg/max/mdev = 18.533/21.011/23.060/1.880 ms

pi64 ~ # ping -I eth3 8.8.8.8

PING 8.8.8.8 (8.8.8.8) from 192.168.8.10 eth3: 56(84) bytes of data.

64 bytes from 8.8.8.8: icmp_seq=1 ttl=53 time=655 ms

64 bytes from 8.8.8.8: icmp_seq=2 ttl=53 time=635 ms

64 bytes from 8.8.8.8: icmp_seq=3 ttl=53 time=75.4 ms

^C

--- 8.8.8.8 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 1999ms

rtt min/avg/max/mdev = 75.436/455.283/655.183/268.715 ms

pi64 ~ #

```

Ok, both are working. Now it is time to configure http://multipath-tcp.org/pmwiki.php/Users/ConfigureRouting

 *Quote:*   

> pi64 ~ # # This creates two different routing tables, that we use based on the source-address.
> 
> pi64 ~ # ip rule add from 192.168.1.118 table 1
> 
> pi64 ~ # ip rule add from 192.168.8.10 table 2
> ...

 

Now lets check resulting settings, as specified here http://multipath-tcp.org/pmwiki.php/Users/ConfigureRouting

```
pi64 ~ # ip rule show

0:      from all lookup local

32764:  from 192.168.8.10 lookup 2

32765:  from 192.168.1.118 lookup 1

32766:  from all lookup main

32767:  from all lookup default

pi64 ~ # ip route

default via 192.168.1.1 dev eth0

default via 192.168.1.1 dev eth0 src 192.168.1.118 metric 202

default via 192.168.8.1 dev eth3 src 192.168.8.10 metric 204

192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.118 metric 202

192.168.8.0/24 dev eth3 proto kernel scope link src 192.168.8.10 metric 204

pi64 ~ # ip route show table 1

default via 192.168.1.1 dev eth0

192.168.1.0/24 dev eth0 scope link

pi64 ~ # ip route show table 2

default via 192.168.8.1 dev eth3

192.168.8.0/24 dev eth3 scope link

```

So Ok.. @blind@ configuring was done (blind because I dont understand what I'm doing...)

Now lets check network connectivity. Due of setting Default MPTCP Scheduler (Redundant) in the kernel, I suppose that pings' ICMP packet will be sent thru both networks, so I will widrawn eth0 network cable during the experiment and will expectate that connectivity will be still available:

```
 ping -O 8.8.8.8

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=20.0 ms

64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=19.3 ms

64 bytes from 8.8.8.8: icmp_seq=3 ttl=55 time=19.9 ms

no answer yet for icmp_seq=4

no answer yet for icmp_seq=5

no answer yet for icmp_seq=6

```

So, as can be seen, as soon eth0 cable was interrupter (AFTER its network router, that why we do not see From 192.168.1.1 icmp_seq=18 Destination Net Unreachable) the data transmission also aborts.

NOTE that network cards are still accessible directly:

```
pi64 ~ # ping -I eth0 8.8.8.8

PING 8.8.8.8 (8.8.8.8) from 192.168.1.118 eth0: 56(84) bytes of data.

64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=20.7 ms

64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=17.9 ms

^C

--- 8.8.8.8 ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 1001ms

rtt min/avg/max/mdev = 17.964/19.376/20.789/1.419 ms

pi64 ~ # ping -I eth3 8.8.8.8

PING 8.8.8.8 (8.8.8.8) from 192.168.8.10 eth3: 56(84) bytes of data.

64 bytes from 8.8.8.8: icmp_seq=1 ttl=53 time=660 ms

64 bytes from 8.8.8.8: icmp_seq=2 ttl=53 time=349 ms

^C

--- 8.8.8.8 ping statistics ---

3 packets transmitted, 2 received, 33.3333% packet loss, time 2002ms

rtt min/avg/max/mdev = 349.341/504.981/660.622/155.642 ms

```

So, I don't understand WHAT to DO to make redundant connection, based on two network cards. Seems that MPTCP is not working in my setup  :Sad: 

If someone have any experience working with MPTCP, please suggest something..

(I'm not experience user in linux, so some typical mistakes are also possible)Last edited by orion777 on Thu Mar 15, 2018 8:02 am; edited 1 time in total

----------

## orion777

So, as I understood, nobody was/is using MPTCP on gentoo?

----------

## Hu

Nobody satisfies all of:Reads and posts on the forumsUses MPTCP on GentooCan leverage #2 to help youThere might be MPTCP users who do not read the forums and have not seen this thread, or who use it, but lack the expertise to assist you.

----------

## Ant P.

Is there anything you need MPTCP for that you can't do using the simpler, built in multipath ip routing?

----------

## orion777

Initially I was planning to use NetworkBonding to be able to send same packet thru different network cards simultaneously.

Due of the fact that I'm not experience user, I spend two months to get network configuration running (to understand interrelationship between NetworkManager, dhcpcd, dhclient and what is written in gentoo tutorial https://wiki.gentoo.org/wiki/Handbook:X86/Full/Networking ). And ONLY after that I understand that Network Bonding cannot be build using different network providers... (it was discussed and I was getting the answer here https://forums.gentoo.org/viewtopic-t-1074802.html . Also there is an advice to use two VPN connections to be able to bond them togehter, but I was not truing this solution yet) 

So, to be able to send same packet over two different networks (in fact, two 3G/4G modems that emulates virtual ethernet cards) simultaneously I try to use mentioned here https://forums.gentoo.org/viewtopic-t-1060694-start-0.html MPTCP solution. But due of the fact that I'm dummy, I cant force it to work  :Sad: 

What You means under built in multipath ip routing ?

----------

## Ant P.

Enaable CONFIG_IP_ROUTE_MULTIPATH and the kernel will automatically spread packets among routes that point to the same destination.

----------

## orion777

Do You mean this?

```
 Symbol: IP_ROUTE_MULTIPATH [=y]                                         

  Type  : boolean                                                         

  Prompt: IP: equal cost multipath                                        

    Location:                                                             

      -> Networking support (NET [=y])                                    

        -> Networking options                                             

          -> TCP/IP networking (INET [=y])                                

  (4)       -> IP: advanced router (IP_ADVANCED_ROUTER [=y])              

    Defined at net/ipv4/Kconfig:81                                        

    Depends on: NET [=y] && INET [=y] && IP_ADVANCED_ROUTER [=y] 
```

As can be seen, it is enabled in my kernel. Also I have iproute2 4.14.1 installed. 

What then I have to do to be able to build redundant connection (that is sending packets over two interfaces in parallel)? This one?

 *https://www.opennet.ru/openforum/vsluhforumID3/233.html wrote:*   

> <p><b>Использование 2 линков</b>:<br>
> 
> В ядре: CONFIG_IP_ADVANCED_ROUTER=y, CONFIG_IP_MULTIPLE_TABLES=y,
> 
> CONFIG_IP_ROUTE_MULTIPATH=y<br>
> ...

 

----------

## bbgermany

Hi,

bonding of wan connections is possible, but your provider MUST support this. Usually they dont. Did you have a look at this: http://lartc.org/howto/lartc.rpdb.multiple-links.html

Since you are trying to "load balance" connections via your two 3G/4G interfaces, this could help you.

greets, bb

----------

## orion777

 *bbgermany wrote:*   

> bonding of wan connections is possible, but your provider MUST support this. Usually they dont. Did you have a look at this: http://lartc.org/howto/lartc.rpdb.multiple-links.html
> 
> Since you are trying to "load balance" connections via your two 3G/4G interfaces, this could help you.

 

I'm mostly looking for the solution how to get fail-save operation (it is necessary to build network solution for the remotely piloted robot, whet one provider may not have sufficient coverage, signal strength, etc.. that why I wish to use two operators simultaneously).

In the how-to mentioned above I cant understand what means  Finally, let $P1_NET be the IP network $P1 is in,

$IF1 - interface name, like eth0

$IP1 - interface IP, like 192.168.0.10

$P1 - gateway IP, like 192.168.0.1

$P1_NET - ?? net mask? 192.168.0.10/24 ? or something like 192.168.0.0 ?

----------

## bbgermany

Thats quite easy:

if you get the ip address 192.168.0.1 ($IP1) from your provider on your first 3G/4G interface ($IF1). The fitting network ($P1_NET) could be 192.168.0.0. Gateways and mask and even the subnet is generated by the provider. You have to look it up via ifconfig.

greets, bb

----------

## orion777

So, here is my existing configs "by default"

```
 # ip route show

default via 192.168.1.1 dev eth0 src 192.168.1.118 metric 202

default via 192.168.8.1 dev eth3 src 192.168.8.10 metric 204

192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.118 metric 202

192.168.8.0/24 dev eth3 proto kernel scope link src 192.168.8.10 metric 204

```

So, as is specified here, at first we have to create two network tables, called T1 and T2.

I we found solution HOW to do this in http://www.allgoodbits.org/articles/view/24 in the section  Create a custom routing table: echo "1   myorg" >> /etc/iproute2/rt_tables

Then configuration has to be applied. So:

```
echo "T1" >> /etc/iproute2/rt_tables

echo "T2" >> /etc/iproute2/rt_tables

ip route add $192.168.1.0 dev eth0 src 192.168.1.118 table T1

ip route add default via 192.168.1.1 table T1

ip route add 192.168.8.0 dev eth3 src 192.168.8.10 table T2

ip route add default via 192.168.8.1 table T2

ip route add 192.168.1.0 dev eth0 src 192.168.1.118

ip route add 192.168.8.0 dev eth3 src 192.168.8.10

ip route add default via 192.168.1.1

ip rule add from 192.168.1.118 table T1

ip rule add from 192.168.8.10 table T2

ip route add default scope global nexthop via 192.168.1.1 dev eth0 weight 1

nexthop via 192.168.8.1 dev eth3 weight 1
```

However, whet two T1 and T2 has been added, nothing can be writen inside into it:

```
# ip route add $192.168.1.0 dev eth0 src 192.168.1.118 table T1

Database /etc/iproute2/rt_tables is corrupted at T1

Error: argument "T1" is wrong: "table" value is invalid

```

```
pi64 /etc/iproute2 # cat rt_tables

#

# reserved values

#

255     local

254     main

253     default

0       unspec

#

# local

#

#1      inr.ruhep

T1

T2

```

----------

## bbgermany

I think you did it wrong a bit; You need to add a number for your routing as well:

```

echo "1   myorg" >> /etc/iproute2/rt_tables

```

Not only:

```

echo "myorg" >> /etc/iproute2/rt_tables

```

greets, bb

----------

## orion777

Yes, number and name, separated by "TAB", like echo "1	T1" >> /etc/iproute2/1rt_tables

But now I have two more problems..

```
# ip route add $192.168.1.0 dev eth0 src 192.168.1.118 table T1

# ip route add default via 192.168.1.1 table T1

# ip route add 192.168.8.0 dev eth3 src 192.168.8.10 table T2

# ip route add default via 192.168.8.1 table T2

# ip route add 192.168.1.0 dev eth0 src 192.168.1.118

# ip route add 192.168.8.0 dev eth3 src 192.168.8.10

# ip route add default via 192.168.1.1

# ip rule add from 192.168.1.118 table T1

# ip rule add from 192.168.8.10 table T2

# ip route add default scope global nexthop via 192.168.1.1 dev eth0 weight 1

RTNETLINK answers: File exists

# nexthop via 192.168.8.1 dev eth3 weight 1

-bash: nexthop: command not found
```

----------

## bbgermany

```

ip route add default scope global nexthop via 192.168.1.1 dev eth0 weight 1  nexthop via 192.168.8.1 dev eth3 weight 1 

```

thats one command not two.

greets, bb

----------

## orion777

 *bbgermany wrote:*   

>  thats one command not two. 

 

Ok... However, now I get: 

```
ip route add default scope global nexthop via 192.168.1.1 dev eth0 weight 1 nexthop via 192.168.8.1 dev eth3 weight 1

RTNETLINK answers: File exists
```

So, as was specified here for some similar problem, I was replacing ip route add by ip route replace , so now it looks like this:

```
ip route replace $192.168.1.0 dev eth0 src 192.168.1.118 table T1

ip route replace default via 192.168.1.1 table T1

ip route replace 192.168.8.0 dev eth3 src 192.168.8.10 table T2

ip route replace default via 192.168.8.1 table T2

ip route replace 192.168.1.0 dev eth0 src 192.168.1.118

ip route replace 192.168.8.0 dev eth3 src 192.168.8.10

ip route replace default via 192.168.1.1

ip rule add from 192.168.1.118 table T1

ip rule add from 192.168.8.10 table T2

ip route replace default scope global nexthop via 192.168.1.1 dev eth0 weight 1 nexthop via 192.168.8.1 dev eth3 weight 1

 
```

Now ping, as well any wget is done thru eth3 only  :Question: 

Yes, metric is different.. So maybe this solution is applicable to same network cards only?

```
pi64 ~ # ip route

default

        nexthop via 192.168.1.1 dev eth0 weight 1

        nexthop via 192.168.8.1 dev eth3 weight 1

default via 192.168.1.1 dev eth0 src 192.168.1.118 metric 202

default via 192.168.8.1 dev eth3 src 192.168.8.10 metric 204

192.168.1.0 dev eth0 scope link src 192.168.1.118

192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.118 metric 202

192.168.8.0 dev eth3 scope link src 192.168.8.10

192.168.8.0/24 dev eth3 proto kernel scope link src 192.168.8.10 metric 204

```

----------

## bbgermany

Im very sorry, but I cannot help you with this, since I connot test it by myself. But I think, it could be related to the default gateways, your provider gives you, when you start up your connections.

greets, bb

----------

## orion777

I have two suggestions:

1) this solution cant work with network cards with different metrics

2) Both my cards gets IP by means of DHCP (dhclient, dhcpcd, NetworkManager). NetworkManager (as I understand) writes default gateway.. So maybe I cant override this default gateway settings?

----------

## orion777

I have an answer.. and it is very simple! I found this in "options->default" description: 'default': This scheduler is the default one. It will first send data on subflows with the lowest RTT until their congestion-window is full. Then, it will start transmitting on the subflows with the next higher RTT. So, I can conclude that MPTCP uses TCP window as a measurement parameter to able to work. This means that MPTCP can work only with TCP. However, in my testing, I was using "ping", so ICMP packets. As I know, ICMP packets does not use windows, so maybe that why all my MPTCP tests were failed. Then I was trying to initiate TCP/IP data flow by wget something and both cards starts to work; also if one card is failed, down, on lost connection and stay live, second card takes all load automatically. 

A few words about settings.

- First, you have to patch your kernel.

- Then it is necessary to enter additional routing settings. This can be done manually each time after system reboot or can be automated by various scripts (suggested gentoo script has mistakes and will not work. If someone know, please suggest me how to fix it ). All related info (manual and auto scripting) is here

- Default settings are specified during kernel make.

- Settings, those are set by default in kernel, can be overridden manually in live system thru sysctl

--- It is recommended to select default. In this mode MPTCP will use both network cards (not only one!) and your speed of connection will be doubled. Traffic balancing is fully flexible, based on 

TCP window (as soon it is full - traffic is sent to the second network card and etc). Round-robin is not recommended because it is experimental,... seems that it has no flexibility and can work fine only with equal speed networks.

----------

