# OpenSwan 2.4.7 and NATed Win XP: PAYLOAD_MALFORMED [SOLVED]

## VinzC

Hi.

I've unmasked (keyword file) and installed OpenSwan 2.4.7 and l2tpd-0.70_pre20031121. The VPN server is behind a router with a firewall that allows forwarding only TCP and UPD ports; UDP ports 500, 4500 and 1701 have been forwarded to the OpenSwan server. My own XP machine is behind a Gentoo Linux NAT. Both machines have dynamic IP adresses that may change in less than 24 hours.

When I try to make a connection to the VPN server from my XP machine, I can see these messages on the server:

```
Jan  1 05:47:46 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #12: sending notification PAYLOAD_MALFORMED to 62.197.xxx.yyy:4500

Jan  1 05:47:49 serenity pluto[6728]: ERROR: asynchronous network error report on eth0 (sport=4500) for message to 62.197.xxx.yyy port 4500, complainant 192.168.1.5: No route to host [errno 113, origin ICMP type 3 code 1 (not authenticated)]

Jan  1 05:47:49 serenity pluto[6728]: ERROR: asynchronous network error report on eth0 (sport=4500) for message to 62.197.xxx.yyy port 4500, complainant 192.168.1.5: No route to host [errno 113, origin ICMP type 3 code 1 (not authenticated)]

Jan  1 05:47:54 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #12: next payload type of ISAKMP Hash Payload has an unknown value: 120

Jan  1 05:47:54 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #12: malformed payload in packet

Jan  1 05:47:54 serenity pluto[6728]: | payload malformed after IV

Jan  1 05:47:54 serenity pluto[6728]: |   30 5e ec b5  a5 3b 39 e2

Jan  1 05:47:54 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #12: sending notification PAYLOAD_MALFORMED to 62.197.xxx.yyy:4500

Jan  1 05:48:10 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #12: next payload type of ISAKMP Hash Payload has an unknown value: 120

Jan  1 05:48:10 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #12: malformed payload in packet

Jan  1 05:48:10 serenity pluto[6728]: | payload malformed after IV

Jan  1 05:48:10 serenity pluto[6728]: |   30 5e ec b5  a5 3b 39 e2

Jan  1 05:48:10 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #12: sending notification PAYLOAD_MALFORMED to 62.197.xxx.yyy:4500

Jan  1 05:48:12 serenity pluto[6728]: ERROR: asynchronous network error report on eth0 (sport=4500) for message to 62.197.xxx.yyy port 4500, complainant 192.168.1.5: No route to host [errno 113, origin ICMP type 3 code 1 (not authenticated)]

Jan  1 05:48:12 serenity pluto[6728]: ERROR: asynchronous network error report on eth0 (sport=4500) for message to 62.197.xxx.yyy port 4500, complainant 192.168.1.5: No route to host [errno 113, origin ICMP type 3 code 1 (not authenticated)]

Jan  1 05:48:12 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #9: received Delete SA(0xfdc36d06) payload: deleting IPSEC State #10

Jan  1 05:48:12 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #9: received and ignored informational message

Jan  1 05:48:12 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #7: received Delete SA(0x30d63173) payload: deleting IPSEC State #8

Jan  1 05:48:12 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #7: received and ignored informational message

Jan  1 05:48:12 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #5: received Delete SA(0x9c8943e3) payload: deleting IPSEC State #6

Jan  1 05:48:12 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #5: received and ignored informational message

Jan  1 05:48:12 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #3: received Delete SA(0x0a342f7a) payload: deleting IPSEC State #4

Jan  1 05:48:12 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #3: received and ignored informational message

Jan  1 05:48:12 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #1: received Delete SA(0x39fe8292) payload: deleting IPSEC State #2

Jan  1 05:48:12 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #1: received and ignored informational message

Jan  1 05:48:12 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #11: received Delete SA payload: deleting ISAKMP State #11

Jan  1 05:48:12 serenity pluto[6728]: packet from 62.197.xxx.yyy:4500: received and ignored informational message

Jan  1 05:48:12 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #9: received Delete SA payload: deleting ISAKMP State #9

Jan  1 05:48:12 serenity pluto[6728]: packet from 62.197.xxx.yyy:4500: received and ignored informational message

Jan  1 05:48:12 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #7: received Delete SA payload: deleting ISAKMP State #7

Jan  1 05:48:12 serenity pluto[6728]: packet from 62.197.xxx.yyy:4500: received and ignored informational message

Jan  1 05:48:12 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #5: received Delete SA payload: deleting ISAKMP State #5

Jan  1 05:48:12 serenity pluto[6728]: packet from 62.197.xxx.yyy:4500: received and ignored informational message

Jan  1 05:48:12 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #3: received Delete SA payload: deleting ISAKMP State #3

Jan  1 05:48:12 serenity pluto[6728]: packet from 62.197.xxx.yyy:4500: received and ignored informational message

Jan  1 05:48:12 serenity pluto[6728]: "roadwarrior-xp"[2] 62.197.xxx.yyy #1: received Delete SA payload: deleting ISAKMP State #1

Jan  1 05:48:12 serenity pluto[6728]: packet from 62.197.xxx.yyy:4500: received and ignored informational message

Jan  1 05:48:15 serenity pluto[6728]: ERROR: asynchronous network error report on eth0 (sport=4500) for message to 62.197.xxx.yyy port 4500, complainant 192.168.1.5: No route to host [errno 113, origin ICMP type 3 code 1 (not authenticated)]

Jan  1 05:48:15 serenity pluto[6728]: ERROR: asynchronous network error report on eth0 (sport=4500) for message to 62.197.xxx.yyy port 4500, complainant 192.168.1.5: No route to host [errno 113, origin ICMP type 3 code 1 (not authenticated)]

Jan  1 05:48:15 serenity pluto[6728]: ERROR: asynchronous network error report on eth0 (sport=4500) for message to 62.197.xxx.yyy port 4500, complainant 192.168.1.5: No route to host [errno 113, origin ICMP type 3 code 1 (not authenticated)]
```

I had OpenSwan 2.4.4 before and other error messages, which exhibited a problem with the server being behind a NAT. The masked version of OpenSwan fixed these problems (actually version 2.4.5 but it's not in portage). 62.197.xxx.yyy is my public IP address, i.e. the one of my Gentoo Linux gateway at home, which happens to do NAT. 192.168.1.5 is the private IP address of the remote OpenSwan server (it has only one NIC since it's inside a private LAN).

Here's the server's ipsec.conf:

```
version 2.0     # conforms to second version of ipsec.conf specification

config setup

        interfaces=%defaultroute

        klipsdebug=none

        plutodebug=none

        overridemtu=1410

        nat_traversal=yes

        virtual_private=%v4:10.0.0.0/8,%v4:172.16.0.0/12,%v4:192.168.0.0/16,%v4:!192.168.1.0/24

conn %default

        keyingtries=3

        compress=no

        disablearrivalcheck=no

        keyexchange=ike

        ikelifetime=240m

        keylife=60m

conn roadwarrior-xp

        leftprotoport=17/1701

        rightprotoport=17/%any

        rekey=no

        also=roadwarrior

conn roadwarrior

        authby=secret

        pfs=no

        type=tunnel

        left=%defaultroute

        right=%any

        rightsubnet=vhost:%no,%priv

        auto=add

include /etc/ipsec/ipsec.d/examples/no_oe.conf
```

The only wrong thing I understand is that there can't be anything at 62.197.xxx.yyy:500 nor 62.197.xxx.yyy:500 since this is my home Gentoo Linux server public address and my XP machine (the VPN client) is behind it. So if I understand correctly the VPN server tries to send responses to a port that doesn't exist. Can anybody help me find out what's wrong?

----------

## VinzC

As I'm using Windows XP SP2 and a NAT'ed VPN server, there is a registry patch to apply ( http://www.jacco2.dds.nl/networking/openswan-l2tp.html#serverNATed ). I also switched to net-dialup/xl2tpd-1.1.06 and disabled Data compression in the XP VPN client connection properties. I'm still receiving error messages but it looks like the connection process goes a little further:

```
Jan  1 14:11:58 serenity pluto[10568]: packet from 62.197.xxx.yyy:500: ignoring Vendor ID payload [MS NT5 ISAKMPOAKLEY 00000004]

Jan  1 14:11:58 serenity pluto[10568]: packet from 62.197.xxx.yyy:500: ignoring Vendor ID payload [FRAGMENTATION]

Jan  1 14:11:58 serenity pluto[10568]: packet from 62.197.xxx.yyy:500: received Vendor ID payload [draft-ietf-ipsec-nat-t-ike-02_n] method set to=106

Jan  1 14:11:58 serenity pluto[10568]: packet from 62.197.xxx.yyy:500: ignoring Vendor ID payload [Vid-Initial-Contact]

Jan  1 14:11:58 serenity pluto[10568]: "roadwarrior-xp"[2] 62.197.xxx.yyy #48: responding to Main Mode from unknown peer 62.197.xxx.yyy

Jan  1 14:11:58 serenity pluto[10568]: "roadwarrior-xp"[2] 62.197.xxx.yyy #48: transition from state STATE_MAIN_R0 to state STATE_MAIN_R1

Jan  1 14:11:58 serenity pluto[10568]: "roadwarrior-xp"[2] 62.197.xxx.yyy #48: STATE_MAIN_R1: sent MR1, expecting MI2

Jan  1 14:11:58 serenity pluto[10568]: "roadwarrior-xp"[2] 62.197.xxx.yyy #48: NAT-Traversal: Result using draft-ietf-ipsec-nat-t-ike-02/03: both are NATed

Jan  1 14:11:58 serenity pluto[10568]: "roadwarrior-xp"[2] 62.197.xxx.yyy #48: transition from state STATE_MAIN_R1 to state STATE_MAIN_R2

Jan  1 14:11:58 serenity pluto[10568]: "roadwarrior-xp"[2] 62.197.xxx.yyy #48: STATE_MAIN_R2: sent MR2, expecting MI3

Jan  1 14:11:58 serenity pluto[10568]: "roadwarrior-xp"[2] 62.197.xxx.yyy #48: Main mode peer ID is ID_FQDN: '@game01'

Jan  1 14:11:58 serenity pluto[10568]: "roadwarrior-xp"[2] 62.197.xxx.yyy #48: I did not send a certificate because I do not have one.

Jan  1 14:11:58 serenity pluto[10568]: "roadwarrior-xp"[2] 62.197.xxx.yyy #48: transition from state STATE_MAIN_R2 to state STATE_MAIN_R3

Jan  1 14:11:58 serenity pluto[10568]: | NAT-T: new mapping 62.197.xxx.yyy:500/4500)

Jan  1 14:11:58 serenity pluto[10568]: "roadwarrior-xp"[2] 62.197.xxx.yyy #48: STATE_MAIN_R3: sent MR3, ISAKMP SA established {auth=OAKLEY_PRESHARED_KEY ci

pher=oakley_3des_cbc_192 prf=oakley_sha group=modp2048}

Jan  1 14:11:58 serenity pluto[10568]: "roadwarrior-xp"[2] 62.197.xxx.yyy #49: responding to Quick Mode {msgid:b322070a}

Jan  1 14:11:58 serenity pluto[10568]: "roadwarrior-xp"[2] 62.197.xxx.yyy #49: transition from state STATE_QUICK_R0 to state STATE_QUICK_R1

Jan  1 14:11:58 serenity pluto[10568]: "roadwarrior-xp"[2] 62.197.xxx.yyy #49: STATE_QUICK_R1: sent QR1, inbound IPsec SA installed, expecting QI2

Jan  1 14:11:58 serenity pluto[10568]: "roadwarrior-xp"[2] 62.197.xxx.yyy #49: transition from state STATE_QUICK_R1 to state STATE_QUICK_R2

Jan  1 14:11:58 serenity pluto[10568]: "roadwarrior-xp"[2] 62.197.xxx.yyy #49: STATE_QUICK_R2: IPsec SA established {ESP=>0x748f1449 <0x94eb6794 xfrm=3DES_

0-HMAC_MD5 NATD=62.197.xxx.yyy:4500 DPD=none}

Jan  1 14:12:00 serenity xl2tpd[10431]: control_finish: Peer requested tunnel 2 twice, ignoring second one.

Jan  1 14:12:01 serenity xl2tpd[10431]: control_finish: Peer requested tunnel 2 twice, ignoring second one.

Jan  1 14:12:03 serenity pluto[10568]: ERROR: asynchronous network error report on eth0 (sport=4500) for message to 62.197.xxx.yyy port 4500, complainant 1

92.168.1.5: No route to host [errno 113, origin ICMP type 3 code 1 (not authenticated)]

Jan  1 14:12:03 serenity pluto[10568]: ERROR: asynchronous network error report on eth0 (sport=4500) for message to 62.197.xxx.yyy port 4500, complainant 1

92.168.1.5: No route to host [errno 113, origin ICMP type 3 code 1 (not authenticated)]

Jan  1 14:12:03 serenity pluto[10568]: ERROR: asynchronous network error report on eth0 (sport=4500) for message to 62.197.xxx.yyy port 4500, complainant 1

92.168.1.5: No route to host [errno 113, origin ICMP type 3 code 1 (not authenticated)]

Jan  1 14:12:05 serenity xl2tpd[10431]: control_finish: Peer requested tunnel 2 twice, ignoring second one.

Jan  1 14:12:05 serenity xl2tpd[10431]: Maximum retries exceeded for tunnel 52425.  Closing.

Jan  1 14:12:05 serenity xl2tpd[10431]: Connection 2 closed to 62.197.xxx.yyy, port 1701 (Timeout)
```

----------

## VinzC

Problem almost solved: I added the following line to the road-warrior section /etc/ipsec/ipsec.conf:

```
leftnexthop=<LAN ip address of the internet router>
```

Now the remote server is inaccessible once I close the VPN connection  :Shocked:  . I must get to the remote site to check what's wrong. Anyway I can make a succesfull connection now.

----------

## VinzC

Problem solved. The server happened to be inaccessible due to something else, pure coïncidence.

----------

