# scp terrible over the Internet

## bssteph

Hello,

I'm having problems with scp over the Internet. I can ssh to a number of sites (some I own running Gentoo, some not) and have relatively decent latency --- nothing stellar, but Wireshark does not complain, so that's a good thing. However, when scp'ing anything big (say, an MP3), the transfer does fine initially, but then becomes a huge storm of TCP retransmissions (much higher than natural) and TCP dup ACKs as reported by Wireshark. scp will think everything transferred fairly quickly and jump to 95% soon on, but then hang for minutes more as data is many times over retransmitted. I can see this doing a LAN -> Internet transfer, Gentoo router box -> Internet transfer, and a Internet -> Gentoo router box transfer.

Oh, I tried rsync -e ssh too, same deal.

I would not be surprised if this was a network technology thing, as I use fixed WiMAX, but I work at my ISP and I am fairly certain they're not hosing anything normally. I've tried setting the MTU on the router, if I drop it to 100 I avoid the retransmissions and dup ACKs, but then throughput obviously suffers anyway.

Transfers over SSL (443) seemed fine when I did some testing with them. It's just SSH, so far. Thoughts or ideas?

----------

## infinite1der

Which side of the router are you setting the MTU on: internal or external interface? I would ASSume that the MTU of the external interface (WiMAX side) is set automagically. Your internal interface MTU should match the external interface MTU.

Anyways, if this were an MTU issue, you'd see fragmented packets in your sniffer. Offhand, I don't think scp specifies the NO_FRAG flag on its packets, but I'm not 100%. Retransmits are a sign of a poor connection. Are you seeing single packet retransmits or three-way handshake + packet?

--JamesT

----------

## Hu

Since you control some Gentoo systems elsewhere, could you temporarily make one of them have sshd listen on port 443?  Test doing an scp from your LAN to that other system.  If the problem is with your network configuration or ssh/sshd installation, you should see problems regardless of the port used.  If someone, whether at your ISP or higher up, is mangling traffic based on port number, ssh on port 443 should be smooth.

Also, have you tried doing a large HTTPS POST?  It could be that HTTPS seemed to work fine because it was not stressing the network, so it had the same mostly acceptable performance as ssh, rather than the terrible performance of scp.

----------

## SeaTiger

Test the ping between those syetem too. Try

```
echo 7 > /proc/sys/net/ipv4/tcp_window_scaling
```

and see if that help.

----------

## bssteph

 *infinite1der wrote:*   

> Which side of the router are you setting the MTU on: internal or external interface? I would ASSume that the MTU of the external interface (WiMAX side) is set automagically. Your internal interface MTU should match the external interface MTU.

 

Sorry, I should have been more specific. The WiMAX client unit is separate from my router box; as far as the router is concerned, both the internal and external networks are over Ethernet. They were 1500 by default. I was setting the MTUs on eth0, the WiMAX-facing side only, but my latest tests were from the router box itself and did not involve the internal network.

 *infinite1der wrote:*   

> Anyways, if this were an MTU issue, you'd see fragmented packets in your sniffer. Offhand, I don't think scp specifies the NO_FRAG flag on its packets, but I'm not 100%. Retransmits are a sign of a poor connection. Are you seeing single packet retransmits or three-way handshake + packet?

 

I believe merely single packet retransmits.

 *Hu wrote:*   

> Since you control some Gentoo systems elsewhere, could you temporarily make one of them have sshd listen on port 443? Test doing an scp from your LAN to that other system. If the problem is with your network configuration or ssh/sshd installation, you should see problems regardless of the port used. If someone, whether at your ISP or higher up, is mangling traffic based on port number, ssh on port 443 should be smooth.
> 
> Also, have you tried doing a large HTTPS POST? It could be that HTTPS seemed to work fine because it was not stressing the network, so it had the same mostly acceptable performance as ssh, rather than the terrible performance of scp.

 

I tried putting sshd on 443 and it was just as slow. I uploaded my testing MP3 over HTTPS, what was taking approx. 150 sec via SCP (regardless of port) took 10 with HTTPS.

 *junksiu wrote:*   

> Test the ping between those syetem too. Try
> 
> ```
> echo 7 > /proc/sys/net/ipv4/tcp_window_scaling
> ```
> ...

 

Ping is fine. Tried playing with the window scaling, didn't help. Also tried playing with various congestion control algorithms, didn't seem to make a difference.

--

I see now (with more reading) that the TCP duplicate ACKs are nothing out of the ordinary, so maybe that's a red herring in this entire thing.

I took a bunch of packet captures, and the pattern I'm seeing (thanks to Wireshark's "expert info") is:

For scp, the router will report some TCP fast retransmission warnings, but the receiving host will instead, throughout the entire transfer, report a fair number (132) of TCP "Previous segment lost". For HTTPS POST, the receiver will report a similar total number of assorted warnings (previous segment lost, unreassembled packet, out-of-order segment) while the router mainly sees unreassembled packet warnings, and vice versa for when I downloaded over HTTPS on the router from the Internet host.

A scp will have approx. 3200 packets, HTTPS POST 2900 (as seen by the receiver), so there's no huge difference in payload between the two.

So I'm still at a lost, but still interested in input. Thanks!

----------

## SeaTiger

Did you try the original ssh port 22 or some port other than 443?

Port 443 is for https(SSL). But scp actually is a ssh session with default port 22. Try any port other than port 80 and 443. There maybe some http cache/proxy service sitting between in your connection and distort the traffic.

----------

## bssteph

 *junksiu wrote:*   

> Did you try the original ssh port 22 or some port other than 443?
> 
> Port 443 is for https(SSL). But scp actually is a ssh session with default port 22. Try any port other than port 80 and 443. There maybe some http cache/proxy service sitting between in your connection and distort the traffic.

 

Yes, I know what the ports normally are for and what ssh (and scp) normally use, the playing around with ssl was just to a) rule out some generic "it's encrypted so I'm going to mess with it" behavior and b) rule out some generic "it's not plaintext so I can't handle it well" behavior and c) rule out some "I hate destination port 22" behavior. Router->host SSL GET/POST are both fine for large files, and others can do GETs of files over SSL on my router just fine too. 443 is not the problem.

To be clear, scp was just as bad when the destination port was 443 as it was when it was 22. Also, to be clear, external hosts scping to/from me (that is, hosts on the Internet connecting to me) suck just as hard as when I am the originator of the connection.

----------

