# [solved] Cannot ssh to other hosts after openssh-7.7_p1-r4

## quinn_jones

Hi all,

I ran an "emerge -uDNav world" this morning, which upgraded openssh to 7.7_p1-r4.  After the upgrade, I cannot make new connections to other hosts.  The error looks like this:

```
$ ssh $host

warning: agent returned different signature type ssh-rsa (expected rsa-sha2-512)

packet_write_wait: Connection to $ipaddr port 22: Broken pipe
```

I noticed that a USE flag changed during the upgrade, 'hpn' was removed.  It is not set in my USE flag either way, but I tried re-emerging with it explicitly enabled with the same result.

I am able to fix the problem by downgrading to the previous available version, net-misc/openssh-7.5_p1-r4.  Interestingly, the 'hpn' USE flag comes back in when I do so.

End points vary; I've tried RHEL 5 and 6, Solaris, another Gentoo box that hasn't upgraded, and localhost.  The only box that I can ssh into post-upgrade is localhost.

Executing ssh -v doesn't provide much illumination.  Things look pretty normal to my eye, but then again I don't use -v or examine the output very often.

```
$ ssh -v $host

OpenSSH_7.7p1, OpenSSL 1.0.2o  27 Mar 2018

debug1: Reading configuration data /home/me/.ssh/config

debug1: /home/me/.ssh/config line 19: Applying options for $host

debug1: /home/me/.ssh/config line 60: Applying options for *

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: Connecting to $host [$ipaddr] port 22.

debug1: Connection established.

debug1: identity file /home/me/.ssh/id_rsa type 0

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_rsa-cert type -1

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_dsa type -1

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_dsa-cert type -1

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_ecdsa type -1

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_ecdsa-cert type -1

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_ed25519 type -1

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_ed25519-cert type -1

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_xmss type -1

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_xmss-cert type -1

debug1: Local version string SSH-2.0-OpenSSH_7.7

debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3

debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000

debug1: Authenticating to $host:22 as 'me'

debug1: SSH2_MSG_KEXINIT sent

debug1: SSH2_MSG_KEXINIT received

debug1: kex: algorithm: diffie-hellman-group-exchange-sha256

debug1: kex: host key algorithm: ssh-rsa

debug1: kex: server->client cipher: aes128-ctr MAC: umac-64@openssh.com compression: none

debug1: kex: client->server cipher: aes128-ctr MAC: umac-64@openssh.com compression: none

debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<3072<8192) sent

debug1: got SSH2_MSG_KEX_DH_GEX_GROUP

debug1: SSH2_MSG_KEX_DH_GEX_INIT sent

debug1: got SSH2_MSG_KEX_DH_GEX_REPLY

debug1: Server host key: ssh-rsa SHA256:<redacted>

debug1: Host '$host' is known and matches the RSA host key.

debug1: Found key in /home/me/.ssh/known_hosts:16

debug1: rekey after 4294967296 blocks

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug1: SSH2_MSG_NEWKEYS received

debug1: rekey after 4294967296 blocks

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive

debug1: Next authentication method: publickey

debug1: Offering public key: RSA SHA256:8of0mv38rXmhTyZktCJQBNrr02MdphSkpKtg9iD3ZA8 /home/me/.ssh/id_rsa

debug1: Server accepts key: pkalg ssh-rsa blen 279

debug1: Authentication succeeded (publickey).

Authenticated to $host ([$ipaddr]:22).

debug1: channel 0: new [client-session]

debug1: Requesting no-more-sessions@openssh.com

debug1: Entering interactive session.

debug1: pledge: network

debug1: Requesting authentication agent forwarding.

debug1: Sending environment.

debug1: Sending env LANG = en_US.utf8

packet_write_wait: Connection to $ipaddr port 22: Broken pipe

$ 
```

I can provide more info, but this doesn't seem to be a compiling problem so much as a config problem.  I've made no changes to /etc/ssh/ssh_config, and the only additions I've made to ~/.ssh/config involve setting usernames and adding "ForwardAgent yes" and "TCPKeepAlive yes" under "Host *".

Finally, I've tried searching the forum and the googles, specifically for variations of "ssh packet_write_wait: connection port 22: broken pipe" but haven't found any (recent) hits or much of anything relevant.

Can anyone explain what's going on?Last edited by quinn_jones on Thu Jun 21, 2018 1:31 pm; edited 1 time in total

----------

## khayyam

quinn_jones ...

that just looks like the ssh-agent running was started prior to the update, and that you need to restart the session/shell/ssh-agent.

best ... khay

----------

## quinn_jones

Hi Khay,

Thanks for the reply, but (as usual) I failed to provide all of the information.  I also think you're concentrating on the wrong part of the issue.

What you didn't know, and don't see, is me logging out and rebooting while troubleshooting this myself.  I did verify that the agent was stopped, until it was clear to me that it wasn't the source of the problem.

The errors started immediately after the upgrade, without a reboot.  I rebooted and they persisted.  Cue a series of downgrades, upgrades, and recompiles with/without the 'hpn' USE flag and some more reboots just to be sure.

I cannot explain the warning you see, "agent returned different signature type ssh-rsa (expected rsa-sha2-512)", but it doesn't seem to affect anything.  I see the same warning when I log into localhost which, as mentioned, works.

I believe that the issue is with the new version of openssh, not the agent specifically.

----------

## Tony0945

Running the same version with, I think, the same flags. No problem. Everything seems to match expect I don't have the broken pipe. Is a firewall on either computer blocking port 22?

```
 # emerge -pv openssh

These are the packages that would be merged, in order:

Calculating dependencies... done!

[ebuild   R    ] net-misc/openssh-7.7_p1-r4::gentoo  USE="X pam pie ssl -X509 -audit   -bindist -debug -hpn -kerberos -ldap -ldns -libedit -libressl -livecd -sctp (-selinux) -skey -static {-test}" 0 KiB

```

----------

## Hu

Verbosity can be increased up to level 3.  OP: you ran only with level 1.  Do higher levels of verbosity help?  Does the server log anything interesting when it closes your connection?

----------

## toralf

a Gentoo dev blames this : https://github.com/rapier1/openssh-portable/commit/e3ea335abeab731c68f2b2141bee85a4b0bf680f

So try to add "-hpn" to make.conf and see if it helps.

OTOH: https://dev.gnupg.org/T3880Last edited by toralf on Sat Jun 16, 2018 1:27 pm; edited 1 time in total

----------

## Whissi

For the agent message, please see https://dev.gnupg.org/T3880

You probably have a bad agent. See https://bugs.gentoo.org/658038 for additional details.

----------

## russK

One of the posts to the bugs referenced seems apropos, to skip use of the agent, ssh directly:

```
ssh -vvv -a -i /path/to/your/key <user>@<host>
```

----------

## khayyam

 *russK wrote:*   

> One of the posts to the bugs referenced seems apropos, to skip use of the agent [...]

 

russK, et al ... I also updated to =net-misc/openssh-7.7_p1-r4 and can use ssh-agent without the reported issue:

```
[khayyam@local: ~]% ps auxwww | grep '[s]sh-agent'

khayyam  1033  0.0  0.0   4980   912 ?    Ss   Jun14   0:00 ssh-agent

[khayyam@local: ~]% ssh -V

OpenSSH_7.7p1, OpenSSL 1.0.2o  27 Mar 2018

[khayyam@local: ~]% ssh username@remote.org

[username@remote: ~]% ssh -V

OpenSSH_7.4p1 Debian-10+deb9u3, OpenSSL 1.0.2l  25 May 2017

[username@remote: ~]% exit

Connection to remote.org closed.

[khayyam@local: ~]% emerge -pvq =net-misc/openssh-7.7_p1-r4

[ebuild   R   ] net-misc/openssh-7.7_p1-r4  USE="pam pie ssl -X -X509 -audit -bindist -debug -hpn -kerberos -ldap -ldns -libedit -libressl -livecd -sctp (-selinux) -skey -static {-test}
```

That suggests that 'agent' probably means something other than /usr/bin/ssh-agent.

best ... khay

----------

## Whissi

 *khayyam wrote:*   

> That suggests that 'agent' probably means something other than /usr/bin/ssh-agent

 Yeah, any agent you are using to provide your key material.

----------

## quinn_jones

Hi all,

I may be getting closer to a solution, or maybe not a solution but an understanding.  I got home and upgraded one of my computers there.  The environments are similar (x86_64, MATE DE, profile v17.0, up-to-date with same-day "portage --sync" and "emerge -uDNav world") but not entirely the same software.  I purposely did not upgrade the public-facing Gentoo server that I tried yesterday so I could run tests.

On my home systems I do not get dumped, i.e. openssh-7.7_p1-r4 is not a problem.  That suggests that there something about the specific configuration on my work system (the one exhibiting the error).

u/Whissi: The agent warning still comes up, but connections don't fail.  That behavior corresponds to your link.  Thank you for the info.

u/Tony0945: thank you for the suggestion, but I'm pretty sure that a firewall isn't the issue.  Downgrading (without making other changes) temporarily solves the problem.

u/Hu: When I get back to my work computer next, I will try logging in again using -vvv and see if there are server-side errors.  Those are both great suggestions, thank you.

----------

## Tony0945

 *quinn_jones wrote:*   

> I got home and upgraded one of my computers there.  The environments are similar (x86_64, MATE DE, profile v17.0, up-to-date with same-day "portage --sync" and "emerge -uDNav world")

 

Which mirrors my setup also. So, something on the work system is different.  The work system is also Gentoo?

My system is OpenRC. If the work system is not Gentoo, i.e. redHat or Ubuntu, it is probably systemd. Keep that in mind, but first priority is Hu's advice to use -vvv for more details.

----------

## khayyam

 *quinn_jones wrote:*   

> On my home systems I do not get dumped, i.e. openssh-7.7_p1-r4 is not a problem.  That suggests that there something about the specific configuration on my work system (the one exhibiting the error).

 

quinn_jones ... ok, but as I show above I can't reproduce connecting to Debian w/ OpenSSH_7.4p1 using openssh-7.7_p1-r4.

best ... khay

----------

## bunder

I had this problem the other day on my laptop, I thought it was gnome-keyring gumming up the works, started working for me after killing it.

(why gnome-keyring for a text terminal using ssh compared to gpg, I don't know)

edit: personal opinion, but you want hpn, it makes ssh much faster for file transfers and the like.

----------

## khayyam

 *bunder wrote:*   

> (why gnome-keyring for a text terminal using ssh compared to gpg, I don't know)

 

bunder ... because higher level UI invariably negates lower level simplicity ... I can envisage the time when gnome-powershell will make such inconvenient distinctions disappear :)

 *bunder wrote:*   

> edit: personal opinion, but you want hpn, it makes ssh much faster for file transfers and the like.

 

True, but that makes me wonder why the useflag was flipped.

best ... khay

----------

## quinn_jones

Hi all,

Thank you for your help and input so far.

I'm back at work, on the "problem" machine, and can test some more.

To put one open question to rest: the system in question is running Gentoo.  I'm connecting to a number of other platforms, but the one having issues making those connections on openssh-7.7_p1-r4 is a Gentoo system.

Here's the results of ssh -vvv:

```
$ ssh -vvv extfound@$host

OpenSSH_7.7p1, OpenSSL 1.0.2o  27 Mar 2018

debug1: Reading configuration data /home/me/.ssh/config

debug1: /home/me/.ssh/config line 60: Applying options for *

debug1: Reading configuration data /etc/ssh/ssh_config

debug2: resolve_canonicalize: hostname $host is address

debug2: ssh_connect_direct: needpriv 0

debug1: Connecting to $host [$ipaddr] port 22.

debug1: Connection established.

debug1: identity file /home/me/.ssh/id_rsa type 0

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_rsa-cert type -1

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_dsa type -1

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_dsa-cert type -1

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_ecdsa type -1

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_ecdsa-cert type -1

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_ed25519 type -1

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_ed25519-cert type -1

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_xmss type -1

debug1: key_load_public: No such file or directory

debug1: identity file /home/me/.ssh/id_xmss-cert type -1

debug1: Local version string SSH-2.0-OpenSSH_7.7

debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3

debug1: match: OpenSSH_4.3 pat OpenSSH_2*,OpenSSH_3*,OpenSSH_4* compat 0x00000000

debug2: fd 3 setting O_NONBLOCK

debug1: Authenticating to $host:22 as 'extfound'

debug3: hostkeys_foreach: reading file "/home/me/.ssh/known_hosts"

debug3: record_hostkey: found key type RSA in file /home/me/.ssh/known_hosts:3

debug3: load_hostkeys: loaded 1 keys from $host

debug3: hostkeys_foreach: reading file "/home/me/.ssh/known_hosts2"

debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa

debug3: send packet: type 20

debug1: SSH2_MSG_KEXINIT sent

debug3: receive packet: type 20

debug1: SSH2_MSG_KEXINIT received

debug2: local client KEXINIT proposal

debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c

debug2: host key algorithms: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519

debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com

debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com

debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

debug2: compression ctos: none,zlib@openssh.com,zlib

debug2: compression stoc: none,zlib@openssh.com,zlib

debug2: languages ctos: 

debug2: languages stoc: 

debug2: first_kex_follows 0 

debug2: reserved 0 

debug2: peer server KEXINIT proposal

debug2: KEX algorithms: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

debug2: host key algorithms: ssh-rsa,ssh-dss

debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se

debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se

debug2: MACs ctos: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96

debug2: MACs stoc: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96

debug2: compression ctos: none,zlib@openssh.com

debug2: compression stoc: none,zlib@openssh.com

debug2: languages ctos: 

debug2: languages stoc: 

debug2: first_kex_follows 0 

debug2: reserved 0 

debug1: kex: algorithm: diffie-hellman-group-exchange-sha1

debug1: kex: host key algorithm: ssh-rsa

debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha1 compression: none

debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha1 compression: none

debug3: send packet: type 34

debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<7680<8192) sent

debug3: receive packet: type 31

debug1: got SSH2_MSG_KEX_DH_GEX_GROUP

debug2: bits set: 3051/6144

debug3: send packet: type 32

debug1: SSH2_MSG_KEX_DH_GEX_INIT sent

debug3: receive packet: type 33

debug1: got SSH2_MSG_KEX_DH_GEX_REPLY

debug1: Server host key: ssh-rsa SHA256:7bb8dV0w13d08GHxtcQmMPBNuZgNdS3h4W4wtmpBDQw

debug3: hostkeys_foreach: reading file "/home/me/.ssh/known_hosts"

debug3: record_hostkey: found key type RSA in file /home/me/.ssh/known_hosts:3

debug3: load_hostkeys: loaded 1 keys from $host

debug3: hostkeys_foreach: reading file "/home/me/.ssh/known_hosts2"

debug1: Host '$host' is known and matches the RSA host key.

debug1: Found key in /home/me/.ssh/known_hosts:3

debug2: bits set: 3079/6144

debug3: send packet: type 21

debug2: set_newkeys: mode 1

debug1: rekey after 4294967296 blocks

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug3: receive packet: type 21

debug1: SSH2_MSG_NEWKEYS received

debug2: set_newkeys: mode 0

debug1: rekey after 4294967296 blocks

debug2: key: /home/me/.ssh/id_rsa (0x564ef8e1acc0), agent

debug2: key: me@gentoo (0x564ef8e2cca0), agent

debug2: key: me@gentoo (0x564ef8e2cd10), agent

debug2: key: /home/me/.ssh/id_dsa ((nil))

debug2: key: /home/me/.ssh/id_ecdsa ((nil))

debug2: key: /home/me/.ssh/id_ed25519 ((nil))

debug2: key: /home/me/.ssh/id_xmss ((nil))

debug3: send packet: type 5

debug3: receive packet: type 6

debug2: service_accept: ssh-userauth

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug3: send packet: type 50

debug3: receive packet: type 51

debug1: Authentications that can continue: publickey,gssapi-with-mic,password,hostbased

debug3: start over, passed a different list publickey,gssapi-with-mic,password,hostbased

debug3: preferred publickey,keyboard-interactive,password

debug3: authmethod_lookup publickey

debug3: remaining preferred: keyboard-interactive,password

debug3: authmethod_is_enabled publickey

debug1: Next authentication method: publickey

debug1: Offering public key: RSA SHA256:8of0mv38rXmhTyZktCJQBNrr02MdphSkpKtg9iD3ZA8 /home/me/.ssh/id_rsa

debug3: send_pubkey_test

debug3: send packet: type 50

debug2: we sent a publickey packet, wait for reply

debug3: receive packet: type 60

debug1: Server accepts key: pkalg ssh-rsa blen 279

debug2: input_userauth_pk_ok: fp SHA256:8of0mv38rXmhTyZktCJQBNrr02MdphSkpKtg9iD3ZA8

debug3: sign_and_send_pubkey: RSA SHA256:8of0mv38rXmhTyZktCJQBNrr02MdphSkpKtg9iD3ZA8

debug3: send packet: type 50

debug3: receive packet: type 52

debug1: Authentication succeeded (publickey).

Authenticated to $host ([$ipaddr]:22).

debug1: channel 0: new [client-session]

debug3: ssh_session2_open: channel_new: 0

debug2: channel 0: send open

debug3: send packet: type 90

debug1: Entering interactive session.

debug1: pledge: network

debug3: receive packet: type 91

debug2: channel_input_open_confirmation: channel 0: callback start

debug1: Requesting authentication agent forwarding.

debug2: channel 0: request auth-agent-req@openssh.com confirm 0

debug3: send packet: type 98

debug2: fd 3 setting TCP_NODELAY

debug3: ssh_packet_set_tos: set IP_TOS 0x48

debug2: client_session2_setup: id 0

debug2: channel 0: request pty-req confirm 1

debug3: send packet: type 98

debug1: Sending environment.

debug3: Ignored env LS_COLORS

debug3: Ignored env XDG_MENU_PREFIX

debug1: Sending env LANG = en_US.utf8

debug2: channel 0: request env confirm 0

debug3: send packet: type 98

debug3: Ignored env LESS

debug3: Ignored env DISPLAY

debug3: Ignored env OPENGL_PROFILE

debug3: Ignored env CONFIG_PROTECT_MASK

debug3: Ignored env EDITOR

debug3: Ignored env GTK_OVERLAY_SCROLLING

debug3: Ignored env COLORTERM

debug3: Ignored env RELEASE_MASTER

debug3: Ignored env JAVA_HOME

debug3: Ignored env GCC_SPECS

debug3: Ignored env SSH_AUTH_SOCK

debug3: Ignored env EIX_LIMIT

debug3: Ignored env USER

debug3: Ignored env PAGER

debug3: Ignored env PWD

debug3: Ignored env MANPAGER

debug3: Ignored env HOME

debug3: Ignored env GSETTINGS_BACKEND

debug3: Ignored env XDG_DATA_DIRS

debug3: Ignored env MATE_DESKTOP_SESSION_ID

debug3: Ignored env FLTK_DOCDIR

debug3: Ignored env JDK_HOME

debug3: Ignored env GTK_MODULES

debug3: Ignored env MAIL

debug3: Ignored env CONFIG_PROTECT

debug3: Ignored env TERM

debug3: Ignored env SHELL

debug3: Ignored env GNOME_KEYRING_CONTROL

debug3: Ignored env JAVAC

debug3: Ignored env XDG_CURRENT_DESKTOP

debug3: Ignored env PERL5LIB

debug3: Ignored env DEVEL

debug3: Ignored env SHLVL

debug3: Ignored env MANPATH

debug3: Ignored env WINDOWID

debug3: Ignored env LOGNAME

debug3: Ignored env DBUS_SESSION_BUS_ADDRESS

debug3: Ignored env XAUTHORITY

debug3: Ignored env ANT_HOME

debug3: Ignored env XDG_CONFIG_DIRS

debug3: Ignored env PATH

debug3: Ignored env INFOPATH

debug3: Ignored env XDG_SESSION_COOKIE

debug3: Ignored env SESSION_MANAGER

debug3: Ignored env LESSOPEN

debug3: Ignored env _

debug2: channel 0: request shell confirm 1

debug3: send packet: type 98

debug2: channel_input_open_confirmation: channel 0: callback done

debug2: channel 0: open confirm rwindow 0 rmax 32768

debug3: send packet: type 1

packet_write_wait: Connection to $host port 22: Broken pipe
```

and here's what I see in the server logs (this happens to be a RHEL5 system, so this is in /var/log/secure):

 *Quote:*   

> Jun 18 07:25:47 fe sshd[30784]: Accepted publickey for $user from $ipaddr port 54755 ssh2
> 
> Jun 18 07:25:47 $host sshd[30784]: pam_unix(sshd:session): session opened for user $user by (uid=0)
> 
> Jun 18 07:25:47 $host sshd[30784]: pam_unix(sshd:session): session closed for user $user

 

Again, downgrading to openssh-7.5_p1-r4 solves the issue.

I will try rebuilding both MATE and gpg to see if an agent is causing the problem.

----------

## Whissi

 *Quote:*   

> $ ssh -vvv extfound@$host 

 ...you are not bypassing agent like we told you.

----------

## quinn_jones

Oops, you are correct.

Here's the diff of the two outputs; they're largely the same.

```
$ diff -uw out1 out2

--- out1   2018-06-18 08:56:35.878939402 -0400

+++ out2   2018-06-18 08:56:05.888939771 -0400

@@ -1,4 +1,4 @@

-$ ssh -vvv $user@$host

+ ssh -vvv -a -i ~/.ssh/id_rsa $user@$host

 OpenSSH_7.7p1, OpenSSL 1.0.2o  27 Mar 2018

 debug1: Reading configuration data /home/me/.ssh/config

 debug1: /home/me/.ssh/config line 60: Applying options for *

@@ -10,22 +10,6 @@

 debug1: identity file /home/me/.ssh/id_rsa type 0

 debug1: key_load_public: No such file or directory

 debug1: identity file /home/me/.ssh/id_rsa-cert type -1

-debug1: key_load_public: No such file or directory

-debug1: identity file /home/me/.ssh/id_dsa type -1

-debug1: key_load_public: No such file or directory

-debug1: identity file /home/me/.ssh/id_dsa-cert type -1

-debug1: key_load_public: No such file or directory

-debug1: identity file /home/me/.ssh/id_ecdsa type -1

-debug1: key_load_public: No such file or directory

-debug1: identity file /home/me/.ssh/id_ecdsa-cert type -1

-debug1: key_load_public: No such file or directory

-debug1: identity file /home/me/.ssh/id_ed25519 type -1

-debug1: key_load_public: No such file or directory

-debug1: identity file /home/me/.ssh/id_ed25519-cert type -1

-debug1: key_load_public: No such file or directory

-debug1: identity file /home/me/.ssh/id_xmss type -1

-debug1: key_load_public: No such file or directory

-debug1: identity file /home/me/.ssh/id_xmss-cert type -1

 debug1: Local version string SSH-2.0-OpenSSH_7.7

 debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3

 debug1: match: OpenSSH_4.3 pat OpenSSH_2*,OpenSSH_3*,OpenSSH_4* compat 0x00000000

@@ -74,7 +58,7 @@

 debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<7680<8192) sent

 debug3: receive packet: type 31

 debug1: got SSH2_MSG_KEX_DH_GEX_GROUP

-debug2: bits set: 3051/6144

+debug2: bits set: 3126/6144

 debug3: send packet: type 32

 debug1: SSH2_MSG_KEX_DH_GEX_INIT sent

 debug3: receive packet: type 33

@@ -86,7 +70,7 @@

 debug3: hostkeys_foreach: reading file "/home/me/.ssh/known_hosts2"

 debug1: Host '10.66.248.31' is known and matches the RSA host key.

 debug1: Found key in /home/me/.ssh/known_hosts:3

-debug2: bits set: 3079/6144

+debug2: bits set: 2997/6144

 debug3: send packet: type 21

 debug2: set_newkeys: mode 1

 debug1: rekey after 4294967296 blocks

@@ -96,13 +80,9 @@

 debug1: SSH2_MSG_NEWKEYS received

 debug2: set_newkeys: mode 0

 debug1: rekey after 4294967296 blocks

-debug2: key: /home/me/.ssh/id_rsa (0x564ef8e1acc0), agent

-debug2: key: me@gentoo (0x564ef8e2cca0), agent

-debug2: key: me@gentoo (0x564ef8e2cd10), agent

-debug2: key: /home/me/.ssh/id_dsa ((nil))

-debug2: key: /home/me/.ssh/id_ecdsa ((nil))

-debug2: key: /home/me/.ssh/id_ed25519 ((nil))

-debug2: key: /home/me/.ssh/id_xmss ((nil))

+debug2: key: /home/me/.ssh/id_rsa (0x55d5f3a9fcd0), explicit, agent

+debug2: key: me@gentoo (0x55d5f3ab1b80), agent

+debug2: key: me@gentoo (0x55d5f3ab1cc0), agent

 debug3: send packet: type 5

 debug3: receive packet: type 6

 debug2: service_accept: ssh-userauth

@@ -136,9 +116,6 @@

 debug1: pledge: network

 debug3: receive packet: type 91

 debug2: channel_input_open_confirmation: channel 0: callback start

-debug1: Requesting authentication agent forwarding.

-debug2: channel 0: request auth-agent-req@openssh.com confirm 0

-debug3: send packet: type 98

 debug2: fd 3 setting TCP_NODELAY

 debug3: ssh_packet_set_tos: set IP_TOS 0x48

 debug2: client_session2_setup: id 0
```

----------

## Whissi

So the agent problem/warning is gone but you still have problems?

Tell us about your setup:

Are ssh client machine and ssh host machine located in same network, i.e. a LAN connection?

If no LAN, tell us more about setup/involved routers

I.e. any special device like a "Hybrid router"?

Show us your /home/me/.ssh/config

You should start with a blank config and then add something like 

```
Host *

  ServerAliveInterval 10

  ServerAliveCountMax 6
```

for testing

----------

## quinn_jones

Hi u/Whissi,

Thank you for the quick follow-ups!

Here's the answers to your questions:

I've tried ssh'ing to hosts on the local LAN and over the internet.  The internet host is my home server, which is also running Gentoo but hasn't been upgraded yet.  (purposefully holding off while we troubleshoot this).  Other hosts are local.

I'm not at liberty to provide the LAN topology, as I'm on an enterprise-level network.  I don't know the particulars and I don't feel comfortable divulging output from nmap or even traceroutes.

I am running this particular instance as a VMware guest on a Win7 host, using NAT networking.  I'm not aware of any special or irregular equipment otherwise.

 my ssh config is pasted below.  I've redacted a long series of 'Host' and 'User' directives as they don't seem relevant and, like above, could be considered sensitive.  I swear that's all that's missing, though.

```
Host *

#    controlmaster auto

#    controlpath /tmp/ssh-%r@%h:%p

    ForwardAgent yes

    TCPKeepAlive yes
```

The commented lines were commented before this all started.

Other things to note about my particular setup

using MATE

openrc

17.0 profile, gcc 6.4.0

WMware guest (as noted above) using NAT and emulated Intel pro/1000 NIC; I also have app-emulation/open-vm-tools installed.

----------

## Whissi

We had a similar report https://bugs.gentoo.org/653960 but I cannot reproduce.

----------

## quinn_jones

Huh, that sounds pretty much like my situation.

Is there more information I can provide to help reproduce?

FWIW, mine is VMware Workstation 12 Player, version 12.5.9 build-07535481.  There are no pending updates.  I've checked both the VMware application log and the vm-specific log and I don't see anything specific to this; no coincident entries.

----------

## Tony0945

Just upgraded my own server and while doing etc-update I see this change in /etc/ssh/ssh_config

```
-#   Protocol 2,1

-#   Cipher 3des

-#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc

-#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160

+#   Protocol 2

```

 So by default Protocol 1 is now dropped. Also cipher changes Is this part of the problem?

in /etc/ssh/sshd_config

```
-PubkeyAuthentication no

+#PubkeyAuthentication yes

```

and 

```
-# the following are HPN related configuration options

-# tcp receive buffer polling. disable in non autotuning kernels

-#TcpRcvBufPoll yes

-

-# disable hpn performance boosts

-#HPNDisabled no

-

-# buffer size for hpn to non-hpn connections

-#HPNBufferSize 2048

-

-# allow the use of the none cipher

-#NoneEnabled no

-

```

I routinely discard changes to these configs  as I have enabled X forwarding and a few other things, so maybe this is why I don't have the problem.

----------

## Whissi

Please try setting IPQoS to a different value, i.e.

```
ssh -o IPQoS=lowdelay
```

or

```
ssh -o IPQoS=throughput
```

and tell us if this will fix your problem.

----------

## bunder

mine came back after a reboot, i'm almost definitely sure its gnome-keyring (i kill it then it stops).  the only problem is, i don't know how/where to clear out its entries or disable the ssh bits.

edit: logged out, killed gnome-keyring, deleted ~/.gnupg/* and ~/.local/share/keyrings/*, logged back in and i still get that signature error, argh

----------

## quinn_jones

Hi Whissi,

 *Whissi wrote:*   

> Please try setting IPQoS to a different value, i.e.
> 
> ```
> ssh -o IPQoS=lowdelay
> ```
> ...

 

That does indeed seem to fix the problem.  Either one works.  Thank you!

Is this considered an actual fix, or just part of the diagnosis?

----------

## Whissi

No, I am considering your problem (bunder) "fixed". I.e. your problem were/is caused by a bad agent.

I am especially talking to quinn_jones or anyone else with VMware and who cannot connect anymore since upgraded to Gentoo's OpenSSH 7.7p1 ebuild.

@ quinn_jones: Thanks for confirming. This is clearly a bug in VMware. Thanks to https://bugs.gentoo.org/658488#c3 I have seen captures showing RST packages from VMware when AF21/CS1 DSCP flags are used (this has nothing to do with OpenSSH, you can reproduce with everything.. you just need to set QoS marker to AF21/CS1 for something and VMware will kill that connection via RST package)

So if you have a valid VMware contract please file a bug against upstream. Would be nice if you could share ticket number.

This won't go away so you will need to keep IPQoS setting for now until VMware fixes their product.

----------

## khayyam

 *quinn_jones wrote:*   

> Is this considered an actual fix, or just part of the diagnosis?

 

quinn_jones ... a fix, because, if I understand what's happening, it's your router/gateway causing this, ssh sets a QoS (quality of service) packet on TCP, and that flag is probably blocked/interrupted. With 'lowdelay' or 'throughput' that is no longer the case (those are probably ignored). You can test that hypothesis by proxying ssh through netcat ('nc', 'ncat', or one of the other various "network swiss army knife" clients), ie (with net-analyzer/nmap[ncat]):

```
% ssh -o ProxyCommand="ncat %h %p" user@host
```

If that similarly passes through the gateway then it's probably the default QoS ssh sets and your fix is to set IPQoS in ~/.ssh/config, eg:

```
Host *

    IPQoS=throughput
```

I should ask, do you have access to the router/gateway?

best ... khay

----------

## bunder

 *Whissi wrote:*   

> No, I am considering your problem (bunder) "fixed". I.e. your problem were/is caused by a bad agent.

 

I don't mean to derail, but what am I supposed to do to fix it then?  I'm already using gnupg 2.2.8 (see bug post)

----------

## Whissi

 *bunder wrote:*   

> I don't mean to derail, but what am I supposed to do to fix it then?  I'm already using gnupg 2.2.8 (see bug post)

 I don't know your system. You first have to understand what's answering agent calls. I quoted the gnupg bug due to the quote of the problem. This doesn't only affect gnupg's agent. So there's a chance that you are using a different agent which also needs a fix. We would need to identify first  :Smile: 

Like said, you can test and confirm that it is an agent problem when you just use "ssh" and bypass any agent using arguments like shown.

----------

## bunder

it's probably too big to post here, so i threw it up on gists.

https://gist.github.com/bunder2015/ca72ce36c3a9d2a197595d67b6a6e74f

both systems are just regular desktops running mate.  i've never installed any ssh agents, except for gnome-keyring which was a dependancy.

----------

## Whissi

We don't have gnome-keyring with https://gitlab.gnome.org/GNOME/gnome-keyring/commit/35a01f8c6eaf3c991aaeb3f66449f41d3f0580bc in repository. I don't know how this program works, if it is hooking someway into ssh even if we call it with "-f" to bypass agent...

Can you make sure gnome-keyring isn't running when testing? Or maybe with some luck, throw the patch into /etc/portage/patches and recompile the package and hope that the patch will apply...?

----------

## bunder

sorry i should have grabbed that output too.  here it is without keyring running, no problems

https://gist.github.com/bunder2015/e3bcf3e256d791351e69f48d4a17f041

sadly the patch doesn't apply cleanly, i guess i'll wait for gnome-keyring to push a new update.  i don't really need it anyways.

----------

## quinn_jones

 *Whissi wrote:*   

> @ quinn_jones: Thanks for confirming. This is clearly a bug in VMware. Thanks to https://bugs.gentoo.org/658488#c3 I have seen captures showing RST packages from VMware when AF21/CS1 DSCP flags are used (this has nothing to do with OpenSSH, you can reproduce with everything.. you just need to set QoS marker to AF21/CS1 for something and VMware will kill that connection via RST package)
> 
> So if you have a valid VMware contract please file a bug against upstream. Would be nice if you could share ticket number.
> 
> This won't go away so you will need to keep IPQoS setting for now until VMware fixes their product.

 

Thank you very much, Whissi.

While my version of VMware player is licensed, I don't have a support contract.  I've filed https://communities.vmware.com/message/2778248 lets see if it goes anywhere.

I guess I'll mark this issue as solved since I have a good work-around and it's not really Gentoo's problem -- though it could be solved by setting a different QoS in the default config, that's a kludge and is messy for anyone *not* running under VMware player.

Thanks again!

Quinn

----------

## Whissi

 *bunder wrote:*   

> sorry i should have grabbed that output too.  here it is without keyring running, no problems
> 
> https://gist.github.com/bunder2015/e3bcf3e256d791351e69f48d4a17f041
> 
> sadly the patch doesn't apply cleanly, i guess i'll wait for gnome-keyring to push a new update.  i don't really need it anyways.

 Thanks for confirming that the issue is gone when gnome-keyring is dead, i.e. not interfering with ssh. I filed a bug against Gentoo's gnome-keyring package, https://bugs.gentoo.org/show_bug.cgi?id=658646

@ Quinn: Thank you for posting at least in VMware's forum.

----------

## bunder

great, thanks for the help.

----------

## 1clue

For those of you who are not yet fixed, I had a problem similar to this a year or two back. Don't remember the details except the error from the OP sounds very much like what I remember on my system. The subsequent investigation led me elsewhere though.

I don't intend for this post to derail the main discussion, just saying that for some of you it might actually be something else.

In my case, the remote hosts had had keys generated years before and the new local ssh version no longer supported those ciphers.

So you might consider regenerating ssh keys on one remote box to see if that fixes the issue.

----------

