# ssh closes after authentication, reset by peer [RESOLVED]

## Jrauch

Hey all,

I've got a really strange ssh issue.  When I'm connecting to one of my Gentoo machines, dosen't matter what from, my ssh connection is reset by peer after authentication is sucessful.  Here is the really wierd part.  If I remove all of the ssh keys from my ~/.ssh folder it logs in fine; remember I'm not using the keys when it fails either.

I'm running openssh-4.2_p1 on the Gentoo box, and until a few days ago this was working perfectly as a backup server with keyed logins for rsync.  At the moment I'm doing this by hand nightly, which isn't much fun to say the least.

As for network conectivity, both machines are on a internal 192.168.0.0/24 net, no address translation or anything like that.  I've removed the firewall from the while working on this, so that isn't a factor.

I've tired a few different versions of openssh, even building it by hand and running it on an alternate port for debuging (see below).  I've made new keys on both machines.  I even installed telnet so I could stop sshd entirely and remove all traces of it, then reinstall.  I've rebuilt ssh with and without pam, sftplogging, and tcpd.  

I even took a quick try with ssh.com's server, it failed too, but I don't know it well enough to make that really useful to this.

Here are the debuging logs from the server, and the verbose client logins.  I guess I'll keep this at level 2 for the moment.

client side, no verbosity:

```

ssh jrauch@192.168.0.254

Password:

Read from remote host 192.168.0.254: Connection reset by peer

Connection to 192.168.0.254 closed.
```

Client side Level 2 verbosity:

```

root]# ssh jrauch@192.168.0.254 -vv

OpenSSH_3.6.1p2, SSH protocols 1.5/2.0, OpenSSL 0x0090701f

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: Applying options for *

debug1: Rhosts Authentication disabled, originating port will not be trusted.

debug2: ssh_connect: needpriv 0

debug1: Connecting to 192.168.0.254 [192.168.0.254] port 22.

debug1: Connection established.

debug2: key_type_from_name: unknown key type '1024'

debug1: identity file /root/.ssh/identity type -1

debug1: identity file /root/.ssh/id_rsa type -1

debug2: key_type_from_name: unknown key type '-----BEGIN'

debug2: key_type_from_name: unknown key type '-----END'

debug1: identity file /root/.ssh/id_dsa type 2

debug1: Remote protocol version 2.0, remote software version OpenSSH_4.2

debug1: match: OpenSSH_4.2 pat OpenSSH*

debug1: Enabling compatibility mode for protocol 2.0

debug1: Local version string SSH-2.0-OpenSSH_3.6.1p2

debug1: SSH2_MSG_KEXINIT sent

debug1: SSH2_MSG_KEXINIT received

debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1

debug2: kex_parse_kexinit: ssh-rsa,ssh-dss

debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se

debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se

debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96

debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96

debug2: kex_parse_kexinit: none,zlib

debug2: kex_parse_kexinit: none,zlib

debug2: kex_parse_kexinit:

debug2: kex_parse_kexinit:

debug2: kex_parse_kexinit: first_kex_follows 0

debug2: kex_parse_kexinit: reserved 0

debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

debug2: kex_parse_kexinit: ssh-rsa,ssh-dss

debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr

debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr

debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96

debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96

debug2: kex_parse_kexinit: none,zlib@openssh.com

debug2: kex_parse_kexinit: none,zlib@openssh.com

debug2: kex_parse_kexinit:

debug2: kex_parse_kexinit:

debug2: kex_parse_kexinit: first_kex_follows 0

debug2: kex_parse_kexinit: reserved 0

debug2: mac_init: found hmac-md5

debug1: kex: server->client aes128-cbc hmac-md5 none

debug2: mac_init: found hmac-md5

debug1: kex: client->server aes128-cbc hmac-md5 none

debug1: SSH2_MSG_KEX_DH_GEX_REQUEST sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP

debug2: dh_gen_key: priv key bits set: 135/256

debug2: bits set: 1025/2048

debug1: SSH2_MSG_KEX_DH_GEX_INIT sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY

debug1: Host '192.168.0.254' is known and matches the RSA host key.

debug1: Found key in /root/.ssh/known_hosts:22

debug2: bits set: 1004/2048

debug1: ssh_rsa_verify: signature correct

debug2: kex_derive_keys

debug2: set_newkeys: mode 1

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug2: set_newkeys: mode 0

debug1: SSH2_MSG_NEWKEYS received

debug1: SSH2_MSG_SERVICE_REQUEST sent

debug2: service_accept: ssh-userauth

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug1: Authentications that can continue: publickey,keyboard-interactive

debug1: Next authentication method: publickey

debug1: Trying private key: /root/.ssh/identity

debug1: Trying private key: /root/.ssh/id_rsa

debug1: Offering public key: /root/.ssh/id_dsa

debug2: we sent a publickey packet, wait for reply

debug1: Authentications that can continue: publickey,keyboard-interactive

debug2: we did not send a packet, disable method

debug1: Next authentication method: keyboard-interactive

debug2: userauth_kbdint

debug2: we sent a keyboard-interactive packet, wait for reply

debug2: input_userauth_info_req

debug2: input_userauth_info_req: num_prompts 1

Password:

debug2: input_userauth_info_req

debug2: input_userauth_info_req: num_prompts 0

debug1: Authentication succeeded (keyboard-interactive).

debug1: channel 0: new [client-session]

debug2: channel 0: send open

debug1: Entering interactive session.

debug1: channel_free: channel 0: client-session, nchannels 1

Read from remote host 192.168.0.254: Connection reset by peer

Connection to 192.168.0.254 closed.

debug1: Transferred: stdin 0, stdout 0, stderr 100 bytes in 0.0 seconds

debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 2591.6

debug1: Exit status -1
```

Ok, now the server side debuging.

Info level

```

Dec 19 16:46:19 gentoo sshd[26348]: Accepted keyboard-interactive/pam for jrauch from 192.168.0.129 port 44809 ssh2
```

Debug level2

```

Dec 19 16:51:03 gentoo sshd[26274]: Received signal 15; terminating.

Dec 19 16:51:04 gentoo sshd[26489]: debug2: fd 3 setting O_NONBLOCK

Dec 19 16:51:04 gentoo sshd[26489]: debug1: Bind to port 22 on 0.0.0.0.

Dec 19 16:51:04 gentoo sshd[26489]: Server listening on 0.0.0.0 port 22.

Dec 19 16:51:04 gentoo sshd[26489]: socket: Address family not supported by protocol

Dec 19 16:51:38 gentoo sshd[26497]: debug1: rexec start in 4 out 4 newsock 4 pipe 6 sock 7

Dec 19 16:51:38 gentoo sshd[26489]: debug1: Forked child 26497.

Dec 19 16:51:38 gentoo sshd[26497]: debug1: inetd sockets after dupping: 3, 3

Dec 19 16:51:38 gentoo sshd[26497]: Connection from 192.168.0.129 port 47637

Dec 19 16:51:38 gentoo sshd[26497]: debug1: Client protocol version 2.0; client software version OpenSSH_3.6.1p2

Dec 19 16:51:38 gentoo sshd[26497]: debug1: match: OpenSSH_3.6.1p2 pat OpenSSH_3.*

Dec 19 16:51:38 gentoo sshd[26497]: debug1: Enabling compatibility mode for protocol 2.0

Dec 19 16:51:38 gentoo sshd[26497]: debug1: Local version string SSH-2.0-OpenSSH_4.2

Dec 19 16:51:38 gentoo sshd[26497]: debug2: fd 3 setting O_NONBLOCK

Dec 19 16:51:38 gentoo sshd[26497]: debug2: Network child is on pid 26501

Dec 19 16:51:38 gentoo sshd[26497]: debug2: monitor_read: 0 used once, disabling now

Dec 19 16:51:38 gentoo sshd[26497]: debug2: monitor_read: 4 used once, disabling now

Dec 19 16:51:38 gentoo sshd[26497]: debug2: monitor_read: 6 used once, disabling now

Dec 19 16:51:38 gentoo sshd[26497]: debug1: PAM: initializing for "jrauch"

Dec 19 16:51:38 gentoo sshd[26497]: debug1: PAM: setting PAM_RHOST to "drop1"

Dec 19 16:51:38 gentoo sshd[26497]: debug1: PAM: setting PAM_TTY to "ssh"

Dec 19 16:51:38 gentoo sshd[26497]: debug2: monitor_read: 45 used once, disabling now

Dec 19 16:51:38 gentoo sshd[26497]: debug2: monitor_read: 3 used once, disabling now

Dec 19 16:51:38 gentoo sshd[26497]: debug1: temporarily_use_uid: 1000/100 (e=0/0)

Dec 19 16:51:38 gentoo sshd[26497]: debug1: trying public key file /home/jrauch/.ssh/authorized_keys

Dec 19 16:51:38 gentoo sshd[26497]: debug1: restore_uid: 0/0

Dec 19 16:51:38 gentoo sshd[26497]: debug1: temporarily_use_uid: 1000/100 (e=0/0)

Dec 19 16:51:38 gentoo sshd[26497]: debug1: trying public key file /home/jrauch/.ssh/authorized_keys2

Dec 19 16:51:38 gentoo sshd[26497]: debug1: restore_uid: 0/0

Dec 19 16:51:41 gentoo sshd[26497]: debug2: PAM: sshpam_respond entering, 1 responses

Dec 19 16:51:41 gentoo sshd[26502]: debug1: do_pam_account: called

Dec 19 16:51:41 gentoo sshd[26497]: debug1: PAM: num PAM env strings 0

Dec 19 16:51:41 gentoo sshd[26497]: debug2: PAM: sshpam_respond entering, 0 responses

Dec 19 16:51:41 gentoo sshd[26497]: debug2: monitor_read: 54 used once, disabling now

Dec 19 16:51:41 gentoo sshd[26497]: debug1: do_pam_account: called

Dec 19 16:51:41 gentoo sshd[26497]: Accepted keyboard-interactive/pam for jrauch from 192.168.0.129 port 47637 ssh2

Dec 19 16:51:41 gentoo sshd[26497]: debug1: monitor_child_preauth: jrauch has been authenticated by privileged process

Dec 19 16:51:41 gentoo sshd[26497]: debug2: mac_init: found hmac-md5

Dec 19 16:51:41 gentoo sshd[26497]: debug2: mac_init: found hmac-md5

Dec 19 16:51:41 gentoo sshd[26497]: debug2: User child is on pid 26503

Dec 19 16:51:41 gentoo sshd[26497]: debug1: do_cleanup

Dec 19 16:51:41 gentoo sshd[26497]: debug1: PAM: cleanup
```

The server's name isn't really Gentoo... I'm sure you understand.

and to anyone that made it through that, thanks.

I've tried everything I can think of with no luck, so any ideas are welcome.Last edited by Jrauch on Wed Dec 21, 2005 4:05 pm; edited 1 time in total

----------

## buzzin

I had this issue and it turned out to be that the host that was being rejected had somehow got listed in /etc/hosts.deny on the ssh server. 

Maybe a quick check of that file will sort it for you?

Hope that helps.

----------

## Jrauch

Good thought, but those files don't even exist.

I even tried building without tcpd, which should completely disable those even if they did exist.

----------

## Jrauch

Any way this could be related to pam or another enviromental variable?

Does anyone know a way to check and see if pam is having an issue that could be causing this?

----------

## Jrauch

Ok, I fixed the issue this morning.

It turns out that nss_ldap was the cause.

I removed all ldap entries from /etc/nssswitch.conf and everything started to work as it did before.  I suspect I caused the issue when I removed the servers from /etc/ldap.conf, but the libraries were still being called.

I still think it's wierd as I was using all local accounts for ssh, but that's the fix.

----------

## buzzin

glad to hear you sorted it, just sorry we couldnt have been more help.  :Wink: 

----------

## Jrauch

Thanks.  I still apriciate the advice.

Hopefuly this will do someone else a little good if they end up with the same thing.

----------

## HeXiLeD

I am having  the same problem and i dont have /etc/nssswitch.conf  or /etc/ldap.conf.

I only have /etc/openldap/ldap.conf and it has all it's lines #commented

I have no idea why. 

```

sshd[7417]: Server listening on <ip> port <port>.

sshd[7432]: Accepted password for <username> from <ip> <port>  ssh2

login(pam_unix)[10247]: session opened for user <username> by (uid=0)

login[10284]: bad user ID `1000' for user `<username>': Illegal seek

login(pam_unix)[10247]: session closed for user <username>
```

----------

## Jrauch

 *Quote:*   

> bad user ID `1000' for user `<username>': Illegal seek

 

That almost seems like it might be disk related to me.  Could /etc/passwd be corrupt, or not mounted properly?

Anything in dmesg that would support disk problems?

----------

## HeXiLeD

i have found the solution here:

https://forums.gentoo.org/viewtopic-t-107518-highlight-bash+fork+resource.html

https://forums.gentoo.org/viewtopic-t-164479-highlight-bash+fork+resource.html

https://forums.gentoo.org/viewtopic-t-311566-highlight-bash+fork+resource.html

at least now i can login with ssh. But i still have general issues with this bash fork problem, even after checking /etc/security/limits.conf and /etc/limits FAG conf files.

basicly 'something' is forcing my user to run only a certain number of process and the fag thing about this is now that  i have set UNLIMITED FOR MY USER i can only run around 19.

After that number  i get "resource unavailable". 

The stupid thing about this is that  if i comment all lines in those to conf files, i can only less than 10 process for my user.

If i set my user to use unlimited  i get less then 20.

This only happened after i decided to upgrade to gcc 4.1 and emerge -e system and emerge -e world.( and yes i had done -e system and -e world before with no issues )

My congrats  to the smart mind behind whatever he decided to change for gentoo security   :Evil or Very Mad:  making me lose hours of time that i dont have.

----------

## Jrauch

I'm glad you found it, I know how sick of looking at ssh debuging I was.

It looks like it's a gresec/pam thing in the kernel, not really a Gentoo thing?

Either way, I've never played with those limits.

----------

## HeXiLeD

I will continue this topic here:

https://forums.gentoo.org/viewtopic-t-498020-start-0-postdays-0-postorder-asc-highlight-.html

----------

