# ssh fails after upgrading from 2.4 to 2.6 kernel

## curmudgeon

Strange as it sounds, everything else works fine after the

upgrade.

Two (Gentoo) machines each running 2.6.7-r12. Both

machines have identical kernel configurations (other than

minor hardware differences - such as the sound card).

I can ssh from machine one to machine two without problem,

I can (after the kernel upgrade) no longer ssh from machine

two to machine one. Both machines have identical files in

/etc/ssh (except for the host keys), and /etc/pam.d

Some output:

```

$ ssh 192.168.0.2

Permission denied, please try again.

Permission denied, please try again.

Permission denied (publickey,password,keyboard-interactive).

$

$ ssh -vv 192.168.0.2

OpenSSH_3.8.1p1, OpenSSL 0.9.7d 17 Mar 2004

debug1: Reading configuration data /etc/ssh/ssh_config

debug2: ssh_connect: needpriv 0

debug1: Connecting to 192.168.0.2 [192.168.0.2] port 22.

debug1: Connection established.

debug1: identity file /home/user/.ssh/identity type -1

debug1: identity file /home/user/.ssh/id_rsa type -1

debug1: identity file /home/user/.ssh/id_dsa type -1

debug1: Remote protocol version 2.0, remote software version OpenSSH_3.8.1p1

debug1: match: OpenSSH_3.8.1p1 pat OpenSSH*

debug1: Enabling compatibility mode for protocol 2.0

debug1: Local version string SSH-2.0-OpenSSH_3.8.1p1

debug1: SSH2_MSG_KEXINIT sent

debug1: SSH2_MSG_KEXINIT received

debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1

debug2: kex_parse_kexinit: ssh-rsa,ssh-dss

debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr

debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr

debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96

debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96

debug2: kex_parse_kexinit: none,zlib

debug2: channel 0: window 32631 sent adjust 32905

debug2: kex_parse_kexinit: none,zlib

debug2: kex_parse_kexinit:

debug2: kex_parse_kexinit:

debug2: kex_parse_kexinit: first_kex_follows 0

debug2: kex_parse_kexinit: reserved 0

debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1

debug2: kex_parse_kexinit: ssh-rsa,ssh-dss

debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr

debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr

debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96

debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96

debug2: kex_parse_kexinit: none,zlib

debug2: kex_parse_kexinit: none,zlib

debug2: kex_parse_kexinit:

debug2: kex_parse_kexinit:

debug2: kex_parse_kexinit: first_kex_follows 0

debug2: kex_parse_kexinit: reserved 0

debug2: mac_init: found hmac-md5

debug1: kex: server->client aes128-cbc hmac-md5 none

debug2: mac_init: found hmac-md5

debug1: kex: client->server aes128-cbc hmac-md5 none

debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP

debug2: dh_gen_key: priv key bits set: 119/256

debug2: bits set: 521/1024

debug1: SSH2_MSG_KEX_DH_GEX_INIT sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY

debug1: Host '192.168.0.2' is known and matches the RSA host key.

debug1: Found key in /home/user/.ssh/known_hosts:1

debug2: bits set: 512/1024

debug1: ssh_rsa_verify: signature correct

debug2: kex_derive_keys

debug2: set_newkeys: mode 1

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug2: set_newkeys: mode 0

debug1: SSH2_MSG_NEWKEYS received

debug1: SSH2_MSG_SERVICE_REQUEST sent

debug2: service_accept: ssh-userauth

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug2: key: /home/user/.ssh/identity ((nil))

debug2: key: /home/user/.ssh/id_rsa ((nil))

debug2: key: /home/user/.ssh/id_dsa ((nil))

debug1: Authentications that can continue: publickey,password,keyboard-interactive

debug1: Next authentication method: publickey

debug1: Trying private key: /home/user/.ssh/identity

debug1: Trying private key: /home/user/.ssh/id_rsa

debug1: Trying private key: /home/user/.ssh/id_dsa

debug2: we did not send a packet, disable method

debug1: Next authentication method: keyboard-interactive

debug2: userauth_kbdint

debug2: we sent a keyboard-interactive packet, wait for reply

debug2: input_userauth_info_req

debug2: input_userauth_info_req: num_prompts 1

debug1: Authentications that can continue: publickey,password,keyboard-interactive

debug2: userauth_kbdint

debug2: we sent a keyboard-interactive packet, wait for reply

debug2: input_userauth_info_req

debug2: input_userauth_info_req: num_prompts 1

debug1: Authentications that can continue: publickey,password,keyboard-interactive

debug2: userauth_kbdint

debug2: we sent a keyboard-interactive packet, wait for reply

debug2: input_userauth_info_req

debug2: input_userauth_info_req: num_prompts 1

debug1: Authentications that can continue: publickey,password,keyboard-interactive

debug2: we did not send a packet, disable method

debug1: Next authentication method: password

debug2: we sent a password packet, wait for reply

debug1: Authentications that can continue: publickey,password,keyboard-interactive

Permission denied, please try again.

debug2: we sent a password packet, wait for reply

debug1: Authentications that can continue: publickey,password,keyboard-interactive

Permission denied, please try again.

debug2: we sent a password packet, wait for reply

debug1: Authentications that can continue: publickey,password,keyboard-interactive

debug2: we did not send a packet, disable method

debug1: No more authentication methods to try.

Permission denied (publickey,password,keyboard-interactive).

$
```

And from /var/log/messages on machine one (timestamps removed)::

```

host1 sshd(pam_unix)[9864]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=host2.localdomain.loc  user=user

host1 sshd[9862]: error: PAM: Authentication failure for user from host2.localdomain.loc

host1 sshd(pam_unix)[9865]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=host2.localdomain.loc  user=user

host1 sshd[9862]: error: PAM: Authentication failure for user from host2.localdomain.loc

host1 sshd(pam_unix)[9866]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=host2.localdomain.loc  user=user

host1 sshd[9862]: error: PAM: Authentication failure for user from host2.localdomain.loc

host1 sshd[9862]: Failed keyboard-interactive/pam for user from 192.168.0.3 port 36603 ssh2

host1 sshd[9862]: Failed password for user from 192.168.0.3 port 36603 ssh2

host1 sshd[9862]: Failed password for user from 192.168.0.3 port 36603 ssh2

host1 sshd[9862]: Failed password for user from 192.168.0.3 port 36603 ssh2

```

This has me baffled. Thank you for your help.

----------

## blackhorse

I have no clue if this will work, but have you tried changing passwords  :Confused:   Hope this helps or that some one else knows the answer.

----------

## curmudgeon

Forgot to mention, I can do ssh localhost from machine one,

and it works fine. I just can't get there (for some reason)

from machine two.

----------

## curmudgeon

I have done some more testing which basically confirms

what I said before. I can't ssh anywhere from the machine

that I updated the kernel on.

----------

## Raffi

With the pam failure, you should make sure that you are meeting all the requirements in your /etc/pam.d/ssh file on the target machines. The one that I see problems with the most often is the users shell not being listed in /etc/shells.

You can also try re-emerging ssh.

----------

## curmudgeon

 *Raffi wrote:*   

> 
> 
> With the pam failure, you should make sure that you are
> 
> meeting all the requirements in your /etc/pam.d/ssh file
> ...

 

I don't see /etc/pam/ssh, only /etc/pam/sshd:

```

#%PAM-1.0

auth       required     pam_stack.so service=system-auth

auth       required     pam_shells.so

auth       required     pam_nologin.so

account    required     pam_stack.so service=system-auth

password   required     pam_stack.so service=system-auth

session    required     pam_stack.so service=system-auth
```

 *Raffi wrote:*   

> 
> 
> The one that I see problems with the most often
> 
> is the users shell not being listed in /etc/shells.

 

I have bash everywhere. Certainly that didn't change on

any of the target machines (and /bin/bash still appears in

/etc/shells on the affected machine.

 *Raffi wrote:*   

> 
> 
> You can also try re-emerging ssh.

 

This looks like the next step. I will try it when I get a chance.

----------

## Raffi

 *curmudgeon wrote:*   

> 
> 
> ```
> 
> debug1: Next authentication method: keyboard-interactive
> ...

 

I just noticed this. ssh should have stopped at this point at prompted you for a password. Sounds like ssh is compile with a different pty mechanism than your kernel is currently supporting. A re-emerge really might do the trick. If not, you will need to play with the pty options in your kernel.

----------

## curmudgeon

An strace helped to locate the source of the problem:

/dev/tty has permissionss 660 on the machine after the

kernel upgrade, and 666 on every other machine I

have looked at.

New questions:

1. Why did this change when I upgraded the kernel?

2. Can someone explain the implications of changing

the permissions, and the BEST way to make the

change permanent?

----------

## devon

Are you using udev and not devfs? See bug 53292 for some information.

----------

## curmudgeon

> Are you using udev and not devfs? See bug 53292 for some information.

Yes, I changed to udev when I upgraded the kernel.

Thank you for the additional information.

I can't believe that they fixed this a month ago, but never

marked any fixed version as stable, so that this bug keeps

catching new people. No wonder they have five duplicate

bug reports for problems caused by this!

----------

