# auth on server is failing

## slaterson

i've had a server running for nearly 10 years, hosting my ldap db (among other things).  today, i did a typical update that i do almost daily (this time it was about 4 days worth of portage updates) and i can no longer login remotely, nor locally.  i noticed nss_ldap was being updated, however lost the other updates.

good news: i have a remote shell open via ssh as root, so i can make updates to config, etc.

bad news: i am not able to login remotely or locally, even as root, to the server. prior to the update today i was able to login locally as anyone, and remotely as anyone except root (su after remote login).

i have reviewed my openldap config and nsswitch.conf.  tried a couple tweaks, it all looks sane and matches client config.  i'm not sure what else to do other than reboot the server, which may result in me being completely locked out of the server.

i need help.  this server has several terabytes of direct attached data.  it's a home server - photos, media, etc...

----------

## CooSee

did you also emerged ' sys-apps/shadow ' to 4.9-r1 ?

i got a problem with this version on fresh installed gentoo system couple days ago.

downgrade sys-apps/shadow to 4.8.1-r4.

good luck

----------

## Hu

When authentication fails, what messages are written to the relevant logs?

----------

## slaterson

 *CooSee wrote:*   

> did you also emerged ' sys-apps/shadow ' to 4.9-r1 ?
> 
> i got a problem with this version on fresh installed gentoo system couple days ago.
> 
> downgrade sys-apps/shadow to 4.8.1-r4.
> ...

 

sys-apps/shadow was never updated, stayed at 4.8.1-r4.  i rebuilt it anyway, and issue persists.

----------

## slaterson

 *Hu wrote:*   

> When authentication fails, what messages are written to the relevant logs?

 

in /var/log/auth.log, when trying to su - <user>, i get 

```
No passwd entry for user 'chris'
```

and... when i tried to login locally as root just now (after rebuilding all packages last night) i am able to successfully.  on the server, i am not able to login as any user in my ldap directory, though.

----------

## Hu

Does chris exist locally on the failed system or is that user only available through ldap?  If chris is through LDAP, what messages are written when you try a console login using a non-LDAP user, preferably root?  My inclination is to first try to repair your ability to get a new root shell, then look at getting LDAP in order, once you have the luxury of starting new root shells at will.

If this is not a systemd-based system, you could try restarting sshd.  (An old news item leads me to believe that, on at least some systemd-based systems, restarting sshd may kill your connection, and since you cannot get a new one, we want to avoid any risk of killing the existing one.)

----------

## slaterson

 *Hu wrote:*   

> Does chris exist locally on the failed system or is that user only available through ldap?  If chris is through LDAP, what messages are written when you try a console login using a non-LDAP user, preferably root?  My inclination is to first try to repair your ability to get a new root shell, then look at getting LDAP in order, once you have the luxury of starting new root shells at will.

 

chris does not exist locally on the server, only in the ldap directory.  i can login locally as root, and i can change sshd_config to allow remote root login (prefer to not to, of course).

auth.log messages when logging as root (successful):

```
Aug 15 11:00:46 bonobo login[28295]: pam_unix(login:session): session opened for user root(uid=0) by LOGIN(uid=0)

Aug 15 11:00:46 bonobo login[4375]: ROOT LOGIN  on '/dev/tty1'
```

 *Quote:*   

> If this is not a systemd-based system, you could try restarting sshd.  (An old news item leads me to believe that, on at least some systemd-based systems, restarting sshd may kill your connection, and since you cannot get a new one, we want to avoid any risk of killing the existing one.)

 

(server is openrc, no systemd) i tried restarting sshd at the server (locally logged in).  my open remote didn't die, and otherwise no other changes.

if i'm not able to login to the server as an ldap only user, imo definitely points to an ldap issue.  the odd thing, i logged in to the server remotely yesterday (as chris), su to root, then updated the system.  after updating, i tried to remote login to the server in a new session (do to something else) and it failed.  has to be something in the update.

i have tried re-emerging nss_ldap, pam_ldap, and shadow.  no luck yet.  i have also reviewed nsswitch.conf, ldap.conf, slapd.conf, and pam.d/system-auth configuration files.  nothing seems wrong compared to working client configurations.

fwiw, here is the full log entry when trying to login as an ldap only user:

```
Aug 15 11:14:16 bonobo login[4896]: pam_unix(login:auth): check pass; user unknown

Aug 15 11:14:16 bonobo login[4896]: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/tty1 ruser= rhost= 

Aug 15 11:14:16 bonobo login[4896]: pam_faillock(login:auth): User unknown

Aug 15 11:14:16 bonobo last message buffered 1 times

Aug 15 11:14:19 bonobo login[4896]: FAILED LOGIN (1) on '/dev/tty1' FOR 'UNKNOWN', Authentication failure
```

edit:

```
getent passwd
```

does NOT include any ldap users.  on clients, it does.

```
slapcat
```

returns everything from the ldap server, just like it should.

----------

## Hu

I agree, this looks like an LDAP issue.  In your opening post, you said you could not log in remotely or locally, even as root.  From that, I focused on fixing that problem so that you would have the ability to administer the system even if your leftover ssh session was terminated.  Logging in locally as root may be less convenient, and I can understand why you want to avoid allowing remote root login, but at least having those as options means that fixing this after that leftover session terminates will not require a liveCD environment.

Everything you've shown leads me to believe that the system no longer lists LDAP users, and your problems flow from there.  Unfortunately, I don't know how to turn that observation into further debugging advice.  You've already checked all the things I would have suggested.

----------

## slaterson

 *Hu wrote:*   

> I agree, this looks like an LDAP issue.  In your opening post, you said you could not log in remotely or locally, even as root.  From that, I focused on fixing that problem so that you would have the ability to administer the system even if your leftover ssh session was terminated.  Logging in locally as root may be less convenient, and I can understand why you want to avoid allowing remote root login, but at least having those as options means that fixing this after that leftover session terminates will not require a liveCD environment.
> 
> Everything you've shown leads me to believe that the system no longer lists LDAP users, and your problems flow from there.  Unfortunately, I don't know how to turn that observation into further debugging advice.  You've already checked all the things I would have suggested.

 

thanks for the thoughts thus far, much appreciated.  and to be sure it's clear, immediately following the update yesterday i was not able to login locally even as root.  now that i can, i feel a little safer in rebooting.  still really odd why ldap is not working.  i may reboot later today, or later this week, depending on how much time i on my hands to go through emergency recovery.

this server houses home directories for four clients on the home network, if auth to the clients starts failing i'm in trouble and will need to make local users/groups that match the server.

----------

## alamahant

Are you using 

nss-pam-ldapd

or

sssd

?

sssd is infinitely superior.

It is pam issue most probably or a cert issue

Plz post the output of

```

cat /etc/pam.d/system-auth

cat /etc/nsswitch.conf

```

from the client.

Also in the client 

/etc/openldap/ldap.conf

append

```

TLS_CACERT /etc/ssl/certs/ca-certificates.crt

TLS_REQCERT allow

```

What is the output from the client(and from the server)

```

ldapsearch -x -D cn=Manager,dc=my,dc=domain -b dc=my,dc=domain -H ldap://<fqdn-or-IP-of-ldap-server>/ -W

ldapsearch -x -D cn=Manager,dc=my,dc=domain -b dc=my,dc=domain -H ldaps://<fqdn-or-IP-of-ldap-server>/ -W

```

Also if firewall plz open 389 and 636 ports.

if selinux plz set permissive.

----------

## slaterson

i just masked sys-auth/nss_ldap-265-r9, re-emerged sys-auth/nss_ldap-265-r5 (previous version) and all is well again.  at least on my system, -r9 breaks the ldap config, so i'll be leaving it masked.

----------

