# last|head ssh entry and problems , GLSA Support pls

## dalu

As posted on Google+, perhaps I'll be more lucky here

Right,

last|head

```

root     pts/0        2a02:8070:c4c2:2 Wed Feb  4 15:52   still logged in

root     ssh          2a02:8070:c4c2:2 Wed Feb  4 15:52   still logged in

root     pts/0        2a02:8070:c4c2:2 Tue Feb  3 14:33 - 14:51  (00:17)

root     ssh          2a02:8070:c4c2:2 Tue Feb  3 14:33 - 14:51  (00:17)

```

As you can see there's a pts/0 and a ssh line

it's the same session however.

Occasionally the ssh one died, for whatever reason.

The pts/* session remains active but no responses are sent to the client since ssh is dead.

Using systemd with a custom config, kernel 3.18.5 gentoo-sources

What might be the reason?

For instance, an Archlinux or Debian system for comparison:

```

root     pts/0        2a02:8070:c4c2:2 Wed Feb  4 16:06   still logged in

root     pts/0        2a02:8070:c4c2:2 Wed Feb  4 11:19 - 11:19  (00:00)

```

Last edited by dalu on Thu Feb 12, 2015 7:00 pm; edited 1 time in total

----------

## Schnulli

Hi

i am wondering also for the same thing since a while and yet no answer why a dead ssh session is kept somehow alive but dead.....

just wondering  :Wink: 

A new Bug?

----------

## Hu

Does the server system know the client has died?  If the client process exited while unable to communicate with the server, then the server will not detect this until it tries and fails to send traffic to the client.  Generate some output on that pty and the session should go away.

----------

## Schnulli

Hi Hu

dosnt matter or stresses me..... this means to me, an open pipe is kept... so... reason one... maybe a bug in ssh itself, or reason tow, ssh is not correctly configured

i for myself kill this session by myself.

i never had intentions to find out what the reason is because i screen my systems and kill pids if they are dead

----------

## dalu

Ok, it just happened again.

Funny thing is

As soon as I noticed I opened a new terminal on the client and ssh'd into the server.

did

w

```

# w

 18:51:35 up  3:09,  2 users,  load average: 0.10, 0.17, 0.11

USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT

root     pts/0     16:16   15.00s  0.06s  0.06s -bash

root     pts/1     18:51    1.00s  0.01s  0.00s w

```

I hit [Enter] like 30 times (on pts/0) and the session re-appeared or better, the terminal became responsive again.

However I'm not sure if that's because pts/1 is active.

```

# last|head

root     pts/1        2a02:8070:c4c2:2 Thu Feb 12 18:51   still logged in

root     ssh          2a02:8070:c4c2:2 Thu Feb 12 18:51   still logged in

root     pts/0        2a02:8070:c4c2:2 Thu Feb 12 16:16   still logged in

root     ssh          2a02:8070:c4c2:2 Thu Feb 12 16:16 - 18:51  (02:35)

```

Hu,

yes when the client terminal gets closed the server detects this and "logs off" pts/0 (for instance).

It often happens when I emerge something.

edit:

I have now closed pts/1 session

```

root     pts/1        2a02:8070:c4c2:2 Thu Feb 12 18:51 - 19:06  (00:14)

root     ssh          2a02:8070:c4c2:2 Thu Feb 12 18:51 - 19:06  (00:14)

root     pts/0        2a02:8070:c4c2:2 Thu Feb 12 16:16   still logged in

root     ssh          2a02:8070:c4c2:2 Thu Feb 12 16:16 - 18:51  (02:35)

```

but pts/0 session is still active and responsive (on my client and the server)

----------

## Schnulli

Hi dalu

the same simptoms here..... 

even killing the PID dosnt affect and its still alive......

possible a Bug and new Backdoor detected? Seems like....again ^^

Hey GLSA ! have a look at  :Wink: 

@dalu

change ur Topic and add "GLSA Support pls"

Regards

----------

## dalu

You think, Schnulli?

Isn't that kind of drastic before doing our own investigation?

Ah well, better safe than sorry, I don't want to bug them if it isn't necessary  :Smile: 

They probably have more stuff to worry about.

However yeah it is the default way it works right now (sshd).

What I did so far was

 check sshd_config, nothing out of the ordinary

 change sshd.service sshd@.service to Archlinux ones add sshgenkeys.service, no effect

Next up:

 find out which package wtmp btmp is  :Very Happy: 

 emerge openssh-6.7 (no -r3, aka without the x509 patch and the other glue patch, not sure what it does) and see if that also happens

netstat doesn't show any other connections for sshd but that doesn't have to mean anything.

----------

## Schnulli

yep

better safe than sorry is also my way of thinking, thats one reason why Gentoo is still so clean and so much masked because dirty  :Wink: 

A backstep is mostly this i use, in this case to risky, the old sshd version is buggy ^^

If you ntop/stat something let it run for at last a month to be sure and >> (pipe) the output it into a file..... file size dosnt matter in our linux thinking mind  :Wink: 

by the way, why not to deep and longterm screen with Wireshark ?

regards

 *dalu wrote:*   

> You think, Schnulli?
> 
> Isn't that kind of drastic before doing our own investigation?
> 
> Ah well, better safe than sorry, I don't want to bug them if it isn't necessary 
> ...

 

----------

## dalu

 *Schnulli wrote:*   

> 
> 
> If you ntop/stat something let it run for at last a month to be sure and >> (pipe) the output it into a file..... file size dosnt matter in our linux thinking mind 
> 
> 

 

Well my rootfs is just 384GB, the rest is dedicated to mongodb.

 *Schnulli wrote:*   

> 
> 
> by the way, why not to deep and longterm screen with Wireshark ?
> 
> 

 

I need to write my auth server and lib (http), this wireshark logging would require 1-2 days extra, time is ticking, each day costs 4€ for running the 3 servers and they're not generating any income yet and there's still so much left to do, in short I don't really believe it's a security issue but an annoying bug and I need to get on with my plan  :Smile:  So much to do, so little time.

Maybe I misunderstood you, I don't like the limited scrollback of screen. And it works on other distros, so it should be working here.

I'll try a modified ebuild without the x509 patches. How is the non-r3 buggy?

```

diff openssh-6.7_p1.ebuild openssh-6.7_p1-r3.ebuild 

3c3

< # $Header: /var/cvsroot/gentoo-x86/net-misc/openssh/openssh-6.7_p1.ebuild,v 1.13 2014/12/31 07:40:01 vapier Exp $

---

> # $Header: /var/cvsroot/gentoo-x86/net-misc/openssh/openssh-6.7_p1-r3.ebuild,v 1.2 2014/12/31 07:29:47 vapier Exp $

14c14

< #X509_VER="8.1" X509_PATCH="${PARCH/6.7/6.6}+x509-${X509_VER}.diff.gz"

---

> X509_VER="8.2" X509_PATCH="${PARCH}+x509-${X509_VER}.diff.gz"

31,32c31,32

< KEYWORDS="alpha amd64 arm arm64 hppa ia64 m68k ~mips ppc ppc64 s390 sh sparc x86 ~amd64-fbsd ~sparc-fbsd ~x86-fbsd ~arm-linux ~x86-linux"

< IUSE="bindist ${HPN_PATCH:++}hpn kerberos ldap ldns libedit pam +pie sctp selinux skey static X X509"

---

> KEYWORDS="~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~s390 ~sh ~sparc ~x86 ~amd64-fbsd ~sparc-fbsd ~x86-fbsd ~arm-linux ~x86-linux"

> IUSE="bindist ${HPN_PATCH:++}hpn kerberos kernel_linux ldap ldns libedit pam +pie sctp selinux skey static X X509"

110,111c110,111

<       epatch "${FILESDIR}"/${PN}-6.6.1_p1-x509-glue.patch

<       use hpn && epatch "${FILESDIR}"/${PN}-6.6.1_p1-x509-hpn14v5-glue.patch

---

>       epatch "${FILESDIR}"/${P}-x509-glue.patch

>       epatch "${FILESDIR}"/${P}-sctp-x509-glue.patch

191c191

<       --with-pid-dir="${EPREFIX}"/var/run \

---

>       --with-pid-dir="${EPREFIX}"$(usex kernel_linux '' '/var')/run \

```

Have you tried "downgrading" to the non-r3 ebuild?

----------

## Schnulli

yes, on other distros it works... thats why i ask myself what the heck is on Gentoo the reason.....

No income yet on ur servers? where they are located? Country? and Line speed? Any Traffic  limitations?

Lets talk and open me a VirtualBox Slot & v-host remote management and i ll help paying ur bills at last a few  :Wink: 

This is more easy instead of renting again another 4HE Rack slot somewhere

No i havent downgraded yet bec. my SSHs are running behind a firewall, thats the reason why i wonder but dont get scared  :Wink: 

 *dalu wrote:*   

>  *Schnulli wrote:*   
> 
> If you ntop/stat something let it run for at last a month to be sure and >> (pipe) the output it into a file..... file size dosnt matter in our linux thinking mind 
> 
>  
> ...

 

----------

## dalu

 *Schnulli wrote:*   

> yes, on other distros it works... thats why i ask myself what the heck is on Gentoo the reason.....
> 
> No income yet on ur servers? where they are located? Country? and Line speed? Any Traffic  limitations?
> 
> Lets talk and open me a VirtualBox Slot & v-host remote management and i ll help paying ur bills at last a few 
> ...

 

Downgrading changed nothing  :Smile: 

I'm not doing virtualization but I can give you the Google+ Profile of someone who just started like a month ago, with 2 E5s using ganetti (and Funtoo).

The 3 servers are cheap refurbished Hetzner Xeon 1245v2 with 16GB ECC RAM and 2x3TB HGST Disks, really low cost.

I'm writing programs in Go since 1¼ years, switched from PHP and picked MongoDB for storage. The 3 cheap ones are cheaper than 1 big server.

I have 

3 nginx on the front,

libreswan (IPsec) on private addrs

services listening on private addrs, for instance domain.tld service listening on s0:10000 s1:10000 s2:10000

where 

s0 = 10.0.1.1

s1 = 10.0.1.2

s2 = 10.0.1.3

and in the back I have 3 replica sets which form a shard and the services talk to the mongos (mongodb shard service)

since mongodb is able to work with files (gridfs) I use this for storage

All services run as their own user with their custom, very limited shell that only accepts git pushes and "update" "build" "env" "ls".

And I have a management service listening on each server to create those services/users.

No virtualization, no containers, just systemd settings to control read/write permissions and capabilities and limits.

Since each service is a static binary that runs in its VM with safe types and 1 "GOMAXPROC" I'm not afraid of off by 1 or other attacks and the mongodb driver sanitizes by default (also because of safe Go types), so no fear of "injections".

And it all costs ~120€ / month.

That's 7.8TB single replicated storage, where 2 nodes can come down and content is still serving, but no writes can be made.

Bandwidth on paper is limited to 200mbit/s per node, real data shows it's less.

Each node is limited to 20TB outgoing / month, good enough for me, for starters  :Smile: 

So that's 12 cores, 3x16GB ECC RAM, 7.8TB replicated storage across nodes and theoretical throughput of 600mbit/s for 120€ / month.

Aka ~5k-15k concurrent connections.

A similar offer from online.net by with just 2x3TB storage costs 155€ incl VAT.

And I don't have to deal with VMs and/or containerization.

So for all my domains I first need to write the base, authentication

then add authorization

then make it openid-connect compatible

and lastly build content sites or "apps"

but all white-hat and legal, no black-hat stuff, nothing illegal.

You know what I'm looking for? Audio ads for mp3 content. If I could monetize audio I could afford to pay people to do coding and travel the world, instead of sitting in front of my PC the whole day long  :Smile:  And no Youtube doesn't cut it.

I can send you a PM if you want the guy's contact on Google+

----------

## Schnulli

u have a PN  :Wink: 

----------

## dalu

Now back on topic.

```

last|head

root     pts/1        2a02:8070:c4c2:2 Mon Feb 16 12:38   still logged in

root     ssh          2a02:8070:c4c2:2 Mon Feb 16 12:38   still logged in

root     pts/0        2a02:8070:c4c2:2 Mon Feb 16 11:59   still logged in

root     ssh          2a02:8070:c4c2:2 Mon Feb 16 11:59 - 12:38  (00:38)

```

I had "less /etc/pam.d/system-auth" running and my ISP decided that it's time to reboot my router (...),

so I got disconnected. Logged in again and pts/0 was still active with "less" still running.

So I killed less and killed bash associated with pts/0.

I've noticed that there's a difference between Archlinux' and Gentoo's /etc/pam.d/systemd-user

and few others

/etc/pam.d/sshd

points to system-remote-login

system-remote-login points to system-login

Gentoo's system-login

```

auth            required        pam_tally2.so onerr=succeed

auth            required        pam_shells.so 

auth            required        pam_nologin.so 

auth            include         system-auth

account         required        pam_access.so 

account         required        pam_nologin.so 

account         include         system-auth

account         required        pam_tally2.so onerr=succeed 

password        include         system-auth

session         optional        pam_loginuid.so

session         required        pam_env.so 

session         optional        pam_lastlog.so silent 

session         include         system-auth

session         optional        pam_motd.so motd=/etc/motd

session         optional        pam_mail.so

```

Archlinux' system-login

```

#%PAM-1.0

auth       required   pam_tally.so         onerr=succeed file=/var/log/faillog

auth       required   pam_shells.so

auth       requisite  pam_nologin.so

auth       include    system-auth

account    required   pam_access.so

account    required   pam_nologin.so

account    include    system-auth

password   include    system-auth

session    optional   pam_loginuid.so

session    include    system-auth

session    optional   pam_motd.so          motd=/etc/motd

session    optional   pam_mail.so          dir=/var/spool/mail standard quiet

-session   optional   pam_systemd.so

session    required   pam_env.so

```

difference

Gentoo has

```

auth            required        pam_nologin.so 

account         required        pam_tally2.so onerr=succeed 

```

Archlinux has

```

auth       requisite  pam_nologin.so

-session   optional   pam_systemd.so

```

However

system-auth

Gentoo's system-auth

```

auth            required        pam_env.so 

auth            required        pam_unix.so try_first_pass likeauth nullok 

auth            optional        pam_permit.so

account         required        pam_unix.so 

account         optional        pam_permit.so

password        required        pam_cracklib.so difok=2 minlen=8 dcredit=2 ocredit=2 retry=3 

password        required        pam_unix.so try_first_pass use_authtok nullok sha512 shadow 

password        optional        pam_permit.so

session         required        pam_limits.so 

session         required        pam_env.so 

session         required        pam_unix.so 

session         optional        pam_permit.so

-session        optional        pam_systemd.so

```

Archlinux' system-auth

```

#%PAM-1.0

auth      required  pam_unix.so     try_first_pass nullok

auth      optional  pam_permit.so

auth      required  pam_env.so

account   required  pam_unix.so

account   optional  pam_permit.so

account   required  pam_time.so

password  required  pam_unix.so     try_first_pass nullok sha512 shadow

password  optional  pam_permit.so

session   required  pam_limits.so

session   required  pam_unix.so

session   optional  pam_permit.so

```

Difference ,amongst other things

Gentoo's has

```

-session        optional        pam_systemd.so

```

Now systemd-user

Gentoo's systemd-user

```

# This file is part of systemd.

#

# Used by systemd --user instances.

account  include system-auth

session  include system-auth

```

Archlinux' systemd-user

```

# This file is part of systemd.

#

# Used by systemd --user instances.

account  include system-login

session  include system-login

```

Gentoo wants system-auth

Arch wants system-login

Downloading and installing Fedora to see how they do it.

Also I should probably read pam's manual  :Smile: 

http://www.linux-pam.org/Linux-PAM-html/Linux-PAM_SAG.html

Fedora 21 uses authconf to generate pam config files (good idea actually).

and its sshd file looks like this

```

#%PAM-1.0

auth       required     pam_sepermit.so

auth       substack     password-auth

auth       include      postlogin

# Used with polkit to reauthorize users in remote sessions

-auth      optional     pam_reauthorize.so prepare

account    required     pam_nologin.so

account    include      password-auth

password   include      password-auth

# pam_selinux.so close should be the first session rule

session    required     pam_selinux.so close

session    required     pam_loginuid.so

# pam_selinux.so open should only be followed by sessions to be executed in the user context

session    required     pam_selinux.so open env_params

session    optional     pam_keyinit.so force revoke

session    include      password-auth

session    include      postlogin

# Used with polkit to reauthorize users in remote sessions

-session   optional     pam_reauthorize.so prepare

```

and the password-auth substack

```

#%PAM-1.0

# This file is auto-generated.

# User changes will be destroyed the next time authconfig is run.

auth        required      pam_env.so

auth        sufficient    pam_unix.so nullok try_first_pass

auth        requisite     pam_succeed_if.so uid >= 1000 quiet_success

auth        required      pam_deny.so

account     required      pam_unix.so

account     sufficient    pam_localuser.so

account     sufficient    pam_succeed_if.so uid < 1000 quiet

account     required      pam_permit.so

password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=

password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok

password    required      pam_deny.so

session     optional      pam_keyinit.so revoke

session     required      pam_limits.so

-session     optional      pam_systemd.so

session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid

session     required      pam_unix.so

```

----------

## Ant P.

Can you justify having PAM installed at all, when you don't understand its security implications?

----------

## dalu

```

root     pts/0        2a02:8070:c48f:3 Tue Mar  3 09:46   still logged in

root     ssh          2a02:8070:c48f:3 Tue Mar  3 09:46   still logged in

root     pts/2        2a02:8070:c48f:3 Mon Mar  2 20:44 - 20:57  (00:13)

root     ssh          2a02:8070:c48f:3 Mon Mar  2 20:44 - 20:57  (00:12)

```

Where is pts/1 ?

net-misc/openssh-6.7_p1-r4::gentoo

sys-apps/systemd-219-r1:0/2::gentoo

seriously, what's going on there?

----------

