# scp won't copy with pubkey setup

## ixion

I posted a thread briefly touching on this, but it had to do with other issues which are now solved (or appear to be), although this issue remains.. I am deeply sorry to the MODs if this is too similar, just didn't want to dig up such an old thread that didn't exactly address this issue.

I have a backup script run by root every night, that backs up some crucial files, places them in a designated dir, and then changes ownership on the backup file (tar.gz) to a normal user. Then shortly after that completes, the normal user's cron scp's these files to my file server using a priv/pubkey authentication. Well this worked beautifully for months and months, until now (well a couple weeks ago) all of a sudden scp will just act like it's logging in, run the user's .bashrc, and then logout, resulting in no files being copied. Sometimes I can log in and do it manually during the day, but most of the time the copies do not go through. No errors, (tailed -f the messages log while doing this when it fails, it just states user logged in, user logged out). I've done a 'revdep-rebuild' on both machines (destination file server and web server), and that seemed to get it to work for a day, but is failing again. I can use ssh to run commands 'ls -lh, etc' all day with no troubles, no errors.

I don't understand what's going on here.. anyone have any clues?  :Crying or Very sad: 

----------

## Raffi

Kind of sounds like the scp program is not available or not working correctly on the remote machine. What scp does is ssh to the remote machine and run scp to setup the connection to copy over. In this case, scp must be exiting.

Try scp -v to see if there is any help in the output. Re-emerge ssh (and do an md5sum on scp). See if that fixes it. If it does for a little while, once it breaks check the md5sum. Could be someone has compromised your system.

----------

## ixion

re-emerging openssh on both machines results in same problem (which is a relief, don't want to think the system was compromised  :Wink:  ).

Here is an example of scp -v (it looks okay to me, what do you think?):

```

$ scp -v ./etc.tar.gz fileserver:/usr/backup/web/etc.tar.gz

Executing: program /usr/bin/ssh host fileserver, user (unspecified), command scp -v -t /usr/backup/web/etc.tar.gz

OpenSSH_3.8.1p1, OpenSSL 0.9.7d 17 Mar 2004

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: Connecting to fileserver [192.168.0.100] port 22.

debug1: Connection established.

debug1: identity file /home/user/.ssh/identity type -1

debug1: identity file /home/user/.ssh/id_rsa type 1

debug1: identity file /home/user/.ssh/id_dsa type -1

debug1: Remote protocol version 2.0, remote software version OpenSSH_3.8.1p1

debug1: match: OpenSSH_3.8.1p1 pat OpenSSH*

debug1: Enabling compatibility mode for protocol 2.0

debug1: Local version string SSH-2.0-OpenSSH_3.8.1p1

debug1: SSH2_MSG_KEXINIT sent

debug1: SSH2_MSG_KEXINIT received

debug1: kex: server->client aes128-cbc hmac-md5 none

debug1: kex: client->server aes128-cbc hmac-md5 none

debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP

debug1: SSH2_MSG_KEX_DH_GEX_INIT sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY

debug1: Host 'fileserver' is known and matches the RSA host key.

debug1: Found key in /home/user/.ssh/known_hosts:1

debug1: ssh_rsa_verify: signature correct

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug1: SSH2_MSG_NEWKEYS received

debug1: SSH2_MSG_SERVICE_REQUEST sent

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug1: Authentications that can continue: publickey,keyboard-interactive

debug1: Next authentication method: publickey

debug1: Trying private key: /home/user/.ssh/identity

debug1: Offering public key: /home/user/.ssh/id_rsa

debug1: Server accepts key: pkalg ssh-rsa blen 533

debug1: read PEM private key done: type RSA

debug1: Authentication succeeded (publickey).

debug1: channel 0: new [client-session]

debug1: Entering interactive session.

debug1: Sending command: scp -v -t /usr/backup/web/etc.tar.gz

<-- right here is where .bashrc is run (who, etc) -->

$ debug1: client_input_channel_req: channel 0 rtype exit-status reply 0

debug1: channel 0: free: client-session, nchannels 1

debug1: fd 0 clearing O_NONBLOCK

debug1: fd 1 clearing O_NONBLOCK

debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.2 seconds

debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0

debug1: Exit status 0

```

As far as the md5sum, what am I comparing scp's md5sum against? It changed after re-emerging openssh.

----------

## Raffi

 *ixion wrote:*   

> 
> 
> As far as the md5sum, what am I comparing scp's md5sum against? It changed after re-emerging openssh.

 

Sorry it took me so long to respond. My ISP bit the dust.

You would want to do an md5sum against a known good machine (with the same cflags and use flags).

When you do a successful scp, does your .bashrc run? What do you get when you type

```

ssh remotemachine 'echo $PATH'

```

----------

## ixion

 *Raffi wrote:*   

> 
> 
> Sorry it took me so long to respond. My ISP bit the dust.
> 
> You would want to do an md5sum against a known good machine (with the same cflags and use flags).
> ...

 

No prob about the response time, I know how that goes..  :Wink: 

I did an md5sum against a similar machine, but with slightly different USE flags, they were different.

yes, the .bashrc runs (I have a 'who' command in it). When typing 

```
"ssh fileserver 'echo $PATH'"
```

I get:

```

$ ssh fileserver "echo $PATH"

root     vc/1         Nov 27 08:37

root     vc/2         Nov 27 12:01

user    pts/1        Nov 27 09:45 (192.168.0.2)

/bin:/usr/bin:/usr/local/bin:/opt/bin:/usr/i686-pc-linux-gnu/gcc-bin/3.3

```

Thank you very much for your help!

----------

## Raffi

The only (important) difference between running from a cron script and running from the command line that I can think of is stdin being a tty or not. Try the following to see if you can break the copy from the command line the same way it is broken from cron

```

scp somefile remotemachine: </dev/null >/tmp/ssh.log 2>&1

```

You can see what happens in /tmp/ssh.log. Also try it with the -v flag to see if it gives any more insights.

----------

## ixion

after doing:

```

$ scp -v tmp.txt fileserver: </dev/null >/tmp/ssh.log 2>&1

```

```

$ cat /tmp/ssh.log 

Executing: program /usr/bin/ssh host fileserver, user (unspecified), command scp -v -t .

OpenSSH_3.8.1p1, OpenSSL 0.9.7d 17 Mar 2004

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: Connecting to fileserver [192.168.0.100] port 22.

debug1: Connection established.

debug1: identity file /home/user/.ssh/identity type -1

debug1: identity file /home/user/.ssh/id_rsa type 1

debug1: identity file /home/user/.ssh/id_dsa type -1

debug1: Remote protocol version 2.0, remote software version OpenSSH_3.8.1p1

debug1: match: OpenSSH_3.8.1p1 pat OpenSSH*

debug1: Enabling compatibility mode for protocol 2.0

debug1: Local version string SSH-2.0-OpenSSH_3.8.1p1

debug1: SSH2_MSG_KEXINIT sent

debug1: SSH2_MSG_KEXINIT received

debug1: kex: server->client aes128-cbc hmac-md5 none

debug1: kex: client->server aes128-cbc hmac-md5 none

debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP

debug1: SSH2_MSG_KEX_DH_GEX_INIT sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY

debug1: Host 'fileserver' is known and matches the RSA host key.

debug1: Found key in /home/user/.ssh/known_hosts:1

debug1: ssh_rsa_verify: signature correct

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug1: SSH2_MSG_NEWKEYS received

debug1: SSH2_MSG_SERVICE_REQUEST sent

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug1: Authentications that can continue: publickey,keyboard-interactive

debug1: Next authentication method: publickey

debug1: Trying private key: /home/user/.ssh/identity

debug1: Offering public key: /home/user/.ssh/id_rsa

debug1: Server accepts key: pkalg ssh-rsa blen 533

debug1: read PEM private key done: type RSA

debug1: Authentication succeeded (publickey).

debug1: channel 0: new [client-session]

debug1: Entering interactive session.

debug1: Sending command: scp -v -t .

root     vc/1         Nov 27 08:37

debug1: client_input_channel_req: channel 0 rtype exit-status reply 0

debug1: channel 0: free: client-session, nchannels 1

debug1: fd 0 clearing O_NONBLOCK

debug1: fd 1 clearing O_NONBLOCK

debug1: fd 2 clearing O_NONBLOCK

debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.1 seconds

debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0

debug1: Exit status 0

```

Actually to do the copying of the backup files I use the following script that is run from cron:

```

#!/bin/bash

echo ' '

echo 'Changing to Local Directory...'

cd /backups

echo 'Copying to File Server...'

scp db.tar.gz fileserver://backup/web/db.tar.gz

scp mail.tar.gz fileserver://backup/web/mail.tar.gz

scp etc.tar.gz fileserver://backup/web/etc.tar.gz

echo ' '

ssh fileserver ls -lh /backup/web/

echo ' '

date

echo ' '

echo 'Done.'

```

----------

## ixion

Ah! found something that might be helpful... when trying to scp the files from the webserver to the fileserver executed from the fileserver I get an error.

```

$ scp webserver://backups/etc.tar.gz /backup/web/etc.tar.gz

Top o' the day to ya!

protocol error: unexpected <newline>

```

The Top o' the day is the ssh banner (I know, a bit weird;))..

edit: Removed the banner from sshd_config and restarted the daemon on the webserver, same protocol error.

edit2: I can ssh from the fileserver to the webserver just fine. Scp is just what's hanging up.

----------

## Raffi

Sounds like the fileserver has the motd option turned on. Might as well turn that off in /etc/ssh/sshd_config.

Not that it should matter, but why to you do fileserver://dir/... instead of fileserver:/dir/...?

I will be out of contact (going home to my lack of ISP)  until tomorro.

----------

## ixion

I think the double slash was due to some issues a long time ago figuring out scp (it can be finicky at times;)), but went ahead and eliminated it.

Set 'PrintMotd no' in both machine's sshd_config files and restarted sshd, but still same issue. This is quite strange, eh? You know all of a sudden samba on the fileserver has been weird about permissions, just out of the blue (Windows claims there being a permissions problem with the remote and local profiles). How can I investigate a possible compromise? If the fileserver's been compromised, that puts my entire network at serious risk.. not a good thing..   :Evil or Very Mad: 

Thank you so much for you help.. have a nice trip home and hope to hear from you tomorrow.

----------

## Raffi

Checking log files is a start, though someone who has compromised you can remote the evidence.  If you have gentoolkit installed, you can use qpkg -c to check md5 sums and mtimes.

----------

## ixion

I ran that, but don't understand the qpkg output. Some packages appear as:

```

www-apps/horde-imp-3.2.5 *

-0/411

```

While others:

```

net-analyzer/snort-2.2.0 *

-44/148

```

And others:

```

net-www/apache-1.3.32-r1 *

----4/727

```

I'm very sorry to be so dense.. what do the num/num and '-' mean?

The logs on the Web server have not seemed to have anything suspicious other than not being able to reverse lookup my work's IP (that I SSH from normally) a couple days ago.

Thank you again for your help!

----------

## Raffi

That output seems odd to me. If I do

qpkg -c ssh

It lists all packages with ssh in them and the first number seems to be the number of files not matching the checksum and the second seems to be the total number of files. Most things I check get 0/somenumber. Things that I have run prelink on get md5sum errors. If you add a -v you should see which files don't match.

----------

## ixion

oops, I must have misunderstood you, I'm sorry. Here is the results (which looks reassuring):

```

# qpkg -c -v ssh   

net-misc/openssh-3.8.1_p1-r1 *

/etc/ssh/sshd_config !md5! !mtime!

1/60

```

I remember that rebooting the server allowed it to work for one or two backup sessions, and then failed. I just tried rebooting the fileserver and it's not coming back up (at least not responding to SSH). Will check the console on that when I get home.

Thanks!

----------

## ixion

okay, the fileserver just had a firewall issue (different story   :Wink:  ). 

Rebooting the fileserver did not fix it this time. It looks to be permanently b0rked. I'm thinking of formatting the thing this weekend, but really don't want to. Can you see any alternatives to that?

Thanks again so very much for all your insight.. I've at least learned some new things through this experience from you..  :Smile: 

----------

## weyhan

Got from the comments in the .bashrc file:

```
# /etc/skel/.bashrc:

# $Header: /home/cvsroot/gentoo-src/rc-scripts/etc/skel/.bashrc,v 1.8 2003/02/28 15:45:35 azarah Exp $

# This file is sourced by all *interactive* bash shells on startup.  This

# file *should generate no output* or it will break the scp and rcp commands.

```

  :Wink: 

----------

## Raffi

That really should say NON interactive, but yes .bashrc should not product output or expect input.

Which brings me to a pet peeve about the X start script sourcing the .bash_profile instead of the .bashrc. With the result of any interactive commands in your .bash_profile breaking your X login.

I first ran into that broken behavior in RedHat years ago, and unfortunately, the gentoo devs have done the same (or at least not fixed it).

----------

## ixion

this is hilarious, but removing 'who' from the .bashrc seems to have fixed it for the time being, wahoo!  :Smile: 

So I should put the 'who' command in ~/.bash_profile, now?

Thanks!

----------

## Raffi

That would be the place for it, yes.

Sorry, I thought you had only added that when you were testing. I missed the fact that you had it in there all along.

----------

## ixion

hey no problem..

I'm a retard for not noticing something so simple..

Thank you so much for all your help.. this has been quite a learning experience for me and am glad it's working now!!! Thank you!!!!!!!!  :Very Happy:  :Very Happy:  :Very Happy:  :Very Happy: 

----------

