# [Problem Back] Rsync over SSH problems

## ltu0105

Im doing an automated backup from a windows 2000 server to my gentoo box across the lan.  The windows 2000 server is running OpenSSH from sshwindows.sourceforge.net.  The gentoo box will eventually be moved outside of the office for offsite backup purposes.  Before integrating the ssh i could do the automated backup flawlessly.  Now when i use rsync over ssh, it seems that after so many files it stops...  Here is my script that i am using...

```

#!/bin/sh

RSYNC=/usr/bin/rsync

OPTS="-avz --delete -P -e"

SSH=/usr/bin/ssh

KEY=/home/user/cron/thishost-rsync-key

RUSER=remoteuser

RHOST=remotehost

LPATH="/home/user/backup/daily/"

$RSYNC $OTPS "$SSH -i $KEY" $RUSER@$RHOST:"Acct\ Dept" $LPATH"Acct Dept"

$RSYNC $OTPS "$SSH -i $KEY" $RUSER@$RHOST:"Backup\ Files" $LPATH"Backup Files"

$RSYNC $OTPS "$SSH -i $KEY" $RUSER@$RHOST:"Gen\ Sector" $LPATH"Gen Sector"

$RSYNC $OTPS "$SSH -i $KEY" $RUSER@$RHOST:"IT\ Dept" $LPATH"IT Dept"

$RSYNC $OTPS "$SSH -i $KEY" $RUSER@$RHOST:"Job\ Files" $LPATH"Job Files"

$RSYNC $OTPS "$SSH -i $KEY" $RUSER@$RHOST:"Off\ Sector" $LPATH"Off Sector"

$RSYNC $OTPS "$SSH -i $KEY" $RUSER@$RHOST:"Orig\ Progs" $LPATH"Orig Progs"

```

The Acct Dept files backup fine but when i get to the second directory to backup "Backup Files" it halts part way through at the same place every time i run the script.  The Acct Dept doesnt have very many files so im wondering if it is a setting somewhere having to do with a maximum number of files that can be transfered per session???  Im not sure if that makes any sense.  Im still new to *nix so bare with me.  Also it doesnt matter whether i use the key auth or not, either way it still halts.

Thanks in advance for the help,

~NathanLast edited by ltu0105 on Wed Jun 01, 2005 7:35 pm; edited 3 times in total

----------

## ivanl

 *Quote:*   

> im wondering if it is a setting somewhere having to do with a maximum number of files that can be transfered per session?

 

There is no such 'max num of files that can be transfered'.

Since rsync works but rsync over ssh not, it must be your ssh settings. I would look for timeout setting(s) on the sshd, and rsync only 1 directory for testing to see if it helps.

----------

## ltu0105

Ill take another look but i didnt see any timeout settings in the sshd config.  Is there some sort of minimum settings that i should have for using ssh with key authentication? i uncommented what i thought i would need but i could be wrong.

~Nathan

----------

## nielchiano

 *ltu0105 wrote:*   

> Is there some sort of minimum settings that i should have for using ssh with key authentication? i uncommented what i thought i would need but i could be wrong.

 

There are some, but since the first rsync-command succeeds; that should be OK.

Does he terminate with some error? or just freezes?

----------

## ltu0105

Just freezes.. no error code or anything.  Kinda odd.  Like i said im new to *nix so i dont really know where to look up the log for the ssh stuff... i installed metalog and i checked the /var/log/ssh to see what was logged but nothing had been.  Any other ideas?

Thanks for the help,

~Nathan

----------

## ltu0105

bump

----------

## ltu0105

bump

is anyone else doing rsync over ssh?

----------

## Gherald

 *ltu0105 wrote:*   

> bump
> 
> is anyone else doing rsync over ssh?

 

Yes... one of my favorite commands is

```
rsync -a -e ssh /local/path u@h:/remote/path
```

I've never had any trouble, and I've transfered more than 50GB at a time.

----------

## nielchiano

 *Gherald wrote:*   

> I've never had any trouble, and I've transfered more than 50GB at a time.

 

same here: daily backup goes over ssh, the full size is 30GB, but off course, rsync cuts that down to few hundreds MB each day. never had a problem

----------

## ltu0105

 *Gherald wrote:*   

> 
> 
> Yes... one of my favorite commands is
> 
> ```
> ...

 

Im trying to look at it with a pair of fresh eyes because no one else seems to be having this problem... im going to start simple like your command Gherald and see where that takes me.  Ill let you know and thanks for the help so far.

~Nathan

----------

## ltu0105

Ok i tryed a few things and i noticed that when i use ssh for local to remote it works fine... but when i use it for remote to local that is when it freezes.  Im not sure why this would happen obviously but hopefully someone else might.

~Nathan

----------

## bluedevils

what's the directory path like where it dies (on the windows box)?  Could it be that there is an illegal character in a file/directory name?  Or has the path exceeded windows 255 character limit?

----------

## ltu0105

 *nielchiano wrote:*   

> daily backup goes over ssh, the full size is 30GB, but off course, rsync cuts that down to few hundreds MB each day. never had a problem

 

question is though have you rsync'd a windows 2000 computer?  :Smile:  what kinds of issues could that cause?  could it be associated with that?

~Nathan

----------

## ltu0105

 *bluedevils wrote:*   

> what's the directory path like where it dies (on the windows box)?  Could it be that there is an illegal character in a file/directory name?  Or has the path exceeded windows 255 character limit?

 

I made the path simple because of this kinds of problems its:

E:\Acct Dept

In order to account for the space i use the following:

$RUSER@$RHOST:"Acct\ Dept"

and like i said before it works for all of the Acct Dept files which doesnt have many but doesnt work for the larger directories im backing up...  so im pretty sure its not my script that is causing the issue... the more i think about it the more i think its something to do with the ssh on the windows box... i could be way off of course but has any one else done an rsync of a windows box?

~Nathan

----------

## nielchiano

 *ltu0105 wrote:*   

> a windows 2000 server

 

it's a feature  :Wink: 

score -1: offtopic... :Embarassed: 

----------

## bluedevils

right.  accounting directory is simple and it works fine.  The point of the question is not your script, but the limitations of the windows file system.  The limitations could give rsync a hard time.

 *ltu0105 wrote:*   

> I made the path simple because of this kinds of problems its:
> 
> E:\Acct Dept
> 
> In order to account for the space i use the following:
> ...

 

----------

## ltu0105

 *bluedevils wrote:*   

> right.  accounting directory is simple and it works fine.  The point of the question is not your script, but the limitations of the windows file system.  The limitations could give rsync a hard time.
> 
> 

 

My thought exactly but how would i begin to troubleshoot this?  Ive search for any kind of documentation online but nothing seems to help.

~Nathan

----------

## bluedevils

yes that is a hard one to fix or diagnose (in my limited knowledge).  It has always been a PIA that I haven't figured a fix (other than changing names to comply).

----------

## ltu0105

 *bluedevils wrote:*   

> yes that is a hard one to fix or diagnose (in my limited knowledge).  It has always been a PIA that I haven't figured a fix (other than changing names to comply).

 

That made me think of something... is there a limitation set by either windows (through cygwin) or linux concerning the depth of the directories accessable?

~Nathan

----------

## bluedevils

googling gets me this:

 Ville Herva   Sep 14 2002, 2:53 pm     show options 

Newsgroups: mailing.unix.rsync 

From: vhe...@niksula.hut.fi (Ville Herva) - Find messages by this author  

Date: Sat, 14 Sep 2002 18:43:50 +0000 (UTC) 

Local: Sat,Sep 14 2002 2:43 pm  

Subject: Re: limit of 256 characters in pathname 

Reply to Author | Forward | Print | Individual Message | Show original | Report Abuse  

On Wed, Sep 11, 2002 at 07:40:01PM +0200, you [bart.conin...@watco.be] wrote: 

> Hi, 

> We use rsync on NT (server and client) and we get errors on filenames with 

> a path that has in total more than 256 characters. This seems to be caused 

> by a limit in rsync, not in NT. Is there a way to break this limit withuo 

> having to manually shorten all this filenames? 

Which rsync version for windows are you using? 

If you are using the cygwin version, the 256 character limit is set by 

the cygwin API. 

You are right in that NT (/w2k/xp) and NTFS allow for longer filenames, but 

not through the normal Win32 API functions. You need to use the unicode 

versions of the file access functions, prepend a '\\?\' prefix to turn of 

file path parsing, and even then, individual path components cannot exceed 

MAX_PAH (255). 

From MSDN: 

  "Process a path as a null-terminated string. The maximum length for a 

  path, including a trailing backslash, is given by MAX_PATH.  The Unicode 

  versions of several functions permit paths that exceed the MAX_PATH length 

  if the path has the "\\?\" prefix. The "\\?\" tells the function to turn 

  off path parsing. However, each component in the path cannot be more than 

  MAX_PATH characters long. Use the "\\?\" prefix with paths for local 

  storage devices and the "\\?\UNC\" prefix with paths having the Universal 

  Naming Convention (UNC) format. The "\\?\" is ignored as part of the path. 

  For example, "\\?\C:\myworld\private" is seen as "C:\myworld\private", and 

  "\\?\UNC\bill_g_1\hotstuff\coo­lapps" is seen as 

  "\\bill_g_1\hotstuff\coolapps"­." 

So applications are able to create paths that have path longer than 255 

characters by changing the working directory to something long (perhaps 

using multiple mkdir/chdir) and creating the file there, or by specifying 

the full path with the unicode/"\\?\" hack. 

Cygwin doesn't do either. 

I talked about this with one of the Cygwin contributors, Chris January (who 

has a unicode filename support patch for cygwin pending btw), but he 

suspected that even if cygwin supported total path lengths greater than 

MAX_PATH (255), many application would break, since they don't support so 

long file names. (Perhaps rising cygwin MAX_PATH to something higher, and 

recompiling the apps would help.) 

Btw, this is not a problem for just cygwin apps. Many other windows apps 

can't access the long path files (virus scanners, backup software have been 

tricked by this.) And not only long paths. You can create files like 

"c:\prn" or "a:\lpt1" with the \\?\ hack, and that confuses even explorer 

(this perhaps has been fixed since.). 

Getting back to rsync/cygwin - it can't access filenames with unicode 

(different char set) letters in them either. As I said, Chris January has 

created an UTF8 patch for cygwin that addresses this, but it hasn't been 

merged so far. 

-- v -- 

v...@iki.fi 

-- 

To unsubscribe or change options: http://lists.samba.org/mailman­/listinfo/rsync 

Before posting, read: http://www.tuxedo.org/~esr/faq­s/smart-questions.html

----------

## bluedevils

I guess I was wrong saying it is the filesystem when this states that it is the API

----------

## ltu0105

 *bluedevils wrote:*   

> I guess I was wrong saying it is the filesystem when this states that it is the API

 

ummm im wondering how it works when i dont use ssh then?

~Nathan

----------

## ltu0105

YAY!  :Smile: 

Ok i solved it but im not exactly sure why this worked...  if you can remember in my script i had the following for each directory i was backing up:

$RSYNC $OTPS "$SSH -i $KEY" $RUSER@$RHOST:"Acct\ Dept" $LPATH"Acct Dept"

well i found out that i dont need the "Acct Dept" after the local path varibale "$LPATH" is used in the rsync command when using ssh.  What was happening was rsync was making an "Acct Dept" directory inside of the "Acct Dept" directory where everything was supposed to go... as soon as i removed this it ran through each directory to backup fine... im not sure if these extra few chars was putting the file path over the 255 chars or what but for some reason it is now working perfectly for the moment  :Smile:   Ill keep you updated on anything else that might arrise but for the moment it is working so thank you for all your help!

~Nathan

----------

## bluedevils

cool.  sorry for pulling you in the wrong direction.

----------

## ltu0105

 *bluedevils wrote:*   

> cool.  sorry for pulling you in the wrong direction.

 

no hey if it wasnt for you i prolly wouldnt have been hacking away at my script and might not have found that for awhile... and it is good to know that the windows api will only allow 255 chars... that might cause some problems down the road.  Thanks for all your help.

~Nathan

----------

## ltu0105

The problem has resurfaced and it seems as though the previous fix wasnt a fix after all...  I am looking for a way to log what is actually going on when it freezes.  I have added this to my metalog.conf file to try and log what is going on:

```

Rsyncd :

  facility = "*"

  program  = "rsyncd"

  logdir   = "/var/log/rsyncd"

```

Seeing as though im still new to *nix in general i have never had to log anything before... that was based off of the manual at http://metalog.sourceforge.net/README.  Is this how i would go about logging what is going on?

~Nathan

----------

## ltu0105

* bump *

anyone?

----------

## ltu0105

Well i found this discussion from a google search:

http://www.ussg.iu.edu/hypermail/linux/kernel/0103.2/0983.html

So i decided that i would break up the job files directory which is where the freezing occured hoping that maybe the number of errors would be much less than if it was one big backup over ssh... to no avail it did the same thing.  :Sad:    Im at a loss for words here... im hoping that someone else is using cygwin/rsync/ssh to backup a windows 2000 computer.

Then i found this...

http://lists.samba.org/archive/rsync/2004-May/009506.html

This is the exact problem that i am having... im running Cygwin DLL version 1.5.15-1 on the windows box and they now have a new release to 1.5.17-1 so i think im going to try the latest to see if it fixes it after i read the release notes.

~Nathan

----------

## ltu0105

Installed the latest version of the cygwin dll and it still hangs  :Mad: 

Anyone know of a version that doesnt hang... i dont mind using older versions so long as it works..

~Nathan

----------

## ltu0105

Can anyone help with the logging of the problem??

Thanks

----------

## ltu0105

*bump*

even if you can only help with the logging that would be great

Thanks

----------

## Moriah

I realize this is off topic re gentoo, but it is on topic re this thread.  :Smile: 

I too am trying to automate backups on my network, and I need to support a few windoze boxes along with the gentoo boxes.  I am using rsync over ssh, with the process initiated by the backup server every night.  I have been fighting this same stalling problem on the windoze boxes.  

This thread seems to imply that the problem is endemic to cygwin and the old api for specifying path/file names (or is that path\file names?   :Surprised:  )

So given that I really cannot control the path/file names on the windoze boxes, how can I back them up after this manner?  Is the utf8 patch available yet for sshd/rsync/cygwin ???

----------

