# unusual network speed issues when using rsync...

## rishi

Hi there,

Hoping for some ideas towards solving this odd behavior.

I am copying about 1000GB of data from three servers to a portable FireWire/1394a drive on my local gentoo workstation using rsync (archiving obsolete job data). The arrangement is like this:

 - Server A (SME Server i686 box).

 - Server B (SuSE 10.2).

 - Server C (Intel ReadyNAS).

 - Minitar gigabit switch, all gigabit networking. 

 - Local workstation running gentoo. This is where the rsync command is being run.

 - Western Digital MyBook 2TB 1394a portable hard disk drive. 

If I run one rsync command copying data from any of servers A, B or C to the 1394 disk I get about 500kB/s total transfer speed, on average, as reported by rsync --progress option. Unsatisfactory performance, will take days longer.

If I run two or three rsync commands at once (on separate terminals obviously), each from separate server sources, EACH rsync transfer runs at around 5,000kB/s, giving me total data writing throughput to the 1394 disk of either 10,000kB/s or 15,000kB/s. Satisfactory performance, will only take a couple of days to complete. 

In this second case, once two servers have finished their work and only one rsync command is left with work to do, it drops back down to 500kB/s transfer speed. Frustrating, takes a long time to get to the finish!    :Mad: 

What could be causing this? It is some sort of QoS issue? Happy to provide any information that might help identify the problem here. 

Any help appreciated. Cheers,

EDIT: Changed subject as this seems to be rsync specific.

----------

## spOOwn

Did you try to transfer a file with, by instance, a simple scp ? I means without using rsync ! 

Without rsync did you get also bad performance on transfering a file ?

Is there any firewall rule in one host to limit transfer perhaps ?

----------

## rishi

Hi sp00wn, thanks for the ideas,

I have the network drives mounted locally two using cifs and one using smbfs. So to test another way I'm using cp to copy a bunch of data across. This goes straight to 5,000kB/s+. 

So yes, perhaps the problem seems to relate to the use of rsync. The rsync command I'm using is something like this (run from within a batch script):

```
rsync -vu --progress /mnt/data2/PD0170/video01/*.avi /media/BACKUPDISK/video1/
```

I'm just using rsync, so that I can interrupt the process whenever I want and just re-run the script to pickup where it left off with no unnecessary data copying. 

Regarding the firewall idea, I get the same behavior with iptables turned off on the local workstation. I can't do this so easily on the servers or switch though!

Anything else I could provide?

Thanks for the help,

----------

## spOOwn

I known that rsync compute a checksum on each block of your file or something like that, perhaps your computer isn't faster enough to do this, and hence the transfer is only !!

I never did a transfer so much, so I don't really known !

Check your cpu usage when you are doing the rsync !

----------

## rishi

Hi sp00wn,

I don't think it is a CPU bottleneck as when you work it harder (two transfers at once) it actually runs faster.

The workstation running the rsync is P4 3Ghz 1GB RAM so it should be up to the task anyway. 

Here is my emerge info:

```
# emerge --info

Portage 2.1.4.4 (default-linux/x86/2006.1/desktop, gcc-4.1.2, glibc-2.6.1-r0, 2.6.22-gentoo-r5 i686)

=================================================================

System uname: 2.6.22-gentoo-r5 i686 Intel(R) Pentium(R) 4 CPU 3.00GHz

Timestamp of tree: Tue, 29 Apr 2008 22:45:01 +0000

distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled]

ccache version 2.4 [enabled]

app-shells/bash:     3.2_p17-r1

dev-java/java-config: 1.3.7, 2.1.4

dev-lang/python:     2.4.4-r9

dev-python/pycrypto: 2.0.1-r6

dev-util/ccache:     2.4-r7

sys-apps/baselayout: 1.12.11.1

sys-apps/sandbox:    1.2.18.1-r2

sys-devel/autoconf:  2.13, 2.61-r1

sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.1

sys-devel/binutils:  2.18-r1

sys-devel/gcc-config: 1.4.0-r4

sys-devel/libtool:   1.5.26

virtual/os-headers:  2.6.23-r3

ACCEPT_KEYWORDS="x86"

CBUILD="i686-pc-linux-gnu"

CFLAGS="-O2 -pipe -march=pentium4"

CHOST="i686-pc-linux-gnu"

CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/config /var/bind"

CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/revdep-rebuild /etc/splash /etc/terminfo /etc/udev/rules.d"

CXXFLAGS="-O2 -pipe -march=pentium4"

DISTDIR="/usr/portage/distfiles"

FEATURES="ccache distlocks fixpackages metadata-transfer parallel-fetch sandbox sfperms strict unmerge-orphans userfetch"

GENTOO_MIRRORS="ftp://distro.ibiblio.org/pub/linux/distributions/gentoo/ http://distro.ibiblio.org/pub/linux/distributions/gentoo/ ftp://ftp.ucsb.edu/pub/mirrors/linux/gentoo/ http://ftp.ucsb.edu/pub/mirrors/linux/gentoo/ http://open-systems.ufl.edu/mirrors/gentoo http://mirror.datapipe.net/gentoo ftp://mirror.datapipe.net/gentoo http://www.mirrorservice.org/sites/www.ibiblio.org/gentoo/ ftp://ftp.mirrorservice.org/sites/www.ibiblio.org/gentoo/ http://gentoo.virginmedia.com/ ftp://gentoo.virginmedia.com/sites/gentoo ftp://mirror.pacific.net.au/linux/Gentoo ftp://ftp.swin.edu.au/gentoo http://ftp.swin.edu.au/gentoo ftp://ftp.ecc.u-tokyo.ac.jp/GENTOO http://ftp.jaist.ac.jp/pub/Linux/Gentoo/ ftp://ftp.jaist.ac.jp/pub/Linux/Gentoo/ http://ftp.iij.ad.jp/pub/linux/gentoo/ ftp://ftp.iij.ad.jp/pub/linux/gentoo/ "

LINGUAS="en_GB"

MAKEOPTS="-j3"

PKGDIR="/usr/portage/packages"

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"

PORTAGE_TMPDIR="/var/tmp"

PORTDIR="/usr/portage"

PORTDIR_OVERLAY="/usr/portage/local/layman/gentoo-gis /usr/portage/local/layman/desktop-effects"

SYNC="rsync://rsync.au.gentoo.org/gentoo-portage"

USE="X acpi alsa arts berkdb bzip2 cairo cdr cli cracklib crypt cups dbus dri dvd dvdr eds emboss encode esd fam firefox fortran gdbm gif gnome gpm gstreamer gtk hal iconv ipv6 irmc isdnlog java jpeg kde kdepim ldap mad midi mikmod mozilla mp3 mpeg mudflap ncurses nls nptl nptlonly nvidia ogg opengl openmp oss pam pcre pda perl png ppds pppd python qt qt3 qt4 quicktime readline reflection samba sdl session spell spl ssl tcpd truetype unicode vorbis win32codecs x86 xml xorg xv zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse synaptics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en_GB" USERLAND="GNU" VIDEO_CARDS="nvidia fbdev vesa"

Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

```

----------

## rishi

I've edited the subject line as this seems to be rsync specific. 

To summarize:

Copying data from a few different file servers on the local network, all mounted locally. Copying to an external FireWire 1394a HDD (mounted locally). (Same result when plugged in via USB). 

If I run one instance of rsync the transfer rate is about 500kB/s. (Much slower than it should run).

If I run two or more instances of rsync, the transfer rate is about 5,000kB/s for EACH instance. (i.e. 10,000kB/s+ total).

Using other programs like cp runs at full speed even if only one instance is run at once. 

So rsync is capable of doing the job nice and fast, but for some reason it is not... why would this be?

----------

## SeaTiger

Try to do the multiple rsync test again and use iftop to monitor the transfer speed.

I am guessing (pure guessing, no real experience) that rsync is reporting fake transfer rate when you run multiple copies at the same time (from the same account?).

Anyway, give iftop a try.

----------

## rishi

Hi junksiu,

I tried iftop to verify the rsync reported speeds, and they look about right / verified. I also have a superkaramba monitor running which is reporting the same speeds too. 

So it looks like the speeds being reported by rsync are about right. It really is running slow when there is one instance and it really is running fast when there are two instances running...   :Confused: 

 :Arrow:   New Info: I have tried this on a SuSE linux workstation we have here. I used the same external drive, the same network shares and the same script running rsync. The rsync transfers run at full speed when there is only one instance of rsync running. Also, as you would expect, when you run multiple instances of rsync they all run a bit slower as they are sharing the available bandwidth. 

So this problem seems to be confined to the gentoo workstation! 

What could be different on the gentoo box? I'll check for:

Different versions of rsync.

Different kernel versions.

Different firewalls / QoS?

----------

## rishi

More testing:

I have a single rsync running and it is plugging away at 500kB/s. Has been running like this steadily for 30mins or more. Total network use is reported at around 500kB/s too. 

I use Konqueror to drag and drop about 1Gb of data from a network share onto a USB stick. Konqueror reports this transfer going through at around 4.5MB/s. I notice total network use has jumped up to over 10MB/s (even getting up to 15). 

I look back at the rsync while this Konqueror data copy is running and it has jumped up to 5MB/s+. 

As soon as the Konqueror copy finishes, the rsync rate drops back down to 500kB/s. All confirmed against total network usage monitored with iftop. 

Why won't it stay at full speed..?

----------

## SeaTiger

What is your rsync nice level?

----------

