rsync algorithm for large files

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

On 04.09.2009 18:00, eharvey@... wrote:
>
> Why does it take longer the 3rd time I run it?  Shouldn?t the performance
> always be **at least** as good as the initial sync?

Not per se.

First you have to determine THAT the file has changed, then the file is
synced if there was a change. At least that's what you have to do when
the file-size is unchanged and only the timestamp is differs.
(Which is unfortunatly often the case for Virtual Machine Images)

Worst case: Takes double the time if the change is at end of the file.

When the filesize differs rsync immediatly knows that the file has
actual changes and starts the sync right away.

If i understand '--ignore-times' correctly it forces rsync to always
regard the files as changed and so start a sync right away, without
first checking for changes.


There are also some other options that may or may not have a speed
impact for you:
--inplace, so that rsync doesn't create a tmp-copy that is later moved over
the previous file on the target-site.
--whole-file, so that rsync doesn't use delta-transfer but rather copies
the whole file.

Also you may to separate the small from the large files with:
--min-size
--max-size
So you can use different options for the small/large file(s).




Bis denn

--
Real Programmers consider "what you see is what you get" to be just as
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated,
cryptic, powerful, unforgiving, dangerous.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Matthias Schniedermeyer (ms@...) wrote on 5 September 2009 00:34:
 >On 04.09.2009 18:00, eharvey@... wrote:
 >>
 >> Why does it take longer the 3rd time I run it?  Shouldn?t the performance
 >> always be **at least** as good as the initial sync?
 >
 >Not per se.
 >
 >First you have to determine THAT the file has changed, then the file is
 >synced if there was a change. At least that's what you have to do when
 >the file-size is unchanged and only the timestamp is differs.
 >(Which is unfortunatly often the case for Virtual Machine Images)
 >
 >Worst case: Takes double the time if the change is at end of the file.

No, rsync assumes that the file has changed if either the size or the
timestamp differs, and syncs it immediately.

For a new file transfer it's read once in the source and written once
in the destination. For an update it's still read once in the source
but read twice and written once in the destination, no matter how many
or extensive the changes are. The source also has to do the slidding
checksumming. This is usually faster than reading the file, so it'll
only slow down the process if the source is very slow or the cpu is
busy with other tasks. OTOH, the IO on the destination is
significantly higher for big files; this is often the cause of a
slower transfer rate than a full copy.

 >There are also some other options that may or may not have a speed
 >impact for you:
 >--inplace, so that rsync doesn't create a tmp-copy that is later moved over
 >the previous file on the target-site.

Yes, this is useful because it avoids both a second reading and the
full write on the destination (in priciple; I didn't bother to check
the actual implementation). For large files with small changes this
option is probably the best. The problem is that if the update aborts
for any reason you lose your backup. One might want to keep at least
two days of backups in this case.

 >--whole-file, so that rsync doesn't use delta-transfer but rather copies
 >the whole file.

Yes but causes a lot of net traffic. He mentions an average transfer
rate of about 11MB/s, so for a 100Mb/s net whole-file is probably not
suitable. If however he has a free gigabit link it'll be the best if
--inplace is not acceptable.

 >Also you may to separate the small from the large files with:
 >--min-size
 >--max-size
 >So you can use different options for the small/large file(s).

Agreed.

I'd also suggest using rsync v3 because it limits the blocksize.
Previous versions will use quite a large block for big files and if
changes are scattered it'll transfer much more than v3.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

eharvey@... wrote:

I thought rsync, would calculate checksums of large files that have changed timestamps or filesizes, and send only the chunks which changed.  Is this not correct?  My goal is to come up with a reasonable (fast and efficient) way for me to daily incrementally backup my Parallels virtual machine (a directory structure containing mostly small files, and one 20G file)

 

I’m on OSX 10.5, using rsync 2.6.9, and the destination machine has the same versions.  I configured ssh keys, and this is my result:

Upgrade to rsync 3 at least.

Rsync keeps a hash of the blocks of sliding hashes. For older versions of rsync, the has was of a constant size. This meant that files over 3GB in size had a high chance of hash collisions. For a 20G file, the collisions alone might be the cause of your trouble.

Newer rsyncs detect when the hash gets too big, and increase the has size accordingly, thus avoiding the collisions.

In other words - upgrade both sides (but specifically the sender).

Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

I am posting slightly a different qustion..I apologize for deviation.

I am in situation where I have an application running on two websphere nodes generating
and depositing the files locally on each node. I have the rsync running on both nodes ( I do not have
a shared storage/ option to use NFS) which will pick up the files deposited locally and transfer them to
the target nodes.  The setup could potentially cause a problem wherein one node will have an older version
of the files at any point in time base of the fact that it's the websphere application which processes
the files only when it gets a request.  

The real question is, what 'rsync' does if the timestamp of the file on the target location is greater/newer
than the timestamp of the source file?

Please let me know your thoughts.

Thanks,
Raj

Edward Harvey-4 wrote:

I thought rsync, would calculate checksums of large files that have changed
timestamps or filesizes, and send only the chunks which changed.  Is this
not correct?  My goal is to come up with a reasonable (fast and efficient)
way for me to daily incrementally backup my Parallels virtual machine (a
directory structure containing mostly small files, and one 20G file)



I’m on OSX 10.5, using rsync 2.6.9, and the destination machine has the same
versions.  I configured ssh keys, and this is my result:



(Initial sync)

time rsync -a --delete MyVirtualMachine/ myserver:MyVirtualMachine/

                20G

                ~30minutes



(Second time I ran it, with no changes to the VM)

time rsync -a --delete MyVirtualMachine/ myserver:MyVirtualMachine/

                2 seconds



(Then I made some minor changes inside the VM, and I want to send just the
changed blocks)

time rsync -a --delete MyVirtualMachine/ myserver:MyVirtualMachine/

                After waiting 50 minutes, I cancelled the job.



Why does it take longer the 3rd time I run it?  Shouldn’t the performance
always be **at least** as good as the initial sync?



Thanks for any help…

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html