Delay for --remove-source-files

[Martin Scharrer]

Hi,

I'm using rsync with -aP --remove-source-files to move files from one machine to another while watching the progress. I'm under the impression that rsync is deleting the transmitted source files on-the-fly, not at the very end, but with a delay of 2-3 files, i.e. if 10 files are moved the first source file is deleted after the third of fourth file got transmitted. However, if rsync is aborted (CTRL+C) all fully source transmitted files are deleted.
Can anyone tell me if this delay is intended behavior and if it can be manipulated/configured differently?
My source machine has only limited space (10GB), my files are rather big (.4 - 1.2GB each) and my bandwidth around 300-500KB, so sometimes I want to get rid of the sent files as fast as possible without waiting for the next files being transmitted.

I couldn't find more information about this. The --delete* options are all for receiving side deletions, not for the source files.

Thanks,
Martin

[Matt McCutchen-7]

On Tue, 2009-10-13 at 03:51 -0700, Martin Scharrer wrote:

> I'm using rsync with -aP --remove-source-files to move files from one
> machine to another while watching the progress. I'm under the impression
> that rsync is deleting the transmitted source files on-the-fly, not at the
> very end, but with a delay of 2-3 files, i.e. if 10 files are moved the
> first source file is deleted after the third of fourth file got transmitted.
> However, if rsync is aborted (CTRL+C) all fully source transmitted files are
> deleted.
> Can anyone tell me if this delay is intended behavior and if it can be
> manipulated/configured differently?
> My source machine has only limited space (10GB), my files are rather big (.4
> - 1.2GB each) and my bandwidth around 300-500KB, so sometimes I want to get
> rid of the sent files as fast as possible without waiting for the next files
> being transmitted.

IIUC, the delay occurs due to the pipelining in rsync; since it is not a
problem for most users, no special effort was made to avoid it.  If it
is a problem for you, you might consider running rsync once per file.
It sounds like the files are big enough that doing so wouldn't be
unreasonably wasteful.

--
Matt

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Martin Scharrer]

Matt McCutchen-7 wrote:

On Tue, 2009-10-13 at 03:51 -0700, Martin Scharrer wrote:
> I'm using rsync with -aP --remove-source-files to move files from one
> machine to another while watching the progress. I'm under the impression
> that rsync is deleting the transmitted source files on-the-fly, not at the
> very end.
> [...]

IIUC, the delay occurs due to the pipelining in rsync; since it is not a
problem for most users, no special effort was made to avoid it.  If it
is a problem for you, you might consider running rsync once per file.
It sounds like the files are big enough that doing so wouldn't be
unreasonably wasteful.

Thanks Matt for pointing this out.
I will consider to change my rsync script.

Martin