Nabble - Samba - rsync

Re: Rsync shortcutting output?

2009-12-18T07:12:35Z

> On Thu 17 Dec 2009, Joel Peabody wrote:
>>
>> The meaning of my question is twofold. First, if this is a known issue
>> and what I'm seeing is, in fact, what's intended to happen on systems of
>> this size. Second, is there any feature, flag, or setting that is
>> either
>> being overlooked in my rsync command or is otherwise undocumented. Is
>> there some sort of switch that can be flicked either in the rsync setup
>> or
>> my unitilization of it to force it to put out full filenames that is not
>> covered in the manpage, or is there and I'm not seeing it?
>
> All I can say is that:
>
> --progress is not suitable for logging to a file (or reporting to a
> mailinglist)
> you use -i (itemize changes) but I see no related output in your
> message
>
> So something in the way you log the output isn't quite right.
>
>
> Paul
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>

That's what I thought. I already have 'i' in my flags (-rpti) so I'm kind
of at a dead end. I found one other post on the web where someone was
having a similar problem. He had 28 million files in his system, and
upgrading to 3.0 solved it. I guess that's going to be my ammunition in
the battle for getting these systems upgraded. Till then I guess it's
going to be some kind of post-transfer job with the ls command.

Thanks,
JP

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync shortcutting output?

2009-12-18T04:33:57Z

On Thu 17 Dec 2009, Joel Peabody wrote:
>
> The meaning of my question is twofold. First, if this is a known issue
> and what I'm seeing is, in fact, what's intended to happen on systems of
> this size. Second, is there any feature, flag, or setting that is either
> being overlooked in my rsync command or is otherwise undocumented. Is
> there some sort of switch that can be flicked either in the rsync setup or
> my unitilization of it to force it to put out full filenames that is not
> covered in the manpage, or is there and I'm not seeing it?

All I can say is that:

--progress is not suitable for logging to a file (or reporting to a
mailinglist)
you use -i (itemize changes) but I see no related output in your
message

So something in the way you log the output isn't quite right.

Paul
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --timeout not honoured

2009-12-18T04:29:16Z

At 14:19 16.12.2009 -0800, Wayne Davison wrote:
>On Tue, Dec 15, 2009 at 11:41 PM, Fabian Cenedese <<mailto:Cenedese@...>Cenedese@...> wrote:
>I think I know now where my confusion comes from. The reported time is not
>the actual time without action but the time since the start of the rsync command.
>
>
>It shouldn't be, since it is reporting the time elapsed since the last input was received. Â I'd suggest running an strace on the rsync and seeing what it is doing. Â I'd imagine that some system call does not return for an extended amount of time, and then, when rsync finally gets control back and checks for the elapsed time since the last received input, it is a really long amount.

I now have a trace with a timeout case (almost 300MB). What do I need to look
for? I don't see immediately what part lead to the timeout. A short snippet:

13:18:51 ioctl(1, TIOCGPGRP, [17723]) = 0
13:18:51 write(1, " 955908096 45% 44.22kB/s "..., 41) = 41
13:18:51 select(5, NULL, [4], [4], {60, 0}) = 1 (out [4], left {60, 0})
13:18:51 write(4, "\374\17\0\7", 4) = 4
...
13:18:52 select(5, NULL, [4], [4], {60, 0}) = 1 (out [4], left {60, 0})
13:18:52 write(4, "qo\223\16\352\240\31z`\374;\350\203\374\2551\252\354\335"..., 4092) = 4092
13:18:52 time(NULL) = 1261138732
13:18:52 select(5, NULL, [4], [4], {60, 0}) = 1 (out [4], left {60, 0})
13:18:52 write(4, "\374\17\0\7", 4) = 4
13:18:52 time(NULL) = 1261138732
13:18:52 select(5, NULL, [4], [4], {60, 0}) = 1 (out [4], left {60, 0})
13:18:52 write(4, "\367\344\356\325UUM3\273*<17\7\256D\255\225x0\345^{\203"..., 4092) = 4092
13:18:52 time(NULL) = 1261138732
13:18:52 select(5, NULL, [4], [4], {60, 0}) = 0 (Timeout)
13:19:52 time(NULL) = 1261138792
13:19:52 write(2, "io timeout after 9720 seconds --"..., 40) = 40
13:19:52 write(2, "\n", 1) = 1
13:19:52 rt_sigaction(SIGUSR1, {SIG_IGN}, NULL, 8) = 0
13:19:52 rt_sigaction(SIGUSR2, {SIG_IGN}, NULL, 8) = 0

Should I trace with different flags?

Thanks

bye Fabi

PS: I'll be away for two weeks so I can only give more info in the new year.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync shortcutting output?

2009-12-17T11:14:55Z

> On Thu 17 Dec 2009, Joel Peabody wrote:
>>
>> PS, this is between two Redhat machines using rsync 2.6.8 on both ends.
>> Upgrading rsync is not an option in this case.
>
> What I always wonder when reading such messages, is what type of
> response is expected, if a problem with the tool is suspected but
> upgrading "is not an option"? (Why not?) That we travel back in time
> to fix the tool before the person in question installs it? :-)
>

Yes, if you could get on that immediately, I bet there would be multiple
applications for such technology. ;)

The reason I included that tidbit is because a) I knew it would be asked
and b) I knew that upgrading is far and away the most common piece of
advice given on lists like these. The 'why' is a combination of
bureaucracy and technology, and something that I, as a temp worker, cannot
overcome at this time.

The meaning of my question is twofold. First, if this is a known issue
and what I'm seeing is, in fact, what's intended to happen on systems of
this size. Second, is there any feature, flag, or setting that is either
being overlooked in my rsync command or is otherwise undocumented. Is
there some sort of switch that can be flicked either in the rsync setup or
my unitilization of it to force it to put out full filenames that is not
covered in the manpage, or is there and I'm not seeing it?

If the answer to these questions are 'yes' for 1 and 'no' for 2, fine, I
can continue to look at workaround options. But if there is a working
solution to this quandary I'm eager to read it.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync shortcutting output?

2009-12-17T07:19:17Z

On Thu 17 Dec 2009, Joel Peabody wrote:
>
> PS, this is between two Redhat machines using rsync 2.6.8 on both ends.
> Upgrading rsync is not an option in this case.

What I always wonder when reading such messages, is what type of
response is expected, if a problem with the tool is suspected but
upgrading "is not an option"? (Why not?) That we travel back in time
to fix the tool before the person in question installs it? :-)

Paul
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync shortcutting output?

2009-12-17T06:49:01Z

From what Ive read, there are no hard limits on what rsync will accept as
far as amount of data to transfer, number of files, or the size of any one
file. However, in looking at the output of one particular job it looks
like its just putting out directories when it gets to a new one, and not
the filename of everything its transferring. And in some cases where
I've tracked the files down by file size, it will skip the directory name
when it moves on. For instance (snipped from
top of output):
print/relationship/brochures/2010/Brochure/layouts/final 4096
print/relationship/brochures/2010-q2/Brochure/layouts/final/translations/DE
4096
0 0% 0.00kB/s 0:00:00
28093832 55% 26.77MB/s 0:00:00
42970704 84% 20.41MB/s 0:00:00
49764768 98% 15.70MB/s 0:00:00
50647040 100% 15.48MB/s 0:00:03 (xfer#1, to-check=69938/80475)
print/relationship/brochures/2010/Brochure/layouts/final/Translations/EN-GB
4096
print/relationship/brochures/2010/Brochure/layouts/final/Translations/ES
4096
print/relationship/brochures/2010/Brochure/layouts/final/Translations/IT
4096
0 0% 0.00kB/s 0:00:00
2714504 5% 2.56MB/s 0:00:17
4593352 9% 2.17MB/s 0:00:19
19630992 41% 6.19MB/s 0:00:04
33308520 69% 7.85MB/s 0:00:01
42350280 88% 9.32MB/s 0:00:00
44311136 92% 9.35MB/s 0:00:00
47788032 100% 7.06MB/s 0:00:06 (xfer#2, to-check=69899/80475)
0 0% 0.00kB/s 0:00:00
565947 51% 529.39kB/s 0:00:01
1109277 100% 803.62kB/s 0:00:01 (xfer#3, to-check=69898/80475)

Id really like to know the name of each file that is being transferred.
However, this job covers apparently >80K files comprising 800GB worth of
data, so I suspect its skipping the explicit filename line that I see in
smaller jobs. This job runs daily, and there are usually fewer than 1000
files actually transferred in any one run.

The command I use is:

/usr/bin/rsync -rpti --progress --stats --timeout=20 --exclude='.*'
--delete-after --log-format="%f %l" source.server.com:/raid/studio/
/raid/studio-bak/ > /var/www/html/rsmod/logs/FileList.txt

Ive tried adding v and vv to the flags, with no help.

PS, this is between two Redhat machines using rsync 2.6.8 on both ends.
Upgrading rsync is not an option in this case.

Any advice would be appreciated.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

DO NOT REPLY [Bug 6996] New: syncing backups - autodetect older variants already existing on receiver

2009-12-17T00:56:53Z

https://bugzilla.samba.org/show_bug.cgi?id=6996

Summary: syncing backups - autodetect older variants already
existing on receiver
Product: rsync
Version: 3.1.0
Platform: Other
OS/Version: Linux
Status: NEW
Severity: enhancement
Priority: P3
Component: core
AssignedTo: wayned@...
ReportedBy: bugzilla.samba.org@...
QAContact: rsync-qa@...

We use virtualization, each server makes it's backup to a central backup
server. That backup server rsyncs every day the diffs to another site.

Many of the servers make a "backup.tar.bz2", and keep the older version as
"backup.tar.bz2.old" or "backup_old.tar.bz2". These files tend to be large, so
it would be nice not to sync them, just rename them on the receiver site.

rsync currently transfers the "*old*" version despite it exists already on the
receiver, it's just still named "backup.tar.bz2" and would need to be renamed
to "backup_old.tar.bz2" (or backup.tar.bz2.old depending on how it is named
now).

It would be very interesting to have a feature to search
- within the same dir for files with "similar" names and autodetect if they are
the same
- within the same dir for subdirs which have the same content and have just
been renamed. This is for backups that do "backup/0", "backup/1", and so on,
just renaming the dirs each day.

That should result in just a rename on the receiver side. It could reduce the
backup traffic a lot. Maybe name that options "--search-similar"

I think nearly everybody uses "generations" of backups, just renaming them on
the original server. So this would free the Internet of unneccessary traffic
during the night.

--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

DO NOT REPLY [Bug 6916] Avoid bundling a modified zlib

2009-12-17T00:23:31Z

https://bugzilla.samba.org/show_bug.cgi?id=6916

------- Comment #5 from jzeleny@... 2009-12-17 02:23 CST -------
Well, it's possible. I'm not rsync maintainer for long, so I don't have that
much knowledge about the situation. But my point was that zlib's upstream is
willing to consider some changes in order to reach compromise. Could you please
contact them and try to discuss the situation, so it isn't necessary to bundle
modified zlib? Thanks

--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

DO NOT REPLY [Bug 6995] New: Suggested new verbosity level

2009-12-16T17:25:31Z

https://bugzilla.samba.org/show_bug.cgi?id=6995

Summary: Suggested new verbosity level
Product: rsync
Version: 2.6.9
Platform: All
OS/Version: Other
Status: NEW
Severity: minor
Priority: P3
Component: core
AssignedTo: wayned@...
ReportedBy: phoffman@...
QAContact: rsync-qa@...

It would be nice to have a verbosity level that prints out new files that are
copied, but not the (in my case zillions of) directories that are looked into.

--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Option to create ancestors of destination, like "mkdir -p

2009-12-16T14:30:04Z

On Wed, Dec 16, 2009 at 1:38 PM, Tony Abernethy <tony@...> wrote:

The existing default has saved my bacon several times.

Yeah. I don't think I'd ever change the default, though adding an option is possible. However, there are several ways to create paths remotely with rsync now. For example:

mkdir -p /var/tmp/backup.$$/please/create/this/path

rsync -aivR /var/tmp/backup.$$/./please/create/this/path host:/dest/

rm -rf /var/tmp/backup.$$

rsync -aiv /src/ host:/dest/please/create/this/path/

But, admittedly, that would be a good deal easier like this (picking an option out of thin air):

rsync -aiv --create-dest-dirs /src/ host:/dest/please/create/this/path/

Something like that does seem fairly useful to me.

..wayne..

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --timeout not honoured

2009-12-16T14:19:46Z

On Tue, Dec 15, 2009 at 11:41 PM, Fabian Cenedese <Cenedese@...> wrote:

I think I know now where my confusion comes from. The reported time is not
the actual time without action but the time since the start of the rsync command.

It shouldn't be, since it is reporting the time elapsed since the last input was received. I'd suggest running an strace on the rsync and seeing what it is doing. I'd imagine that some system call does not return for an extended amount of time, and then, when rsync finally gets control back and checks for the elapsed time since the last received input, it is a really long amount.

..wayne..
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

RE: Option to create ancestors of destination, like "mkdir -p

2009-12-16T13:38:27Z

henri wrote:

> >> > I agree with everyone else that it would be incredibly useful for
> rsync to
> >> > either adopt such behaviour by default (assuming always doing
> 'mkdir -p' isn't
> >> > harmful in any way), or have a tunable to enable it.
> >>
> >> I also agree the suggested behavior makes a lot of sense and I think
> it is a good idea.
> >>
> >> However, some deployments may rely upon the current behavior. As an
> example, a setup may rely on rsync failing if a file system is not
> mounted. In such a situation, it is worth contemplating whether you
> would you want rsync to generate the mount point and any
> subdirectories, specified for this rsync.
> >>
> >> I personally think that such checks should be external from rsync.
> However, changing the current default behavior has the potential to
> create problems on setups which may rely on the current default
> behavior.
> >>
> >> I recommend the consideration of potential problems which could
> occur if the deployed setups are dependent on the current rsync
> behavior and that default behavior changes. Then based upon the
> severity of these potential problems move ahead or delay the change to
> a major revision change?
> >
>
> > #!/bin/sh
> > #backup friday
> > #"Spinning up backup drive and mounting it .."
> > cd /
> > mount /mnt/hd2
> > #"Starting backup procedures"
> > rsync -avx --exclude="/home/gary/.thumbnails/"
> > --exclude="/home/gary/tmp/" --delete --ignore-errors /home/gary
> > /mnt/hd2/2010
> > umount /mnt/hd2
> > #END
>
> The quoted script above is from another thread which was just posted to
> the rsync mailing list : <http://www.mail-
> archive.com/rsync@.../msg25132.html>
>
> This script above is a good example of the potential problems with
> making 'mkdir -p' the default. I agree that this feature is the way to
> move forward. However, changing the default behavior may cause
> potential issues for various scripts.
>
> I have no idea how many scripts are there like this, which could
> potentially start start backing up to the wrong file system if the
> drive is not successfully mounted.
>
> I suggest considering these potential issues and suggest that leaving
> such a change to the next major release is seriously considered.
>
> my 4ç
>
>
> ---------------------------------------------------------------------
> This email is protected by LBackup, an open source backup solution.
> Free as in freedom; LBackup is licensed under the GNU GPL
> Download LBackup now : http://www.lbackup.org
> ---------------------------------------------------------------------
>

Allow me to chime in with my own not-so-expert experience.
With large masses of data to keep updated, with bad internet connection
at time between locations, with the masses of data sometimes being moved
around, I experience the following:
If I mistype something in the path to the destination, the rsync errors
out and does so very quickly. Making new subdirectories to conform to my
mistyping looks friendlier ... at least until I suffer the consequences.
The existing default has saved my bacon several times.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Option to create ancestors of destination, like "mkdir -p

2009-12-16T13:25:43Z

>> > I agree with everyone else that it would be incredibly useful for rsync to
>> > either adopt such behaviour by default (assuming always doing 'mkdir -p' isn't
>> > harmful in any way), or have a tunable to enable it.
>>
>> I also agree the suggested behavior makes a lot of sense and I think it is a good idea.
>>
>> However, some deployments may rely upon the current behavior. As an example, a setup may rely on rsync failing if a file system is not mounted. In such a situation, it is worth contemplating whether you would you want rsync to generate the mount point and any subdirectories, specified for this rsync.
>>
>> I personally think that such checks should be external from rsync. However, changing the current default behavior has the potential to create problems on setups which may rely on the current default behavior.
>>
>> I recommend the consideration of potential problems which could occur if the deployed setups are dependent on the current rsync behavior and that default behavior changes. Then based upon the severity of these potential problems move ahead or delay the change to a major revision change?
>

> #!/bin/sh
> #backup friday
> #"Spinning up backup drive and mounting it .."
> cd /
> mount /mnt/hd2
> #"Starting backup procedures"
> rsync -avx --exclude="/home/gary/.thumbnails/"
> --exclude="/home/gary/tmp/" --delete --ignore-errors /home/gary
> /mnt/hd2/2010
> umount /mnt/hd2
> #END

The quoted script above is from another thread which was just posted to the rsync mailing list : <http://www.mail-archive.com/rsync@.../msg25132.html>

This script above is a good example of the potential problems with making 'mkdir -p' the default. I agree that this feature is the way to move forward. However, changing the default behavior may cause potential issues for various scripts.

I have no idea how many scripts are there like this, which could potentially start start backing up to the wrong file system if the drive is not successfully mounted.

I suggest considering these potential issues and suggest that leaving such a change to the next major release is seriously considered.

my 4ç

---------------------------------------------------------------------
This email is protected by LBackup, an open source backup solution.
Free as in freedom; LBackup is licensed under the GNU GPL
Download LBackup now : http://www.lbackup.org
---------------------------------------------------------------------

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync exclude

2009-12-16T13:08:50Z

That worked. Thank you Tony.

Gary

Tony wrote:

> try:
> --exclude=.thumbnails/***
> --exclude=tmp/***
>
> On Dec 16, 2009, at 1:40 PM, Gary Montalbine wrote:
>
>>
>> Tony Abernethy wrote:
>>> Gary Montalbine wrote:
>>>> I am trying to backup my /home directory. A friend helped me with this
>>>> script:
>>>>
>>>> #!/bin/sh
>>>> #backup friday
>>>> #"Spinning up backup drive and mounting it .."
>>>> cd /
>>>> mount /mnt/hd2
>>>> #"Starting backup procedures"
>>>> rsync -avx --exclude="/home/gary/.thumbnails/"
>>>> --exclude="/home/gary/tmp/" --delete --ignore-errors /home/gary
>>>> /mnt/hd2/2010
>>>> umount /mnt/hd2
>>>> #END
>>>>
>>>> It backs up /home OK. However it does not exclude .thumbnails or tmp. I
>>>> have read the manual and am confused on the use of --exclude. I am
>>>> strictly a user. I am using rsync supplied by ML2010.
>>>>
>>>> Help would be greatly appreciated.
>>>>
>>>> Thanks,
>>>>
>>>> Gary

Re: rsync exclude

2009-12-16T12:26:46Z

try:
--exclude=.thumbnails/***
--exclude=tmp/***

On Dec 16, 2009, at 1:40 PM, Gary Montalbine wrote:

>
>
> Tony Abernethy wrote:
>> Gary Montalbine wrote:
>>> I am trying to backup my /home directory. A friend helped me with this
>>> script:
>>>
>>> #!/bin/sh
>>> #backup friday
>>> #"Spinning up backup drive and mounting it .."
>>> cd /
>>> mount /mnt/hd2
>>> #"Starting backup procedures"
>>> rsync -avx --exclude="/home/gary/.thumbnails/"
>>> --exclude="/home/gary/tmp/" --delete --ignore-errors /home/gary
>>> /mnt/hd2/2010
>>> umount /mnt/hd2
>>> #END
>>>
>>> It backs up /home OK. However it does not exclude .thumbnails or tmp. I
>>> have read the manual and am confused on the use of --exclude. I am
>>> strictly a user. I am using rsync supplied by ML2010.
>>>
>>> Help would be greatly appreciated.
>>>
>>> Thanks,
>>>
>>> Gary
>> from man page
>> --exclude=PATTERN exclude files matching PATTERN
>> Seems like you are excluding /home/gary/home/gary/.thumbnails/
>> /home/gary/home/gary/tmp/
>> you probably want something like
>> --exclude=/.thumbnails
>> --exclude=/.tmp
>> to match where /home/gary/ leaves off.
>
> I tried various combinations of what you said. However I could not get it to exclude either file.
>
> With my low level of expertise I need a precise script.
>
> Thanks, Gary
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync exclude

2009-12-16T10:40:11Z

Tony Abernethy wrote:

> Gary Montalbine wrote:
>> I am trying to backup my /home directory. A friend helped me with this
>> script:
>>
>> #!/bin/sh
>> #backup friday
>> #"Spinning up backup drive and mounting it .."
>> cd /
>> mount /mnt/hd2
>> #"Starting backup procedures"
>> rsync -avx --exclude="/home/gary/.thumbnails/"
>> --exclude="/home/gary/tmp/" --delete --ignore-errors /home/gary
>> /mnt/hd2/2010
>> umount /mnt/hd2
>> #END
>>
>> It backs up /home OK. However it does not exclude .thumbnails or tmp. I
>> have read the manual and am confused on the use of --exclude. I am
>> strictly a user. I am using rsync supplied by ML2010.
>>
>> Help would be greatly appreciated.
>>
>> Thanks,
>>
>> Gary
>
> from man page
> --exclude=PATTERN exclude files matching PATTERN
> Seems like you are excluding
> /home/gary/home/gary/.thumbnails/
> /home/gary/home/gary/tmp/
> you probably want something like
> --exclude=/.thumbnails
> --exclude=/.tmp
> to match where /home/gary/ leaves off.
>

I tried various combinations of what you said. However I could not get
it to exclude either file.

With my low level of expertise I need a precise script.

Thanks, Gary
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

RE: rsync exclude

2009-12-16T08:41:56Z

Gary Montalbine wrote:

> I am trying to backup my /home directory. A friend helped me with this
> script:
>
> #!/bin/sh
> #backup friday
> #"Spinning up backup drive and mounting it .."
> cd /
> mount /mnt/hd2
> #"Starting backup procedures"
> rsync -avx --exclude="/home/gary/.thumbnails/"
> --exclude="/home/gary/tmp/" --delete --ignore-errors /home/gary
> /mnt/hd2/2010
> umount /mnt/hd2
> #END
>
> It backs up /home OK. However it does not exclude .thumbnails or tmp. I
> have read the manual and am confused on the use of --exclude. I am
> strictly a user. I am using rsync supplied by ML2010.
>
> Help would be greatly appreciated.
>
> Thanks,
>
> Gary

from man page
--exclude=PATTERN exclude files matching PATTERN
Seems like you are excluding
/home/gary/home/gary/.thumbnails/
/home/gary/home/gary/tmp/
you probably want something like
--exclude=/.thumbnails
--exclude=/.tmp
to match where /home/gary/ leaves off.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync exclude

2009-12-16T07:47:27Z

I am trying to backup my /home directory. A friend helped me with this
script:

#!/bin/sh
#backup friday
#"Spinning up backup drive and mounting it .."
cd /
mount /mnt/hd2
#"Starting backup procedures"
rsync -avx --exclude="/home/gary/.thumbnails/"
--exclude="/home/gary/tmp/" --delete --ignore-errors /home/gary
/mnt/hd2/2010
umount /mnt/hd2
#END

It backs up /home OK. However it does not exclude .thumbnails or tmp. I
have read the manual and am confused on the use of --exclude. I am
strictly a user. I am using rsync supplied by ML2010.

Help would be greatly appreciated.

Thanks,

Gary

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync hang on large file in a set

2009-12-16T06:34:22Z

This is to both the rsync and cygwin lists, and I
regret that it is vaguer than I would like, but ...

With cygwin 1.7 (latest beta) and the rsync released
with it (3.0.6-1) I try to sync a couple of Gb of
Thunderbird mail files. The flag settings were
-avzuP plus a --chmod and some includes/excludes
to control which files to move.

I would repeatably get "hangs" on particular files,
which are generally modified by appending (mail
added at the end). These files (two in particular)
are approximately 100Mb or a little more in size,
and that seemed to have something to do with it.

The "hangs" I speak of are where rsync and ssh spin,
consuming cpu time (fan running fast!) but making
no progress.

Now here's the odd thing. If I kill of the rsync and
ssh processes and rsync the one file by itself, it
will go through. (Of course the --partial implied
by -P means that I may have to transfer a lot of
data :-( ).

Curiously, I got the same behavior with a recent rsync
head (20091129).

Last night I made two changes and have not seen a hang
since (though I am monitoring this, of course). One
was to substitute --progress and --partial-dir for
-P.

The other -- which is the cygwin relevant part -- was
the compile with socketpair turned off (manually in the
config file). I did this because Corinna and I had found
issues between socketpair and BLODA on my (Windows 7) box,
which she appeared to have fixed in revisions to 1.7 The
problems had to do with fork, dup, and such, not with
actual data transfer AFAIK. At her request I still have
the BLODA installed, and other cygwin things generally
work right.

I will also play with putting socketpair back in and
keeping --partial-dir. But the various reports of rsync
hangs, and a particular report a little while ago of
busy looping in buffer transfer (and I do have -z on,
might that impact?) struck me as possibly related. I
am not sure how to debug this when it happens, so if
there are steps I can take upon such a hang to get you
better info, do enlighten me!

Best wishes -- Eliot Moss
==============================================================================
J. Eliot B. Moss, Professor http://www.cs.umass.edu/~moss www
Director, Arch. and Lang. Impl. Lab. +1-413-545-4206 voice
Department of Computer Science +1-413-695-4226 cell
140 Governor's Drive, Room 372 +1-413-545-1249 fax
University of Massachusetts at Amherst moss@... email
Amherst, MA 01003-9264 USA +1-413-545-2746 Laurie Downey sec'y
==============================================================================
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --timeout not honoured

2009-12-15T23:41:43Z

At 10:27 15.12.2009 +0100, Fabian Cenedese wrote:

>Hi
>
>I already wrote about this problem half a year ago but didn't get
>an answer: http://lists.samba.org/archive/rsync/2009-June/023412.html
>
>I'm using rsync 3.0.3 on a NAS. In the parameter list I use --timeout=1800.
>But still I often have rsync stall much longer than that.
>
>2009/12/14 23:14:35 [8707] io timeout after 11670 seconds -- exiting
>2009/12/14 23:14:35 [8707] rsync error: timeout in data send/receive (code 30) at io.c(239) [sender=3.0.3pre1]
>
>2009/12/14 00:20:04 [19046] io timeout after 15485 seconds -- exiting
>2009/12/14 00:20:04 [19046] rsync error: timeout in data send/receive (code 30) at io.c(239) [sender=3.0.3pre1]
>
>However, why is it that rsync waits much longer than given in --timeout?
>Shouldn't timeout work that way?

I think I know now where my confusion comes from. The reported time is not
the actual time without action but the time since the start of the rsync command.

2009/12/15 20:00:02 [4260] building file list
2009/12/15 20:00:03 [4260] .d..t...... vmware/FTPServer2/
2009/12/15 20:23:45 [4260] <f.st...... vmware/FTPServer2/FTPServer2-s017.vmdk
2009/12/15 22:12:25 [4260] <f+++++++++ vmware/FTPServer2/FTPServer2-s018.vmdk
2009/12/15 23:10:19 [4260] io timeout after 11372 seconds -- exiting
2009/12/15 23:10:19 [4260] rsync error: timeout in data send/receive (code 30) at io.c(239) [sender=3.0.3pre1]

11372 seconds is 189.something minutes, so 20:00+3h09 makes about the 23:10
timeout time. Same seems to be true for my previous examples, all started at 20:00.

So --timeout does seem to be honoured, just the reported timeout time is a bit
misleading. If this is not to be changed maybe an explaining comment in the
docs might be a good idea.

Thanks

bye Fabi

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Option to create ancestors of destination, like "mkdir -p

2009-12-15T17:50:42Z

> I agree with everyone else that it would be incredibly useful for rsync to
> either adopt such behaviour by default (assuming always doing 'mkdir -p' isn't
> harmful in any way), or have a tunable to enable it.

I also agree the suggested behavior makes a lot of sense and I think it is a good idea.

However, some deployments may rely upon the current behavior. As an example, a setup may rely on rsync failing if a file system is not mounted. In such a situation, it is worth contemplating whether you would you want rsync to generate the mount point and any subdirectories, specified for this rsync.

I personally think that such checks should be external from rsync. However, changing the current default behavior has the potential to create problems on setups which may rely on the current default behavior.

I recommend the consideration of potential problems which could occur if the deployed setups are dependent on the current rsync behavior and that default behavior changes. Then based upon the severity of these potential problems move ahead or delay the change to a major revision change?

My 2ç

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: retransfer fail of large files with inplace and broken pipe

2009-12-15T09:25:40Z

OK,

have tried now --inplace with --backup option but syncing the files does
consume much more time than a normal rsync process,
so this is not a reliable solution.

Thx
Tom

"Tom" <raschel@...> schrieb im Newsbeitrag
news:hg7dg6$emj$1@......

> Hi,
>
> retransfer of large fail with inplace after a broken pipe is working now.
> (thx again to wayne)
>
> but it is much more slow as if a "normal" rsync job.
>
> I have read that setting the --backup option could help. (have not tried
> it
> yet)
>
> But --backup option would halve the space, which is not desirable.
>
> Is there a way to tell rsync to delete the --backup file after an
> successful
> sync.
>
> thx
> Tom
>
>
> "tom raschel" <raschel@...> schrieb im Newsbeitrag
> news:loom.20091213T075221-530@......
>> Hi,
>>
>> i have to tranfer large files each 5-100 GB (mo-fri) over dsl line.
>> unfortunately dsl lines are often not very stable and i got a broken pipe
>> error.
>> (dsl lines are getting a new ip if they are broken or at least after a
>> reconnect every 24 hours)
>>
>> i had a script which detect the rsync error and restart the transmission.
>> this means that if a file has transfered e.g. 80 % i start again from
>> beginning.
>>
>> using partial and partial-dir was no solution to resync because rsync cut
>> the
>> original file (e.g. from 20 GB to 15 GB) which means that i have to
>> transfer
>> the whole rest of 5 GB.
>>
>> so i had a look at --inplace which I thougt could do the trick, but
>> inplace is
>> updating the timestamp and if the script start a retransfer after a
>> broken pipe
>> it fails because the --inplace file is newer than the original file of
>> the
>> sender.
>>
>> using ignore-times could be a solution but slow down the whole process to
>> much.
>>
>> is there a option to tell rsync not to change the time of a --inplace
>> transfered file, or maybe preserve the mtime and do a comparison of mtime
>> instead of ctime.
>>
>> Thx
>> Tom
>>
>>
>>
>> --
>> Please use reply-all for most replies to avoid omitting the mailing list.
>> To unsubscribe or change options:
>> https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>>
>
>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>

Uninstalling rsync 3.0.6

2009-12-15T08:12:35Z

Hello,

I have installed rsync from source but when i want to remove it im not able to do it can anyone help me that would be great.

Kind regards,

Andy

DO NOT REPLY [Bug 4621] Option to create ancestors of destination, like "mkdir -p"

2009-12-15T04:12:28Z

https://bugzilla.samba.org/show_bug.cgi?id=4621

lists@... changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |lists@...

------- Comment #4 from lists@... 2009-12-15 06:12 CST -------
Greetings,

I just noticed my rsync backup scripts failing and discovered it was due to a
missing ancestor of the remote target directory. I immediately searched for a
tunable that would signal rsync to use 'mkdir -p' instead of just 'mkdir' when
initially creating the target directory, but didn't seem to find one.

I agree with everyone else that it would be incredibly useful for rsync to
either adopt such behaviour by default (assuming always doing 'mkdir -p' isn't
harmful in any way), or have a tunable to enable it.

Kind regards,
moggie

--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--timeout not honoured

2009-12-15T01:27:15Z

Hi

I already wrote about this problem half a year ago but didn't get
an answer: http://lists.samba.org/archive/rsync/2009-June/023412.html

I'm using rsync 3.0.3 on a NAS. In the parameter list I use --timeout=1800.
But still I often have rsync stall much longer than that.

Some examples:

2009/12/14 23:14:35 [8707] io timeout after 11670 seconds -- exiting
2009/12/14 23:14:35 [8707] rsync error: timeout in data send/receive (code 30) at io.c(239) [sender=3.0.3pre1]

2009/12/14 00:20:04 [19046] io timeout after 15485 seconds -- exiting
2009/12/14 00:20:04 [19046] rsync error: timeout in data send/receive (code 30) at io.c(239) [sender=3.0.3pre1]

2009/12/12 22:15:21 [29732] io timeout after 8082 seconds -- exiting
2009/12/12 22:15:21 [29732] rsync error: timeout in data send/receive (code 30) at io.c(239) [sender=3.0.3pre1]

2009/12/11 23:14:23 [30216] io timeout after 11613 seconds -- exiting
2009/12/11 23:14:23 [30216] rsync error: timeout in data send/receive (code 30) at io.c(239) [sender=3.0.3pre1]

I know that 3.0.3pre1 is not the newest version, but as this is running
on a NAS it's not that easy to update or compile a newer version.

However, why is it that rsync waits much longer than given in --timeout?
Shouldn't timeout work that way?

Thanks

bye Fabi

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: retransfer fail of large files with inplace and broken pipe

2009-12-14T23:23:03Z

Hi,

retransfer of large fail with inplace after a broken pipe is working now.
(thx again to wayne)

but it is much more slow as if a "normal" rsync job.

I have read that setting the --backup option could help. (have not tried it
yet)

But --backup option would halve the space, which is not desirable.

Is there a way to tell rsync to delete the --backup file after an successful
sync.

thx
Tom

"tom raschel" <raschel@...> schrieb im Newsbeitrag
news:loom.20091213T075221-530@......

> Hi,
>
> i have to tranfer large files each 5-100 GB (mo-fri) over dsl line.
> unfortunately dsl lines are often not very stable and i got a broken pipe
> error.
> (dsl lines are getting a new ip if they are broken or at least after a
> reconnect every 24 hours)
>
> i had a script which detect the rsync error and restart the transmission.
> this means that if a file has transfered e.g. 80 % i start again from
> beginning.
>
> using partial and partial-dir was no solution to resync because rsync cut
> the
> original file (e.g. from 20 GB to 15 GB) which means that i have to
> transfer
> the whole rest of 5 GB.
>
> so i had a look at --inplace which I thougt could do the trick, but
> inplace is
> updating the timestamp and if the script start a retransfer after a broken
> pipe
> it fails because the --inplace file is newer than the original file of the
> sender.
>
> using ignore-times could be a solution but slow down the whole process to
> much.
>
> is there a option to tell rsync not to change the time of a --inplace
> transfered file, or maybe preserve the mtime and do a comparison of mtime
> instead of ctime.
>
> Thx
> Tom
>
>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>

RE: Compression error (Re: rsync hangs during transfer)

2009-12-14T02:36:03Z

Hi Matt,

Thanks for your elaborate reply.

I am able to reproduce the error indeed running the suggested script:
inflate (token) returned -5
rsync error: error in rsync protocol data stream (code 12) at
token.c(476) [receiver=2.6.9]
rsync: connection unexpectedly closed (30 bytes received so far)
[generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(454)
[generator=2.6.9]
rsync: connection unexpectedly closed (40 bytes received so far)
[sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(454)
[sender=2.6.9]

The issue appears to be definitely related to the zlib library as the
rsync is running fine when the compression is turned off. I.e. running
rsync -axv --delete-after -e ssh /SRCDIR/ blabla@DEST:/DESTDIR.

I will also try to run the synchronization using the option
'--whole-file' for the next couple of days and will let you know my
findings.

Regards,
Dimitar

-----Original Message-----
From: Matt McCutchen [mailto:matt@...]
Sent: 14 December 2009 05:47
To: Dimitar Dimitrov; rsync
Subject: Compression error (Re: rsync hangs during transfer)

On Fri, 2009-12-11 at 08:58 +0100, Dimitar Dimitrov wrote:

> I am running the following rsync command to synchronize directories
> between two servers:
> rsync -axvz --delete-after -e ssh /SRCDIR/ blabla@DEST:/DESTDIR
>
> The transfer starts and after a short while it appears to hang after
> some files have been transferred. The process establishes connection
> on both sides so I did an strace from the remote machine (using the
> rsync-debug script as described in the troubleshooting procedures).
> The strace output is here
> http://www.adrive.com/public/142cbf351c4b73a47c6e54ec3302b856041957d61
> 2ca68f440d16eccf950225c.html
>
> rsync --version on both machines shows 'rsync version 2.6.9 protocol

> version 29' and both machines are running Debian.
>
> I would appreciate some ideas on where to look next.

I don't know about the hang. I see rsync working for a few minutes and
then exiting when the receiver encounters an error:

write(1, "\34\0\0\10inflate (token) returned -5\n", 32) = 32

This is the "compression error" that has occasionally come up before:

http://lists.samba.org/archive/rsync/2004-December/011119.html
http://lists.samba.org/archive/rsync/2006-May/015621.html
http://lists.samba.org/archive/rsync/2008-September/021668.html
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=528730

And after some study, I think I figured out why it is happening. The
error message comes from "see_deflate_token" in token.c, and the code -5
is Z_BUF_ERROR, meaning that the inflate call didn't make progress. But
a CHUNK_SIZE output buffer is always provided, and input is provided
unless len == 0 at the start of the loop. If len == 0, the loop would
have exited unless the previous "inflate" filled the output buffer, in
which case we want to call it again to obtain any remaining output. But
if the data block was exactly CHUNK_SIZE (32816), it would fill the
output buffer with nothing remaining, and the next call to "inflate"
would return Z_BUF_ERROR.

This case is in fact mentioned in the zlib FAQ
(http://www.zlib.net/zlib_faq.html#faq05): "A Z_BUF_ERROR may in fact be
unavoidable depending on how the functions are used, since it is not
possible to tell whether or not there is more output pending when
strm.avail_out returns with zero."

The block size is indeed 32816, as one can see in the second 32-bit
field of the sum head. The sum head is shown in the following line
after two 6-byte itemizations:

[pid 5213] write(1,
"\252\26\0\0\10\0\262\26\0\0\f\2002\200\0\0000\200\0\0\3"..., 4092) =
4092

Here's a simple script to reproduce the problem:

#!/bin/bash
head -c 32816 /dev/zero >srcfile
cp srcfile destfile
rsync -I -z --no-whole-file --block-size=32816 srcfile destfile

So, we need to do something to "see_deflate_token". It would probably
work to ignore the Z_BUF_ERROR and let the loop exit because the output
buffer wasn't filled.

It seems inconsistent that the sender uses Z_INSERT_ONLY while the
receiver uses this hack of synthesizing part of the compressed stream.
(Previously, I had imagined Z_INSERT_ONLY worked on both sides.)

--
Matt

The content of this e-mail and accompanying communications and attachments (collectively, this "e-mail") are confidential to Markit Group Holdings Limited, its subsidiaries and affiliates (collectively, "Markit") and may contain information which is legally privileged or protected from disclosure under applicable law or agreement. This email may be read and used only by the intended recipient, and any disclosure, printing, copying, distribution (including forwarding), use, saving, or taking any action based on, the information contained herein (including any reliance thereon) is expressly prohibited. If you received this email in error, please contact the sender immediately by return e-mail or by telephoning +44 20 7260 2000 and delete it. You agree to take full responsibility for checking this email for viruses, and Markit shall not be responsible or liable for any damages arising from or relating to its use. Markit reserves the right to monitor all e-mail communications
through its networks. Markit makes no warranty as to the accuracy or completeness of this email and hereby disclaims any liability of any kind for the information contained herein. Any opinions expressed in this email are those of the author and do not necessarily reflect the opinions of Markit. For full details about Markit, its offerings and legal terms and conditions, please see Markit's website at http://www.markit.com <http://www.markit.com/> .
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync speed on slow wireless links

2009-12-14T00:18:35Z

On Mon, Dec 14, 2009 at 02:23:26AM -0500, Matt McCutchen wrote:
> On Thu, 2009-12-10 at 13:08 +0700, Max Arnold wrote:
> > I've noticed that rsync performs significantly worse than wget on slow congested wireless
> > links (GPRS in my case). I don't have large statistics, but in my tests rsync often stalls
> > for 3-5 minutes, while wget stalls only for several seconds and then continues download.
> >
> Rsync isn't doing anything fancy that would cause it to be especially
> affected by packet loss or delay. The protocol takes a few round trips
> to set up, and then it is largely pipelined, so rsync can tolerate some
> amount of latency without slowing down the whole process.

Can size of this initial metadata be approximately calculated? I plan to experiment with
different timeout values to find a balance between link utilization (by preventively aborting
long stalls) and traffic consumption (by not retrying very often to prevent metadata overhead
accumulation).

> I can't explain why rsync would stall much longer than wget. The only
> thought I had is that the network might have a quality-of-service policy
> that favors port 80.

No, it probably haven't, because I've also tried to use OpenVPN which hides internal traffic
from inspection and symptoms are the same.

Is someone here have any experience with different Linux tcp congestion control algorithms
suitable for cellular networks? What about adjusting send/receive buffers? It seems that low
level radio link protocols and equipment do heavy buffering, which reduces packet loss but
introduces unpredictable delays (up to tens of seconds). I.e. link equipment knows that there
is no radio resources available and buffers packets (maybe even does retransmissions if
necessary). And this probably fools tcp protocol which thinks there is congestion and packet
loss.

Thanks for replying!
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

DO NOT REPLY [Bug 6965] Avoid code 23 when source file is concurrently truncated

2009-12-13T23:42:02Z

https://bugzilla.samba.org/show_bug.cgi?id=6965

matt@... changed:

What |Removed |Added
----------------------------------------------------------------------------
Summary|truncated files and exit |Avoid code 23 when source
|code 23 |file is concurrently
| |truncated

------- Comment #1 from matt@... 2009-12-14 01:42 CST -------
Indeed. This is probably one of many kinds of concurrent modification to the
source that rsync handles ungracefully.

In general, when different system calls return inconsistent results, that could
indicate either a concurrent modification or filesystem misbehavior. The
downside to always assuming concurrent modification is that we miss the
opportunity to report filesystem misbehavior if it happens. One example is the
Cygwin "file has vanished" error with foreign characters in filenames (
http://lists.samba.org/archive/rsync/2008-January/019696.html ).

--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync speed on slow wireless links

2009-12-13T23:23:26Z

On Thu, 2009-12-10 at 13:08 +0700, Max Arnold wrote:
> I've noticed that rsync performs significantly worse than wget on slow congested wireless
> links (GPRS in my case). I don't have large statistics, but in my tests rsync often stalls
> for 3-5 minutes, while wget stalls only for several seconds and then continues download.
>
> Is there any rsync protocol features which are sensitive to packet loss and unpredictable delay?
> I understand that rsync transaction has additional metadata transfer in the beginning (file
> list etc.), but after that what can affect the speed? Are there additional handshakes or
> flow control other than tcp ones?

> rsync -rtmz --partial --progress --delete-delay --delay-updates --stats --itemize-changes
> --timeout 120 --contimeout 20 source.example.com::module/file.zip /tmp/

Rsync isn't doing anything fancy that would cause it to be especially
affected by packet loss or delay. The protocol takes a few round trips
to set up, and then it is largely pipelined, so rsync can tolerate some
amount of latency without slowing down the whole process.

I can't explain why rsync would stall much longer than wget. The only
thought I had is that the network might have a quality-of-service policy
that favors port 80.

--
Matt

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: retransfer fail of large files with inplace and broken pipe

2009-12-13T22:06:32Z

Thx to all,

it was the -u option which prevents rsync to resume the file.

Tom

"Tony Abernethy" <tony@...> schrieb im Newsbeitrag
news:AF5EF1769D564645A9ACC947375F0D0215670875E6@......

> Tom wrote:
>> to make things more clear
>>
>> 1.)
>> first transfer is done either a initial setup or with a usb hdd to get
>> sender and receiver in sync.
>>
>> 2.)
>> transfer does not stop because rsync had a timeout, it stops because
>> the dsl
>> line is broken (which i could see at dyndns)
>>
>> 3)
>> if dsl line is stable the transfer is successfull (which works
>> furtunately
>> most of the time)
>>
>> 4.)
>> i am searching for a way to reduce the time to retransfer the file or
>> in
>> other words to resume the filetransfer after a broken pipe
>> (e.g. if you download a 4.4 GB Centos Image it is comfortable to resume
>> the
>> transfer of a 99 % transfered file instead to download all from
>> scratch)
>>
>> Tom
>
>
> But you already have 100% of the image, only it is an older version of the
> image.
> The only thing I've found that works (and this is ONLY on something UNIXy)
> is
> to monitor the temporary file on the target and if it is "big enough",
> rename it
> to the intended target file before the target rsync destroys it.
> For real disasters, you can attempt to automate this process.
>
>
>> > First: Transfer or re-transfer. I think, particularly with bad
>> > connections,
>> > you need to treat those VERY differently.
>> >
>> > For the initial transfer, --partial should help.
>> >
>> > For retransfers, where stuff in the middle has changed, I would
>> expect the
>> > necessary state information to exist ONLY in the two running
>> processes,
>> > and
>> > that information is lost if the connection goes down.
>> > This includes the connection dying because both sides are going
>> through
>> > the
>> > file and have nothing worthwhile to say to each other.
>> >
>> > As usual, flames invited if I've got any of this wrong.
>> > --
>> > Please use reply-all for most replies to avoid omitting the mailing
>> list.
>> > To unsubscribe or change options:
>> > https://lists.samba.org/mailman/listinfo/rsync
>> > Before posting, read: http://www.catb.org/~esr/faqs/smart-
>> questions.html
>> >
>>
>>
>>
>> --
>> Please use reply-all for most replies to avoid omitting the mailing
>> list.
>> To unsubscribe or change options:
>> https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-
>> questions.html
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>

Compression error (Re: rsync hangs during transfer)

2009-12-13T20:47:23Z

On Fri, 2009-12-11 at 08:58 +0100, Dimitar Dimitrov wrote:

> I am running the following rsync command to synchronize directories
> between two servers:
> rsync -axvz --delete-after -e ssh /SRCDIR/ blabla@DEST:/DESTDIR
>
> The transfer starts and after a short while it appears to hang after
> some files have been transferred. The process establishes connection
> on both sides so I did an strace from the remote machine (using the
> rsync-debug script as described in the troubleshooting procedures).
> The strace output is here
> http://www.adrive.com/public/142cbf351c4b73a47c6e54ec3302b856041957d612ca68f440d16eccf950225c.html
>
> rsync --version on both machines shows 'rsync version 2.6.9 protocol
> version 29' and both machines are running Debian.
>
> I would appreciate some ideas on where to look next.

I don't know about the hang. I see rsync working for a few minutes and
then exiting when the receiver encounters an error:

write(1, "\34\0\0\10inflate (token) returned -5\n", 32) = 32

This is the "compression error" that has occasionally come up before:

http://lists.samba.org/archive/rsync/2004-December/011119.html
http://lists.samba.org/archive/rsync/2006-May/015621.html
http://lists.samba.org/archive/rsync/2008-September/021668.html
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=528730

And after some study, I think I figured out why it is happening. The
error message comes from "see_deflate_token" in token.c, and the code -5
is Z_BUF_ERROR, meaning that the inflate call didn't make progress. But
a CHUNK_SIZE output buffer is always provided, and input is provided
unless len == 0 at the start of the loop. If len == 0, the loop would
have exited unless the previous "inflate" filled the output buffer, in
which case we want to call it again to obtain any remaining output. But
if the data block was exactly CHUNK_SIZE (32816), it would fill the
output buffer with nothing remaining, and the next call to "inflate"
would return Z_BUF_ERROR.

This case is in fact mentioned in the zlib FAQ
(http://www.zlib.net/zlib_faq.html#faq05): "A Z_BUF_ERROR may in fact be
unavoidable depending on how the functions are used, since it is not
possible to tell whether or not there is more output pending when
strm.avail_out returns with zero."

The block size is indeed 32816, as one can see in the second 32-bit
field of the sum head. The sum head is shown in the following line
after two 6-byte itemizations:

[pid 5213] write(1, "\252\26\0\0\10\0\262\26\0\0\f\2002\200\0\0000\200\0\0\3"..., 4092) = 4092

Here's a simple script to reproduce the problem:

#!/bin/bash
head -c 32816 /dev/zero >srcfile
cp srcfile destfile
rsync -I -z --no-whole-file --block-size=32816 srcfile destfile

So, we need to do something to "see_deflate_token". It would probably
work to ignore the Z_BUF_ERROR and let the loop exit because the output
buffer wasn't filled.

It seems inconsistent that the sender uses Z_INSERT_ONLY while the
receiver uses this hack of synthesizing part of the compressed stream.
(Previously, I had imagined Z_INSERT_ONLY worked on both sides.)

--
Matt

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: v3.0.4 and OSXS - Exit code: 1

2009-12-13T14:21:27Z

> When launchd executed this command, it did not work. The system.log file reads "Exited with exit code: 1".
>
> Can you redirect the output to a file so it can tell you why it failed? e.g. >file.out 2>&1
>
> One common thing that prevents non-interactive sudo is this tty setting:
>
> Defaults requiretty

I agree, it is very likely that sudo is asking for a password and this may be causing the "Exited with exit code:1" error message.

On possible alternative is to issue the rsync command directly as root.

This approach may not be suitable or even required in your setup. It will probably depend upon the owners and permissions of the files located within the /Volumes/File_Storage/docs/ directory on your system.

I would also suggest that you consider using the absolute path to the rsync command when it is run via LaunchD. Three common possibilities are listed below :

/opt/local/bin/rsync
/usr/local/bin/rsync_v3.0.6
/usr/bin/rsync

To find the absolute path to the version you are running, login to a shell on the server and issue the command below. You should find that the absolute path to the first rsync command in your search path will be returned :

which rsync

The reason I suggest specifying an absolute path is because, when you login to a terminal, it is very common that a search path has been set. This search path will be used to locate many commands, such as rsync on your system. The search path provides a places to search and also order if more than one command exists with the same name. It is possible to have more than one command with the same name installed on a system. The search path set in the terminal may be different to the search path set when the command is executed from LaunchD. As such this is quite important.

To determine your path issue the following command :

echo $PATH

It is quite likely that the search path returned is different when you are logged in to a terminal session differs from the search path provided when commands are started from LaunchD or cron.

More details are available from : http://en.wikipedia.org/wiki/PATH_(variable)

Hopefully this information is helpful and you will have your script working soon.

---------------------------------------------------------------------
This email is protected by LBackup, an open source backup solution.
Free as in freedom; LBackup is licensed under the GNU GPL
Download LBackup now : http://www.lbackup.org
---------------------------------------------------------------------

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: v3.0.4 and OSXS - Exit code: 1

2009-12-13T10:21:48Z

Wayne Davison wrote:

On Sun, Dec 13, 2009 at 8:49 AM, Jonathan S. Abrams <hociman@...> wrote:

When launchd executed this command, it did not work. The system.log file reads "Exited with exit code: 1".

Can you redirect the output to a file so it can tell you why it failed? e.g. >file.out 2>&1

One common thing that prevents non-interactive sudo is this tty setting:

Defaults requiretty

..wayne..

Wayne,

Thanks for responding. Would it be enough to instruct rsync to log activity to a file?

sudo rsync -aAHvWX --log-file=docs_transfer /Volumes/File_Storage/docs/ /Volumes/File_Storage_Mirror/docs

-Jonathan

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: retransfer fail of large files with inplace and broken pipe

2009-12-13T09:47:36Z

On Sat, Dec 12, 2009 at 11:21 PM, tom raschel <raschel@...> wrote:

so i had a look at --inplace which I thougt could do the trick, but inplace is updating the timestamp and if the script start a retransfer after a broken pipe it fails because the --inplace file is newer than the original file of the sender.

Are you using --update (-u)? If so, turn that off. If not, rsync won't skip a file that is newer, so something else is afoot.

..wayne..

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html