avoid rsync deleting vanished files on receiver

[Hans-Christian Jehg]

Hello All

I have a rather curious problem. I rsync from a fileserver shared in via
cifs. This connection sometimes fails (don't worry, that is not the
question I have).
I am not at all sure where it goes wrong, but in the end the result is
that rsync thinks certain files has disappeared. It could for instance
go wrong in the cifs client, but also in rsync when trying to read the
remote file (I am not using --ignore-errors) if no proper read checking
is in place (I do not think that this is the case but I would mention it
anyway).
My immediate problem is that rsync proceeds happily and deletes any
vanished files in the end (--delete-after).
rsync stops with exit code 24, but by then the damage is already done
(files deleted) .

The question is: Is there a way to avoid the deletion when vanished
files are detected? That would be "no deletion at all" or just "no
deleting of vanished files".

Best regards

Hans-Chr. Jehg

rsync  version 3.0.2  protocol version 30
Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, no symtimes

on Mandriva 2009


 rsync --modify-window=2 --times --omit-dir-times --verbose --stats
--recursive --partial-dir=.rsync-partial --include "some includes"
--exclude "*" --compress -e "ssh -o ServerAliveInterval=10 -p 22"
--delete --delete-after "user@...:~/somedirectory/"
"/data/user/somedirectory/"
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Hans-Christian Jehg]

Hello Again

The circumstance where this happens is now known, it is when requests to the shared in files returns "permission denied" from the file server...

Hmmmm...... Any ideas on how to avoid "permission denied" files ?

Would delete-delayed do the trick, do you think ?

Thanks in advance

HC Jehg

---------------------

Hans-Christian Jehg wrote:

Hello All

I have a rather curious problem. I rsync from a fileserver shared in via cifs. This connection sometimes fails (don't worry, that is not the question I have).
I am not at all sure where it goes wrong, but in the end the result is that rsync thinks certain files has disappeared. It could for instance go wrong in the cifs client, but also in rsync when trying to read the remote file (I am not using --ignore-errors) if no proper read checking is in place (I do not think that this is the case but I would mention it anyway).
My immediate problem is that rsync proceeds happily and deletes any vanished files in the end (--delete-after).
rsync stops with exit code 24, but by then the damage is already done (files deleted) .

The question is: Is there a way to avoid the deletion when vanished files are detected? That would be "no deletion at all" or just "no deleting of vanished files".

Best regards

Hans-Chr. Jehg

rsync  version 3.0.2  protocol version 30
Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
   64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
   socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
   append, ACLs, xattrs, iconv, no symtimes

on Mandriva 2009


rsync --modify-window=2 --times --omit-dir-times --verbose --stats --recursive --partial-dir=.rsync-partial --include "some includes" --exclude "*" --compress -e "ssh -o ServerAliveInterval=10 -p 22" --delete --delete-after user@... "/data/user/somedirectory/"

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Carlos Carvalho-3]

Hans-Christian Jehg (hc@...) wrote on 23 October 2009 17:32:
 >The circumstance where this happens is now known, it is when requests
 >to the shared in files returns "permission denied" from the file
 >server...
[snip]
 >I am not at all sure where it goes wrong, but in the end the result is
 >that rsync thinks certain files has disappeared. It could for instance
 >go wrong in the cifs client, but also in rsync when trying to read the
 >remote file (I am not using --ignore-errors) if no proper read checking
 >is in place (I do not think that this is the case but I would mention
 >it anyway).

 >My immediate problem is that rsync proceeds happily and deletes any
 >vanished files in the end (--delete-after).

 >rsync stops with exit code 24, but by then the damage is already done
 >(files deleted) .

Strange, I have code 23 when rsync gets permission denied. This is
important because vanished files are a normal occurrence and files
should be removed in this case. OTOH, code 23 is an error, and rsync
should not do deletions if you don't use --ignore-errors.

If the exit status is 23 you're likely hitting this bug:

Matt McCutchen (matt@...) wrote on 23 September 2009 22:19:
 >On Thu, 2009-09-17 at 20:59 +0200, Enno Middelberg wrote:
 >> Here the two directories which couldn't be read have been deleted from
 >> the target directory (ie, the files contained in them were deleted,
 >> the directories did still exist, if I remember correctly). If I
 >> understand the man page correctly, rsync should not delete files from
 >> the target directory when an error occurs. This can be overridden with
 >> the "--ignore-errors" command line switch, but this switch has
 >> definitely not been used.
 >
 >Maybe you are hitting this issue:
 >
 >https://bugzilla.samba.org/show_bug.cgi?id=6719
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Hans-Christian Jehg]

Hello Carlos

Thank you for your answer.

no the exit code is 24...

I also sometimes get exit code 23, but then rsync does it correctly...

What would be the effect of the file system reporting "permission denied" while reading directories? A list with files missing ?

HC

Carlos Carvalho skrev:

Hans-Christian Jehg (hc@...) wrote on 23 October 2009 17:32:
 >The circumstance where this happens is now known, it is when requests
 >to the shared in files returns "permission denied" from the file
 >server...
[snip]
 >I am not at all sure where it goes wrong, but in the end the result is
 >that rsync thinks certain files has disappeared. It could for instance
 >go wrong in the cifs client, but also in rsync when trying to read the
 >remote file (I am not using --ignore-errors) if no proper read checking
 >is in place (I do not think that this is the case but I would mention
 >it anyway).

 >My immediate problem is that rsync proceeds happily and deletes any
 >vanished files in the end (--delete-after).

 >rsync stops with exit code 24, but by then the damage is already done
 >(files deleted) .

Strange, I have code 23 when rsync gets permission denied. This is
important because vanished files are a normal occurrence and files
should be removed in this case. OTOH, code 23 is an error, and rsync
should not do deletions if you don't use --ignore-errors.

If the exit status is 23 you're likely hitting this bug:

Matt McCutchen (matt@...) wrote on 23 September 2009 22:19:
 >On Thu, 2009-09-17 at 20:59 +0200, Enno Middelberg wrote:
 >> Here the two directories which couldn't be read have been deleted from
 >> the target directory (ie, the files contained in them were deleted,
 >> the directories did still exist, if I remember correctly). If I
 >> understand the man page correctly, rsync should not delete files from
 >> the target directory when an error occurs. This can be overridden with
 >> the "--ignore-errors" command line switch, but this switch has
 >> definitely not been used.
 >
 >Maybe you are hitting this issue:
 >
 >https://bugzilla.samba.org/show_bug.cgi?id=6719
  

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Matt McCutchen-7]

On Wed, 2009-10-21 at 12:14 +0200, Hans-Christian Jehg wrote:

> I have a rather curious problem. I rsync from a fileserver shared in via
> cifs. This connection sometimes fails (don't worry, that is not the
> question I have).
> I am not at all sure where it goes wrong, but in the end the result is
> that rsync thinks certain files has disappeared. It could for instance
> go wrong in the cifs client, but also in rsync when trying to read the
> remote file (I am not using --ignore-errors) if no proper read checking
> is in place (I do not think that this is the case but I would mention it
> anyway).
> My immediate problem is that rsync proceeds happily and deletes any
> vanished files in the end (--delete-after).
> rsync stops with exit code 24, but by then the damage is already done
> (files deleted) .

On Fri, 2009-10-23 at 17:32 +0200, Hans-Christian Jehg wrote:
> The circumstance where this happens is now known, it is when requests
> to the shared in files returns "permission denied" from the file
> server...

Where are you seeing the "permission denied" text?  It would help if you
would post the exact output of rsync pertaining to the affected files.

When rsync's "stat" call on a source file fails, the behavior depends on
the returned error code.  If the error is ENOENT ("No such file or
directory"), rsync reports the file as vanished and sets exit code 24.
But if the error is anything else, including EACCES ("Permission
denied"), rsync sets the IOERR_GENERAL flag (which disables deletion
unless --ignore-errors is passed) and sets exit code 23.  So if rsync is
truly printing "Permission denied", it makes no sense that it would then
exit with code 24.

If your source filesystem is returning ENOENT for what is really a
permission denied error, that's the filesystem's problem, not rsync's.
Disabling the deletion of vanished files would be a misfeature, as
Carlos suggested.

On Sun, 2009-10-25 at 08:52 +0100, Hans-Christian Jehg wrote:
> no the exit code is 24...
>
> I also sometimes get exit code 23, but then rsync does it correctly...
>
> What would be the effect of the file system reporting "permission
> denied" while reading directories? A list with files missing ?

Precisely; that's why rsync disables deletion in such cases.

--
Matt

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html