RE: Rsync link-dest not working for a host with the most recent rsync/cygwin 1.7

[Steven Monai-4]

Chris Francy wrote:

> It appears something about the about rsync or the 1.7 version of
> cygwin is preventing the --link-destination function of rsync from
> working.  Files that have not been changed at all should be linked
> together.  On the hosts still running a cygwin 1.5 version of rsync
> the link-destination seems to work exactly as I expect it to.
>
> I am using dirvish (http://www.dirvish.org/) on a Linux box (deb
> 5.0.3) to backup some of the data on windows boxes in the network.  I
> have the cygwin+rsync setup in daemon mode on the servers.  I recently
> upgraded a couple of the boxes to the latest cygwin 1.7.  On the
> systems with cygwin 1.5 files are occasionally missed because of the
> path length limitations so I really want to get a cygwin 1.7 setup
> working.

I'm in a situation somewhat similar to yours. Up to now I have been
using the current stable Cygwin (v1.5) to test rsync as a possible
backup solution. I have a home-baked rsync/--link-dest script that
creates periodic snapshots of the Windows boxes on my LAN, and it
hiccups noisily on long filenames, and on filenames with non-ASCII chars
in them. I am considering trying out the next release of Cygwin (v1.7)
for its supposed fixes to the v1.5 path-length and filename encoding
limitations, but it would really suck if the --link-dest magic stopped
working in v1.7.

> This is the command that is being issued by dirvish to perform the backup.
>
> rsync -vrltH --delete -pgo --stats -z -D --numeric-ids --timeout=300
> --exclude-from=/srv/dirvish/srv-01/2009101610/exclude
> --link-dest=/srv/dirvish/srv-01/2009101420/tree 10.2.35.241::data/
> /srv/dirvish/srv-01/2009101610/tree

I'm not at all familiar with dirvish, but I've been using rsync quite a
lot. I don't see anything out of the ordinary on the command line. It
appears you are (or rather dirvish is) keeping successive backups on
your Debian box under
/srv/dirvish/srv-01/<date-time-derived-dir-name>/tree/.

Probably a dumb question, but: Are you 100% certain that
2009101420/tree/ and 2009101610/tree/ are on the same filesystem?

> Here is the checksums and the output of stat of two files on my Linux
> box that I should have been linked together.  No changes where made to
> this file on the windows box between the first and second backups.

[Snip the output of 'sha256sum' and 'stat' proving that the following
files on the Debian box are essentially identical:
`2009101420/tree/staff/Supply letter preschool.doc'
`2009101610/tree/staff/Supply letter preschool.doc']

Okay, so the backed-up files appear to be identical in successive
backups on the Debian (destination) side, including many attributes that
one imagines rsync would check for differences (contents, name, size,
type, access perms, ownerships, mod time). If --link-dest were working
correctly, I would expect to see Links > 1 and the two files' Inode
numbers to be the same, but neither is true (Links == 1, and the Inode
numbers are different). Definitely proof that --link-dest is failing to
work as intended. Clearly, rsync on your Debian box thinks that the
source file 'staff/Supply letter preschool.doc' is somehow different
from the existing file in the --link-dest directory.

Just curious: What do 'sha256sum' and 'stat' say about that file on the
Cygwin (source) side?

> I have posted the output of 'cygcheck -s -v -r' here
> (http://pastebin.com/m68d01c02).

Could you post the contents of your rsync daemon's rsyncd.conf ? Are you
using identical rsyncd.conf files in your 1.5 and 1.7 installations?

> I would appreciate some help getting the --list-dest to work properly.
>  If you have some ideas please share.  If you need more information to
> help diagnose or find a solution please ask.

Is --link-dest failing completely (i.e. no hardlinks created at all), or
only partially (i.e. some unchanged files are hardlinked, while some are
merely copied)? What version of rsync is running on the Debian box?

> Thanks,
> Chris Francy

Regards,
-SM
--


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Chris Francy-3]

On Tue, Oct 20, 2009 at 10:12 PM, Steven Monai <smonaica@...> wrote:
> Chris Francy wrote:
>
> I'm in a situation somewhat similar to yours. Up to now I have been
> using the current stable Cygwin (v1.5) to test rsync as a possible
> backup solution. I have a home-baked rsync/--link-dest script that
> ...
> limitations, but it would really suck if the --link-dest magic stopped
> working in v1.7.

I have learned that if I don't copy the permissions/ownership
information then the link-dest works fine.  I only used those options
in the past because older versions of rsync would not do link-dest
unless you copied ownership/permissions.

> Probably a dumb question, but: Are you 100% certain that
> 2009101420/tree/ and 2009101610/tree/ are on the same filesystem?

Yup, 100% certain.

> Could you post the contents of your rsync daemon's rsyncd.conf ? Are you
> using identical rsyncd.conf files in your 1.5 and 1.7 installations?

Yup, i am sure they are identical, and the config is very simple.

--- rsyncd.conf
use chroot = false
strict modes = false
hosts allow = 10.x.x.x
log file = rsyncd.log
pid file = rsyncd.pid

[data]
path = /cygdrive/d
read only = true
transfer logging = no
---

> Is --link-dest failing completely (i.e. no hardlinks created at all), or
> only partially (i.e. some unchanged files are hardlinked, while some are
> merely copied)? What version of rsync is running on the Debian box?

Absolutely nothing was getting linked together.

root@linuxhost:$ rsync --version
rsync  version 3.0.3  protocol version 30
Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes

Thanks,
Chris

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Chris Francy-3]

On Thu, 22 Oct 2009 20:08 PM, Steven Monai <smonaica@...> wrote:

>> Since I don't care about the retaining the
>> permissions, and I now have a version of rsync on both sides that
>> allows me to use link-dest without copying permissions I am happy.  If
>> someone wants to continue digging into this further I am still
>> available.
>
> Thanks in advance for indulging me just a little bit further.
>
> Could you re-run your Cygwin-1.7 test case with --itemize-changes added
> to the rsync command-line? That should provide a hint at what Debian's
> rsync thinks is different between its existing file copy and Cygwin's
> source file. Also, I would like to see what Cygwin's 'stat' command says
> about the source file on the Windows box.

The output of stat of my testfile on the box with cygwin 1.7 is

stat testfile
  File: `testfile'
  Size: 1048576         Blocks: 1024       IO Block: 65536  regular file
Device: bcdb6416h/3168494614d   Inode: 27303072741079683  Links: 1
Access: (0700/-rwx------)  Uid: (4294967295/ UNKNOWN)   Gid: (  401/mkpasswd)
Access: 2009-10-20 23:37:29.577294700 -0700
Modify: 2009-10-21 00:36:28.000000000 -0700
Change: 2009-10-20 23:37:30.030425500 -0700

Here (http://pastebin.com/m6c414439) is the output of my rerunning my
tests against the server with cygwin 1.7 with the  --itemize-changes
and -vv rsync options.

Chris

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Steven Monai-4]

Chris Francy wrote:

> On Thu, 22 Oct 2009 20:08 PM, Steven Monai <smonaica@...> wrote:
>> Could you re-run your Cygwin-1.7 test case with --itemize-changes added
>> to the rsync command-line? That should provide a hint at what Debian's
>> rsync thinks is different between its existing file copy and Cygwin's
>> source file. Also, I would like to see what Cygwin's 'stat' command says
>> about the source file on the Windows box.
>
> The output of stat of my testfile on the box with cygwin 1.7 is
>
> stat testfile
>   File: `testfile'
>   Size: 1048576         Blocks: 1024       IO Block: 65536  regular file
> Device: bcdb6416h/3168494614d   Inode: 27303072741079683  Links: 1
> Access: (0700/-rwx------)  Uid: (4294967295/ UNKNOWN)   Gid: (  401/mkpasswd)
> Access: 2009-10-20 23:37:29.577294700 -0700
> Modify: 2009-10-21 00:36:28.000000000 -0700
> Change: 2009-10-20 23:37:30.030425500 -0700
>
> Here (http://pastebin.com/m6c414439) is the output of my rerunning my
> tests against the server with cygwin 1.7 with the  --itemize-changes
> and -vv rsync options.

Notice that the Uid/Gid of 'testfile' on Cygwin 1.7 is a bit crazy. It
is likely that '/etc/passwd' and/or '/etc/group' have not been set up
correctly on that box. On the Debian side, it appears that rsync is
unable to preserve ownership when presented with that strange Uid,
instead saving the file with Uid/Gid = 0/0. (Presumably, rsync is
running as 'root' on the Debian box.) On subsequent runs, Debian's rsync
determines that the source file and its local copy are different,
because the source Uid (4294967295) is different from the local Uid (0).
Thus, when asked to preserve ownerships, rsync with the --link-dest
option effectively never creates any hard links.

That explains why omitting --owner and --group from the rsync command
line is a passable work-around in your case. Arguably, a better solution
might be to get sane Uid/Gids associated with files on the Cygwin box,
but only you can decide whether the effort would be worth it. (It may be
as simple as running 'mkpasswd' and 'mkgroup' with appropriate options.)

-SM
--


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple