Take2: rsync over ssh with --link-dest

[Alex-325]

Ugh, sorry, fumbled the keys. Here is the proper message.

Hi,

I'm trying to backup a directory over ssh that is named according to
the date, such as "20091104" on the remote side, and compare it to
yesterday's backup on the local side, such as "20091103". I'm pretty
sure I understand how rsync include/exclude works, and I believe that
is the root of the problem.

In my ~/.ssh/authorized_keys file I have the key, with the leading command:

command="/usr/bin/rsync --server --sender -vnlogDtprze.is .
/backup/servers/serverA/cdrive/"

Running rsync on the remote side with this key prints the directories
under cdrive/, which are the previous backups, like 20091102,
20091103, and 20091104. I can solve this problem by making the key
contain the full path to the specific date directory, but that changes
every day, so it's not very practical.

The rsync command I'd like to be able to use is this:

rsync -avze "ssh -i keyname"
  --link-dest=/backup/host/server/cdrive/20091103 --include=20091104/ \
   root@remotehost:/ /backup/host/server/cdrive/

However, it sends all the directories in the /cdrive/ tree, not just
the one I'm including.  Unless I specify the exact directory I want in
the authorized_keys file, the compare then fails. Is this even
possible?

Would it make any difference if I did it using an rsync server instead
of over ssh?

Thanks,
Alex
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Matt McCutchen-7]

On Wed, 2009-11-04 at 20:13 -0500, Alex wrote:
> In my ~/.ssh/authorized_keys file I have the key, with the leading command:
>
> command="/usr/bin/rsync --server --sender -vnlogDtprze.is .
> /backup/servers/serverA/cdrive/"

Overriding the server command line is generally a bad idea for the
reasons you went on to note.  Instead, use an rsync daemon.  You can
even have the daemon invoked over ssh with a forced command, see:

https://bugzilla.samba.org/show_bug.cgi?id=4163#c4

--
Matt

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Alex-325]

Hi,

> Overriding the server command line is generally a bad idea for the
> reasons you went on to note.  Instead, use an rsync daemon.  You can
> even have the daemon invoked over ssh with a forced command, see:
>
> https://bugzilla.samba.org/show_bug.cgi?id=4163#c4

That looks like the way to go. I've got it set up and running for a
few days now, and it's working okay. I'm a bit disappointed that I
can't do it by overriding the server command line, but it's okay.

Thanks again,
Alex
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Matt McCutchen-7]

On Sun, 2009-11-08 at 21:01 -0500, Alex wrote:
> > Overriding the server command line is generally a bad idea for the
> > reasons you went on to note.  Instead, use an rsync daemon.  You can
> > even have the daemon invoked over ssh with a forced command, see:
> >
> > https://bugzilla.samba.org/show_bug.cgi?id=4163#c4
>
> That looks like the way to go. I've got it set up and running for a
> few days now, and it's working okay.

Good.

> I'm a bit disappointed that I
> can't do it by overriding the server command line

Why?  Does it have some advantage over the use of an rsync daemon (I
can't imagine what)?  One can't really expect overriding the server
arguments not to lead to the inconsistent behavior you saw.

--
Matt

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Alex-325]

>> I'm a bit disappointed that I
>> can't do it by overriding the server command line
>
> Why?  Does it have some advantage over the use of an rsync daemon (I
> can't imagine what)?  One can't really expect overriding the server
> arguments not to lead to the inconsistent behavior you saw.

Simplicity. I'd like to be able to have the server side feed me all
data beginning with the path I specify, such /backup/serverA, and have
the client side be able to "--include=/usr/bin" to be able to
'download' only the data from /backup/serverA/usr/bin and exclude
everything else.

Instead, I have about eight sections of my rsyncd.conf to do this; one
for each directory tree that I want to back up to a different
location, primarily based on dates, to have multiple incrementals.

Thanks,
Alex
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Matt McCutchen-7]

On Mon, 2009-11-09 at 00:28 -0500, Alex wrote:
> >> I'm a bit disappointed that I
> >> can't do it by overriding the server command line
> >
> > Why?  Does it have some advantage over the use of an rsync daemon (I
> > can't imagine what)?  One can't really expect overriding the server
> > arguments not to lead to the inconsistent behavior you saw.
>
> Simplicity. I'd like to be able to have the server side feed me all
> data beginning with the path I specify, such /backup/serverA,

In other words, you were specifying a bogus source path and counting on
sshd to override it with the real one.  Clever.  I didn't catch that the
first time.

Realize that the ability to set a module's "path" in the rsyncd.conf
file and have the client refer to the module by a well-known name gives
you the same functionality, and actually more, because the client can
specify a subdirectory path within the module.

> and have
> the client side be able to "--include=/usr/bin" to be able to
> 'download' only the data from /backup/serverA/usr/bin and exclude
> everything else.

That --include means "exclude everything else" is a common
misconception; see the man page.  But I imagine the issue will become
moot if you have the client specify a subdirectory of the module instead
of using filters.

> Instead, I have about eight sections of my rsyncd.conf to do this; one
> for each directory tree that I want to back up to a different
> location, primarily based on dates, to have multiple incrementals.

If the daemon solution is looking fundamentally more complicated than
the set of forced server commands you previously had, you're probably
going about it the wrong way and I could help if you post the
rsyncd.conf file.

--
Matt

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Alex-325]

Hi,

>> Simplicity. I'd like to be able to have the server side feed me all
>> data beginning with the path I specify, such /backup/serverA,
>
> In other words, you were specifying a bogus source path and counting on
> sshd to override it with the real one.  Clever.  I didn't catch that the
> first time.

Yes, not sure if I mentioned previously, but I had:

command="/usr/bin/rsync ... -server ... /path"

followed by the actual ssh key.

> Realize that the ability to set a module's "path" in the rsyncd.conf
> file and have the client refer to the module by a well-known name gives
> you the same functionality, and actually more, because the client can
> specify a subdirectory path within the module.

I'm not sure I understand fully, but this may be what I'm already
doing. An example would be helpful.

>> and have
>> the client side be able to "--include=/usr/bin" to be able to
>> 'download' only the data from /backup/serverA/usr/bin and exclude
>> everything else.
>
> That --include means "exclude everything else" is a common
> misconception; see the man page.  But I imagine the issue will become

Yes, understood; it is what I would _like_ to happen.

> If the daemon solution is looking fundamentally more complicated than
> the set of forced server commands you previously had, you're probably
> going about it the wrong way and I could help if you post the
> rsyncd.conf file.

Okay, great, it's included below.

To make it even more complicated, on the receiving (client) side, I
have to pass the backup script I use through sed to rewrite the date
which gets appended to the source path, to update every day so I can
use the --link-dest rsync argument to create a backup of only the
changes from day-to-day, and hard link the rest from the reference
location, which is usually the previous day.

use chroot = no
syslog facility = local5

[cdrive]
   list = true
   path = /backup/servers/xxx.xxx.228/cdrive
   comment = 228-cdrive
   read only = true
   gid = root
   uid = root

[ddrive]
   list = true
   path = /backup/servers/xxx.xxx.228/ddrive
   comment = 228-ddrive
   read only = true
   gid = root
   uid = root

[edrive]
   list = true
   path = /backup/servers/xxx.xxx.228/edrive
   comment = 228-edrive
   read only = true
   gid = root
   uid = root

[229]
   list = true
   path = /backup/servers/xxx.xxx.229/
   comment = server229
   read only = true
   gid = root
   uid = root

[249]
   list = true
   path = /backup/servers/xxx.xxx.249/
   comment = server249
   read only = true
   gid = root
   uid = root

The client side command looks something like this:

rsync -avz --delete --numeric-ids --exclude-from=/backup/exclude-file.txt \
     --link-dest=/backup/seg/serverA/xxx.xxx.228/cdrive/20091109 \
     rsync://serverA/cdrive/20091110/ \
     /backup/seg/serverA/xxx.xxx.228/cdrive/20091110

Thanks,
Alex
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Matt McCutchen-7]

On Tue, 2009-11-10 at 22:47 -0500, Alex wrote:
> > If the daemon solution is looking fundamentally more complicated than
> > the set of forced server commands you previously had, you're probably
> > going about it the wrong way and I could help if you post the
> > rsyncd.conf file.
>
> Okay, great, it's included below.

> use chroot = no
> syslog facility = local5
>
> [cdrive]
>    list = true
>    path = /backup/servers/xxx.xxx.228/cdrive
>    comment = 228-cdrive
>    read only = true
>    gid = root
>    uid = root

[more modules like this...]

> The client side command looks something like this:
>
> rsync -avz --delete --numeric-ids --exclude-from=/backup/exclude-file.txt \
>      --link-dest=/backup/seg/serverA/xxx.xxx.228/cdrive/20091109 \
>      rsync://serverA/cdrive/20091110/ \
>      /backup/seg/serverA/xxx.xxx.228/cdrive/20091110

Your daemon setup looks exactly right.  I don't understand why you
consider it more complicated than the previous setup.  Previously you
had one SSH authorized key for each backup point with the appropriate
path forced; now the daemon configuration contains the same information.
The daemon configuration looks more verbose, but that could be helped by
moving the parameters that are the same for all modules ("list", ...) to
the global section at the top, or you could even generate the
configuration file with M4.  The client command is the same except for
the specification of a module name instead of an SSH key.  Am I missing
something?

> > Realize that the ability to set a module's "path" in the rsyncd.conf
> > file and have the client refer to the module by a well-known name gives
> > you the same functionality, and actually more, because the client can
> > specify a subdirectory path within the module.
>
> I'm not sure I understand fully, but this may be what I'm already
> doing. An example would be helpful.

You are indeed doing this.  Your source is:

rsync://serverA/cdrive/20091110/

The rsync://serverA/cdrive/ is mapped to
/backup/servers/xxx.xxx.228/cdrive according to the configuration file,
resulting in:

/backup/servers/xxx.xxx.228/cdrive/20091110/

In contrast, the forced-server-command approach would clobber the
20091110/ part, preventing you from setting up the --link-dest properly.
Is it clear now?

> To make it even more complicated, on the receiving (client) side, I
> have to pass the backup script I use through sed to rewrite the date
> which gets appended to the source path, to update every day so I can
> use the --link-dest rsync argument to create a backup of only the
> changes from day-to-day, and hard link the rest from the reference
> location, which is usually the previous day.

It seems to me that you would have had to do the same thing under the
forced-server-command setup.  No?

In any case, sed is overkill to vary the date.  You can just use a shell
variable:

DATE=$(date +%Y%m%d)
rsync ... /backup/seg/serverA/xxx.xxx.228/cdrive/$DATE

--
Matt

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Alex-325]

Hi,

>> rsync -avz --delete --numeric-ids --exclude-from=/backup/exclude-file.txt \
>>      --link-dest=/backup/seg/serverA/xxx.xxx.228/cdrive/20091109 \
>>      rsync://serverA/cdrive/20091110/ \
>>      /backup/seg/serverA/xxx.xxx.228/cdrive/20091110
>
> Your daemon setup looks exactly right.  I don't understand why you
> consider it more complicated than the previous setup.  Previously you
> had one SSH authorized key for each backup point with the appropriate

Well, no, I didn't have that because it doesn't work. I've been using
empty passphrase keys with command= for many years, but never really
had a requirement to override the path in this way.

> In contrast, the forced-server-command approach would clobber the
> 20091110/ part, preventing you from setting up the --link-dest properly.
> Is it clear now?

Yes, and hopefully I've clarified what I'm doing now too. :-)

> It seems to me that you would have had to do the same thing under the
> forced-server-command setup.  No?

Yep, exactly :-)

> In any case, sed is overkill to vary the date.  You can just use a shell
> variable:
>
> DATE=$(date +%Y%m%d)
> rsync ... /backup/seg/serverA/xxx.xxx.228/cdrive/$DATE

It's part of a larger backup script with its own configuration file, actually.

Thanks so much.
Best regards,
Alex
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html