taking advantage of filles in folder other than destination folder

[rssrik]

Dear Folks,

I have to backup pgsql database dump everynight on a routine. The database dump actually contains sql(text) statements. The actual size of the database dump is aroung 800 MB. Between two days backup, only few lines of statements are modified/added/deleted.

I dont want to do incremental backup. Instead, I want everyday's full dump to be stored on to a folder named DD-MM-YY(ex: 01-05-08). Everyday there will be a folder created with name DD-MM-YY and the dump will be stored there.

Since the difference between the two day's backup are pretty minimal, I dont want to resend the 800 MB files everyday.

Suppose,

1. I have taken yesterdays dump in a folder named 04-05-08 in database server
2. I have rsynced the folder 04-05-08 to the backup server.
3. I have taken today's dump in folder named 05-05-08 in database server.
4. Now while doing rsync, I want rsync to compare the <database server's 05-05-08/dump file>  with the <04-05-08/dump file> which already exists in backup server. (then eventually,  I want rsync to send only the differences and store the output in folder 05-05-08).

Typicaly I want to take advantage of yesterday's file which already present in a different folder.

Can anyone suggest a working method to implement this ? (I tried --copy-dest / --compare-dest .. but couldnt get it to work)

Any response in this regard is most appreciated.

Regards,
rssrik

[Paul Slootman-5]

On Mon 05 May 2008, rssrik wrote:

> Suppose,
>
> 1. I have taken yesterdays dump in a folder named 04-05-08 in database
> server
> 2. I have rsynced the folder 04-05-08 to the backup server.
> 3. I have taken today's dump in folder named 05-05-08 in database server.
> 4. Now while doing rsync, I want rsync to compare the <database server's
> 05-05-08/dump file>  with the <04-05-08/dump file> which already exists in
> backup server. (then eventually,  I want rsync to send only the differences
> and store the output in folder 05-05-08).
>
> Typicaly I want to take advantage of yesterday's file which already present
> in a different folder.
>
> Can anyone suggest a working method to implement this ? (I tried --copy-dest
> / --compare-dest .. but couldnt get it to work)

Something like this should work...

BASE=/backup/psql # edit for your situation
REMOTESOURCE=dbhost::psqldumps/ # again, edit
TODAY=`date +%d-%m-%y`  # I prefer %Y-%m-%d, sorts better
cd $BASE
YESTERDAYS_BACKUP=`ls -td ??-??-?? | head -n 1`
rsync -a --compare-dest=../$YESTERDAYS_BACKUP $REMOTESOURCE $TODAY/


Paul Slootman
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[rssrik]

Thanks paul,

but the requirement is to upload to the backup server (from database server),

"rsync -a --compare-dest=../$YESTERDAYS_BACKUP  $TODAY $REMOTESOURCE  #change the src and dest"

this command does not seem to work for me !! I mean that the whole file is getting transferred !

Any thoughts on this ?

Thanks again,
rssrik

Paul Slootman-5 wrote:

On Mon 05 May 2008, rssrik wrote:

> Suppose,
>
> 1. I have taken yesterdays dump in a folder named 04-05-08 in database
> server
> 2. I have rsynced the folder 04-05-08 to the backup server.
> 3. I have taken today's dump in folder named 05-05-08 in database server.
> 4. Now while doing rsync, I want rsync to compare the <database server's
> 05-05-08/dump file>  with the <04-05-08/dump file> which already exists in
> backup server. (then eventually,  I want rsync to send only the differences
> and store the output in folder 05-05-08).
>
> Typicaly I want to take advantage of yesterday's file which already present
> in a different folder.
>
> Can anyone suggest a working method to implement this ? (I tried --copy-dest
> / --compare-dest .. but couldnt get it to work)

Something like this should work...

BASE=/backup/psql # edit for your situation
REMOTESOURCE=dbhost::psqldumps/ # again, edit
TODAY=`date +%d-%m-%y`  # I prefer %Y-%m-%d, sorts better
cd $BASE
YESTERDAYS_BACKUP=`ls -td ??-??-?? | head -n 1`
rsync -a --compare-dest=../$YESTERDAYS_BACKUP $REMOTESOURCE $TODAY/


Paul Slootman
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Paul Slootman-5]

On Mon 05 May 2008, rssrik wrote:

> Thanks paul,
>
> but the requirement is to upload to the backup server (from database
> server),
>
> "rsync -a --compare-dest=../$YESTERDAYS_BACKUP  $TODAY $REMOTESOURCE
> #change the src and dest"

As you don't show what you've put into those variables, notably
$REMOTESOURCE (which you should have renamed REMOTEDESTINATION),
the destination origin is probably one directory higher than you expect,
hence the relative compare-dest goes wrong.
I find it useful to explicitly end the destination with a slash (as I
already showed).

Also, how does "$YESTERDAYS_BACKUP" get determined if you have no access
to the destination?

You'll probably need to add a slash to $TODAY and the destination:

rsync -a --compare-dest=../$YESTERDAYS_BACKUP  $TODAY/ $REMOTEDESTINATION/
>
> this command does not seem to work for me !! I mean that the whole file is
> getting transferred !

How do you know? Show some stats.


Paul Slootman
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[rssrik]

Thanks again Paul,

I found a solution last night. The following command works,
rsync -az --compare-dest=../20080101/    20080102/   rsync://<REMOTEHOST>/<SHARENAME>/<20080102>/

On adding / to the source directory name, only the contents of that directory gets created in the target. The workaround is to directly specify the required Directory Name in the TARGET URL itself.

When using Directory Names without  /, compare-dest does not work.
# This transfers the whole file <compare dest doesnot work>
rsync -az --compare-dest=../20080101   20080102   rsync://<REMOTEHOST>/<SHARENAME>/

# This compares the file but the directory 20080102 is not created in TARGET
rsync -az --compare-dest=../20080101   20080102   rsync://<REMOTEHOST>/<SHARENAME>/
 
One more request, is there any document which explains the syntax of rsync and its params in detail. It could be more useful, If you link me to one <with man pages I couldnt get the syntax correctly>.

Paul ji, thanks again for your most valuable time,
rssrik

Paul Slootman-5 wrote:

On Mon 05 May 2008, rssrik wrote:

> Thanks paul,
>
> but the requirement is to upload to the backup server (from database
> server),
>
> "rsync -a --compare-dest=../$YESTERDAYS_BACKUP  $TODAY $REMOTESOURCE
> #change the src and dest"

As you don't show what you've put into those variables, notably
$REMOTESOURCE (which you should have renamed REMOTEDESTINATION),
the destination origin is probably one directory higher than you expect,
hence the relative compare-dest goes wrong.
I find it useful to explicitly end the destination with a slash (as I
already showed).

Also, how does "$YESTERDAYS_BACKUP" get determined if you have no access
to the destination?

You'll probably need to add a slash to $TODAY and the destination:

rsync -a --compare-dest=../$YESTERDAYS_BACKUP  $TODAY/ $REMOTEDESTINATION/
>
> this command does not seem to work for me !! I mean that the whole file is
> getting transferred !

How do you know? Show some stats.


Paul Slootman
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Paul Slootman-5]

On Mon 05 May 2008, rssrik wrote:
>
> I found a solution last night. The following command works,
> rsync -az --compare-dest=../20080101/    20080102/  
> rsync://<REMOTEHOST>/<SHARENAME>/<20080102>/
>
> On adding / to the source directory name, only the contents of that
> directory gets created in the target. The workaround is to directly specify
> the required Directory Name in the TARGET URL itself.

Nothing new, that's what I had in my original posting.
As I sais, as you didn't show what the values were of the variables you
used in your version, I could not say exactly what to do.

> One more request, is there any document which explains the syntax of rsync
> and its params in detail. It could be more useful, If you link me to one
> <with man pages I couldnt get the syntax correctly>.

It's all in the manpage...


Paul Slootman
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html