Command line multi-part copy program

[Parag P. Doke]

Hello Everyone.
Axel is a utility similar to wget that allows multi-part downloads. Here is
the homepage for it:
http://axel.alioth.debian.org/
Does anyone know of a similar free tool, to copy data ?

I tried the robocopy and xxcopy (free version) for copying a few files over
a Windows share...one of the files being around 300 MB in size...but found
the performance too low.

[As an alternative, I could request the remote system administrator to host
the files over http...but just wanted to see if a multi-part copy program
exists already.]

Thanks in advance for all your inputs,
--
Parag P. Doke
http://paragpdoke.blogspot.com


[Non-text portions of this message have been removed]

[foxidrive-7]

On Thu, 21 Aug 2008 15:08:37 +0530, "Parag P. Doke" <paragpdoke@...>
wrote:

>I tried the robocopy and xxcopy (free version) for copying a few files over
>a Windows share...one of the files being around 300 MB in size...but found
>the performance too low.

What performance did you get?  Is it copying over 100 Mbit LAN?  Are the
files tiny?  

With large files I'd expect 10 MB/sec but small files will be far slower
due to the file system overheads.

[Parag P. Doke]

Hi Mic.
Pardon my lack of knowledge about the networking part.
In the Windows taskbar, the status shows 1 GB/sec.
Except for a 340 MB file, rest of them are tiny (the 2nd largest being 2MB).

Robocopy did display a speed of around 10 MB/sec (when it was copying the
largest one).
I have sent a request to the Admin, assuming such a tool does not exist.

[Has anyone heard of such a tool in the Linux world ? I read about one named
"rsync"...but haven't tried the Windows version yet.]

Thanks,
Parag P. Doke

On Thu, Aug 21, 2008 at 5:18 PM, foxidrive <foxidrive@...> wrote:

>   On Thu, 21 Aug 2008 15:08:37 +0530, "Parag P. Doke" <
> paragpdoke@... <paragpdoke%40gmail.com>>
> wrote:
>
>
> >I tried the robocopy and xxcopy (free version) for copying a few files
> over
> >a Windows share...one of the files being around 300 MB in size...but found
> >the performance too low.
>
> What performance did you get? Is it copying over 100 Mbit LAN? Are the
> files tiny?
>
> With large files I'd expect 10 MB/sec but small files will be far slower
> due to the file system overheads.
>
>  
>

--
Parag P. Doke
http://paragpdoke.blogspot.com


[Non-text portions of this message have been removed]

[Andrew Palm-2]

Some more info is required I think.

Robocopy rocks as a file copy tool. so....

How much data are you trying to copy?

How many files?

Is it all local LAN or are you copying to a remote server over the WAN?

Are you trying to keep the two locations in sync or just doing a one off
copy?

Is the copy one way only? ie changes only happen at the source and you want
to copy to the dest server?

Andrew




On Thu, Aug 21, 2008 at 7:38 PM, Parag P. Doke <paragpdoke@...> wrote:

>   Hello Everyone.
> Axel is a utility similar to wget that allows multi-part downloads. Here is
> the homepage for it:
> http://axel.alioth.debian.org/
> Does anyone know of a similar free tool, to copy data ?
>
> I tried the robocopy and xxcopy (free version) for copying a few files over
> a Windows share...one of the files being around 300 MB in size...but found
> the performance too low.
>
> [As an alternative, I could request the remote system administrator to host
> the files over http...but just wanted to see if a multi-part copy program
> exists already.]
>
> Thanks in advance for all your inputs,
> --
> Parag P. Doke
> http://paragpdoke.blogspot.com
>
> [Non-text portions of this message have been removed]
>
>  
>

[Non-text portions of this message have been removed]

[Parag P. Doke]

Hello Andrew.
Here are the answers:
1) Total size of data is somewhere around 350 MB.
2) Total # of files: roughly 50 (contained in 3 folders). Size of one
file is to the tune of 330 MB. Rest total up to around 20 MB.
3) Pardon my lack of knowledge...but I'm copying over
\\MachineName\SharedFolder. The system "MachineName" is located in the
US while I'm located in India (Pune). I'm not sure whether to call it
LAN or WAN.
4) Yes, I'm trying to keep 2 locations in sync. The copy process is
triggered each morning.
5) Data is copied only one way.

I have already requested the remote system administrator to host the
folders over http and he agreed to it. Currently I'm using axel
(homepage: http://axel.alioth.debian.org/) to download the largest
file (the 330 MB one) and xcopy for the rest of them. The reason why I
had to to this is that I need not change the list of files to download
if they add / remove some of the 50 files to be copied.

If the xcopy finishes before axel finishes downloading, I make my
batch file wait till tasklist does not list axel. My current process
gets me all the data in around 5 or 6 minutes.

In short, my script is all setup now...but I'm still open to explore
great tools / utilities :-).

Thanks,
Parag P. Doke


On Sun, Aug 31, 2008 at 6:13 AM, Andrew Palm <ajpalm@...> wrote:

> Some more info is required I think.
>
> Robocopy rocks as a file copy tool. so....
>
> How much data are you trying to copy?
>
> How many files?
>
> Is it all local LAN or are you copying to a remote server over the WAN?
>
> Are you trying to keep the two locations in sync or just doing a one off
> copy?
>
> Is the copy one way only? ie changes only happen at the source and you want
> to copy to the dest server?
>
> Andrew
>
> On Thu, Aug 21, 2008 at 7:38 PM, Parag P. Doke <paragpdoke@...> wrote:
>
>> Hello Everyone.
>> Axel is a utility similar to wget that allows multi-part downloads. Here
>> is
>> the homepage for it:
>> http://axel.alioth.debian.org/
>> Does anyone know of a similar free tool, to copy data ?
>>
>> I tried the robocopy and xxcopy (free version) for copying a few files
>> over
>> a Windows share...one of the files being around 300 MB in size...but found
>> the performance too low.
>>
>> [As an alternative, I could request the remote system administrator to
>> host
>> the files over http...but just wanted to see if a multi-part copy program
>> exists already.]
>>
>> Thanks in advance for all your inputs,
>> --
>> Parag P. Doke
>> http://paragpdoke.blogspot.com
>>
>> [Non-text portions of this message have been removed]
>>
>>
>>
>
> [Non-text portions of this message have been removed]
>
>

--
Parag P. Doke
http://paragpdoke.blogspot.com

[Andrew Palm-2]

Sorry for the late reply.

Robocopy will do this without blinking a eyelid. You must not have the right
command line options

I use the following

robocopy.exe "\\sourceserver\source\path\name\to\sync"
"\\destserver\dest\path\name" *.* /E /Z /MIR /R:5 /W:5 /TEE

This does a MIRROR of the SOURCE TO THE DEST. - What does that mean?

It will make the dest look like the source... any changes to the source is
copied to the dest. Any extra files in the dest not in the source are
deleted.

After the first copy, it will mirror and only do changes. For 50 files, this
should take very long at all.

We use a robocopy script to keep a DR site 'in sync' for the file shares.
Its doing about 5TB of data, and millions of files. Only takes a short time.

HINT - Add a /L to the command line options for testing. This does not
delete, or copy anything. It just lists what it would have done. ie copied
xx, deleted xx etc.

Andrew





On Mon, Sep 1, 2008 at 5:23 PM, Parag P. Doke <paragpdoke@...> wrote:

>   Hello Andrew.
> Here are the answers:
> 1) Total size of data is somewhere around 350 MB.
> 2) Total # of files: roughly 50 (contained in 3 folders). Size of one
> file is to the tune of 330 MB. Rest total up to around 20 MB.
> 3) Pardon my lack of knowledge...but I'm copying over
> \\MachineName\SharedFolder. The system "MachineName" is located in the
> US while I'm located in India (Pune). I'm not sure whether to call it
> LAN or WAN.
> 4) Yes, I'm trying to keep 2 locations in sync. The copy process is
> triggered each morning.
> 5) Data is copied only one way.
>
> I have already requested the remote system administrator to host the
> folders over http and he agreed to it. Currently I'm using axel
> (homepage: http://axel.alioth.debian.org/) to download the largest
> file (the 330 MB one) and xcopy for the rest of them. The reason why I
> had to to this is that I need not change the list of files to download
> if they add / remove some of the 50 files to be copied.
>
> If the xcopy finishes before axel finishes downloading, I make my
> batch file wait till tasklist does not list axel. My current process
> gets me all the data in around 5 or 6 minutes.
>
> In short, my script is all setup now...but I'm still open to explore
> great tools / utilities :-).
>
> Thanks,
> Parag P. Doke
>
>
> On Sun, Aug 31, 2008 at 6:13 AM, Andrew Palm <ajpalm@...<ajpalm%40gmail.com>>
> wrote:
> > Some more info is required I think.
> >
> > Robocopy rocks as a file copy tool. so....
> >
> > How much data are you trying to copy?
> >
> > How many files?
> >
> > Is it all local LAN or are you copying to a remote server over the WAN?
> >
> > Are you trying to keep the two locations in sync or just doing a one off
> > copy?
> >
> > Is the copy one way only? ie changes only happen at the source and you
> want
> > to copy to the dest server?
> >
> > Andrew
> >
> > On Thu, Aug 21, 2008 at 7:38 PM, Parag P. Doke <paragpdoke@...<paragpdoke%40gmail.com>>
> wrote:
> >
> >> Hello Everyone.
> >> Axel is a utility similar to wget that allows multi-part downloads. Here
> >> is
> >> the homepage for it:
> >> http://axel.alioth.debian.org/
> >> Does anyone know of a similar free tool, to copy data ?
> >>
> >> I tried the robocopy and xxcopy (free version) for copying a few files
> >> over
> >> a Windows share...one of the files being around 300 MB in size...but
> found
> >> the performance too low.
> >>
> >> [As an alternative, I could request the remote system administrator to
> >> host
> >> the files over http...but just wanted to see if a multi-part copy
> program
> >> exists already.]
> >>
> >> Thanks in advance for all your inputs,
> >> --
> >> Parag P. Doke
> >> http://paragpdoke.blogspot.com
> >>
> >> [Non-text portions of this message have been removed]
> >>
> >>
> >>
> >
> > [Non-text portions of this message have been removed]
> >
> >
>
> --
> Parag P. Doke
> http://paragpdoke.blogspot.com
>  
>

[Non-text portions of this message have been removed]

[bslasch]

Sadly even the mighty robocopy is a short sighted tool.

Most of the common commandline copy utilities are single threaded
applications in nature.

this slows things down significantly during copy operations.

Now if there was a scriptable copy utility with an interface much like
robocopy that utilized multi-threaded programming that aggressively
did read ahead in the source directories and had multiple data mover
threads and verification threads I would be a very happy individual.

Now other things to consider...

What was the file size and data type profiles of the source data?
were you moving the files over a network?
were you moving the data to new directories on the same disk or was it
going to another disk?
If the source data is going to and from the same disk, head seek time
and rpm of the disk is an issue.

if over the network, is any equipment having issues?




--- In batchworld@..., "Andrew Palm" <ajpalm@...> wrote:
>
> Sorry for the late reply.
>
> Robocopy will do this without blinking a eyelid. You must not have
the right

> command line options
>
> I use the following
>
> robocopy.exe "\\sourceserver\source\path\name\to\sync"
> "\\destserver\dest\path\name" *.* /E /Z /MIR /R:5 /W:5 /TEE
>
> This does a MIRROR of the SOURCE TO THE DEST. - What does that mean?
>
> It will make the dest look like the source... any changes to the

source is
> copied to the dest. Any extra files in the dest not in the source are
> deleted.
>
> After the first copy, it will mirror and only do changes. For 50
files, this
> should take very long at all.
>
> We use a robocopy script to keep a DR site 'in sync' for the file
shares.
> Its doing about 5TB of data, and millions of files. Only takes a
short time.
>
> HINT - Add a /L to the command line options for testing. This does not
> delete, or copy anything. It just lists what it would have done. ie
copied

> xx, deleted xx etc.
>
> Andrew
>
>
>
>
>
> On Mon, Sep 1, 2008 at 5:23 PM, Parag P. Doke <paragpdoke@...> wrote:
>
> >   Hello Andrew.
> > Here are the answers:
> > 1) Total size of data is somewhere around 350 MB.
> > 2) Total # of files: roughly 50 (contained in 3 folders). Size of one
> > file is to the tune of 330 MB. Rest total up to around 20 MB.
> > 3) Pardon my lack of knowledge...but I'm copying over
> > \\MachineName\SharedFolder. The system "MachineName" is located in the
> > US while I'm located in India (Pune). I'm not sure whether to call it
> > LAN or WAN.
> > 4) Yes, I'm trying to keep 2 locations in sync. The copy process is
> > triggered each morning.
> > 5) Data is copied only one way.
> >
> > I have already requested the remote system administrator to host the
> > folders over http and he agreed to it. Currently I'm using axel
> > (homepage: http://axel.alioth.debian.org/) to download the largest
> > file (the 330 MB one) and xcopy for the rest of them. The reason why I
> > had to to this is that I need not change the list of files to download
> > if they add / remove some of the 50 files to be copied.
> >
> > If the xcopy finishes before axel finishes downloading, I make my
> > batch file wait till tasklist does not list axel. My current process
> > gets me all the data in around 5 or 6 minutes.
> >
> > In short, my script is all setup now...but I'm still open to explore
> > great tools / utilities :-).
> >
> > Thanks,
> > Parag P. Doke
> >
> >
> > On Sun, Aug 31, 2008 at 6:13 AM, Andrew Palm

<ajpalm@...<ajpalm%40gmail.com>>

> > wrote:
> > > Some more info is required I think.
> > >
> > > Robocopy rocks as a file copy tool. so....
> > >
> > > How much data are you trying to copy?
> > >
> > > How many files?
> > >
> > > Is it all local LAN or are you copying to a remote server over

the WAN?
> > >
> > > Are you trying to keep the two locations in sync or just doing a
one off
> > > copy?
> > >
> > > Is the copy one way only? ie changes only happen at the source
and you
> > want
> > > to copy to the dest server?
> > >
> > > Andrew
> > >
> > > On Thu, Aug 21, 2008 at 7:38 PM, Parag P. Doke
<paragpdoke@...<paragpdoke%40gmail.com>>
> > wrote:
> > >
> > >> Hello Everyone.
> > >> Axel is a utility similar to wget that allows multi-part
downloads. Here
> > >> is
> > >> the homepage for it:
> > >> http://axel.alioth.debian.org/
> > >> Does anyone know of a similar free tool, to copy data ?
> > >>
> > >> I tried the robocopy and xxcopy (free version) for copying a
few files
> > >> over
> > >> a Windows share...one of the files being around 300 MB in
size...but
> > found
> > >> the performance too low.
> > >>
> > >> [As an alternative, I could request the remote system
administrator to

> > >> host
> > >> the files over http...but just wanted to see if a multi-part copy
> > program
> > >> exists already.]
> > >>
> > >> Thanks in advance for all your inputs,
> > >> --
> > >> Parag P. Doke
> > >> http://paragpdoke.blogspot.com
> > >>
> > >> [Non-text portions of this message have been removed]
> > >>
> > >>
> > >>
> > >
> > > [Non-text portions of this message have been removed]
> > >
> > >
> >
> > --
> > Parag P. Doke
> > http://paragpdoke.blogspot.com
> >  
> >
>
>
> [Non-text portions of this message have been removed]
>

[Michael Burek]

@Brian Lasch:

I'm pretty much under the impression that copying is limited to the disk
speed, or bus bandwidth.  I really don't think the processor is involved
enough to make a difference.  Especially at Gigahertz speeds.  Using
firewire drives will take nearly all the processing off of the cpu and put
in onto the firewire interface card.

Also, a hard drive has a single head, so a hard drive itself is most
efficient in a single thread sequential read.  Having 2 threads read from 2
areas of the drive will cause seek time to enter the time equation and slow
things down.  Having a big file, all in a row, will produce the fastest read
result.  Even if drives had 2 separate 1 gigabyte buffers, those buffers
would have to take turns reading with only 1 drive head.  And even with a
solid state drive or a flash drive, those still work faster with a
sequential read than a random read.  But their random read is much faster
than a platter hard drive random read.

PS, if you read the question, you'll get your answers to what the question
is about.

-----
@Parag P. Doke

What is the bandwidth available?  If you max it out during the copying, then
no software can increase the speed.  The only hope is to zip the files on
the source computer, and transfer the zip files, when your bandwidth is the
bottleneck.  You'd have to zip each file separately and update the zip file
on the source computer when the original changes, so that you'd only copy
the updated files.  Zipping into 1 large file would always bring over every
file inside that zip file - causing excess copying for static files.

On the source computer, can an FTP server be run?  FTP servers will allow
for multipart copies.  But again, if you are maxing out your bandwidth,
multipart copying will not push more bits through the pipes than simple
copying.  Although your Axel program already does this through the http
server on the source machine.

-----


[Non-text portions of this message have been removed]

[Parag P. Doke]

Hello Andrew.
Thanks for the explanation about how robocopy helps to create mirrors.
However, that is not really what I was looking for. In my case, the
remote shared folder gets a new build created (in a folder) and my
intent is to copy over only the newly created folder to my local
machine (regardless of what happens to folders / files created in
other locations inside the source).

When I ran the robocopy command as suggested by you (including /l), it
did list the contents it was about to copy over. Without the /l, it
started to copy data.....but one file at a time. Also there was no
indication if the file copy operation of the current file was split
into chunks and then data re-joined on the destination. In short, I
found axel downloading a lot faster than robocopy (over http).

Thanks again.

Hi Brian.
I believe I've answered most of the questions in an earlier mail on
this thread. Pasting it below:
-----
1) Total size of data is somewhere around 350 MB.
2) Total # of files: roughly 50 (contained in 3 folders). Size of one
file is to the tune of 330 MB. Rest total up to around 20 MB.
3) Pardon my lack of knowledge...but I'm copying over
\\MachineName\SharedFolder. The system "MachineName" is located in the
US while I'm located in India (Pune). I'm not sure whether to call it
LAN or WAN.
4) Yes, I'm trying to keep 2 locations in sync. The copy process is
triggered each morning.
[Update: With the explanation from Andrew, let me correct
myself....I'm not trying to keep 2 folders in sync....and not trying
to create a mirror / replica.]
5) Data is copied only one way.
-----
There is no known issue with the equipment.

Thank you.

Hi Michael.
The source of the data is a computer that I don't have administrative
rights on. So I was trying for something that needs least time/effort
for setup, for the source administrator.

Though it will be possible to come up with a script that generates zip
file(s) for the source data and run it as a scheduled task on the
source, and then my local script gets the zip files here and extracts
them is one possible solution, I went for the host data over http
option (similar to the ftp one that you suggested).

Thank you.

For the time being, I have all my scheduled tasks / scripts in place
and the copy takes something close to a few minutes....and I'm happy
with the performance / time observed.

Regards,
Parag P. Doke


On Sat, Sep 6, 2008 at 2:08 PM, Michael Burek <mike@...> wrote:

> @Brian Lasch:
>
> I'm pretty much under the impression that copying is limited to the disk
> speed, or bus bandwidth. I really don't think the processor is involved
> enough to make a difference. Especially at Gigahertz speeds. Using
> firewire drives will take nearly all the processing off of the cpu and put
> in onto the firewire interface card.
>
> Also, a hard drive has a single head, so a hard drive itself is most
> efficient in a single thread sequential read. Having 2 threads read from 2
> areas of the drive will cause seek time to enter the time equation and slow
> things down. Having a big file, all in a row, will produce the fastest read
> result. Even if drives had 2 separate 1 gigabyte buffers, those buffers
> would have to take turns reading with only 1 drive head. And even with a
> solid state drive or a flash drive, those still work faster with a
> sequential read than a random read. But their random read is much faster
> than a platter hard drive random read.
>
> PS, if you read the question, you'll get your answers to what the question
> is about.
>
> -----
> @Parag P. Doke
>
> What is the bandwidth available? If you max it out during the copying, then
> no software can increase the speed. The only hope is to zip the files on
> the source computer, and transfer the zip files, when your bandwidth is the
> bottleneck. You'd have to zip each file separately and update the zip file
> on the source computer when the original changes, so that you'd only copy
> the updated files. Zipping into 1 large file would always bring over every
> file inside that zip file - causing excess copying for static files.
>
> On the source computer, can an FTP server be run? FTP servers will allow
> for multipart copies. But again, if you are maxing out your bandwidth,
> multipart copying will not push more bits through the pipes than simple
> copying. Although your Axel program already does this through the http
> server on the source machine.
>
> -----
>
> [Non-text portions of this message have been removed]
>
>

--
Parag P. Doke
http://paragpdoke.blogspot.com

[Andrew Palm-2]

Hi Brian...... The answers to your questions can be had by reading the
message I replied to!!

While a multi-threaded copy might be great idea, its not going to improve
speed over a WAN link....

We run multiable threads of robocopy, each 'syncing' different areas (aka
shares) of data.

Andrew



On Sat, Sep 6, 2008 at 5:30 PM, Brian Lasch <bslasch@...> wrote:

>   Sadly even the mighty robocopy is a short sighted tool.
>
> Most of the common commandline copy utilities are single threaded
> applications in nature.
>
> this slows things down significantly during copy operations.
>
> Now if there was a scriptable copy utility with an interface much like
> robocopy that utilized multi-threaded programming that aggressively
> did read ahead in the source directories and had multiple data mover
> threads and verification threads I would be a very happy individual.
>
> Now other things to consider...
>
> What was the file size and data type profiles of the source data?
> were you moving the files over a network?
> were you moving the data to new directories on the same disk or was it
> going to another disk?
> If the source data is going to and from the same disk, head seek time
> and rpm of the disk is an issue.
>
> if over the network, is any equipment having issues?
>
>
> --- In batchworld@... <batchworld%40yahoogroups.com>, "Andrew
> Palm" <ajpalm@...> wrote:
> >
> > Sorry for the late reply.
> >
> > Robocopy will do this without blinking a eyelid. You must not have
> the right
> > command line options
> >
> > I use the following
> >
> > robocopy.exe "\\sourceserver\source\path\name\to\sync"
> > "\\destserver\dest\path\name" *.* /E /Z /MIR /R:5 /W:5 /TEE
> >
> > This does a MIRROR of the SOURCE TO THE DEST. - What does that mean?
> >
> > It will make the dest look like the source... any changes to the
> source is
> > copied to the dest. Any extra files in the dest not in the source are
> > deleted.
> >
> > After the first copy, it will mirror and only do changes. For 50
> files, this
> > should take very long at all.
> >
> > We use a robocopy script to keep a DR site 'in sync' for the file
> shares.
> > Its doing about 5TB of data, and millions of files. Only takes a
> short time.
> >
> > HINT - Add a /L to the command line options for testing. This does not
> > delete, or copy anything. It just lists what it would have done. ie
> copied
> > xx, deleted xx etc.
> >
> > Andrew
> >
> >
> >
> >
> >
> > On Mon, Sep 1, 2008 at 5:23 PM, Parag P. Doke <paragpdoke@...> wrote:
> >
> > > Hello Andrew.
> > > Here are the answers:
> > > 1) Total size of data is somewhere around 350 MB.
> > > 2) Total # of files: roughly 50 (contained in 3 folders). Size of one
> > > file is to the tune of 330 MB. Rest total up to around 20 MB.
> > > 3) Pardon my lack of knowledge...but I'm copying over
> > > \\MachineName\SharedFolder. The system "MachineName" is located in the
> > > US while I'm located in India (Pune). I'm not sure whether to call it
> > > LAN or WAN.
> > > 4) Yes, I'm trying to keep 2 locations in sync. The copy process is
> > > triggered each morning.
> > > 5) Data is copied only one way.
> > >
> > > I have already requested the remote system administrator to host the
> > > folders over http and he agreed to it. Currently I'm using axel
> > > (homepage: http://axel.alioth.debian.org/) to download the largest
> > > file (the 330 MB one) and xcopy for the rest of them. The reason why I
> > > had to to this is that I need not change the list of files to download
> > > if they add / remove some of the 50 files to be copied.
> > >
> > > If the xcopy finishes before axel finishes downloading, I make my
> > > batch file wait till tasklist does not list axel. My current process
> > > gets me all the data in around 5 or 6 minutes.
> > >
> > > In short, my script is all setup now...but I'm still open to explore
> > > great tools / utilities :-).
> > >
> > > Thanks,
> > > Parag P. Doke
> > >
> > >
> > > On Sun, Aug 31, 2008 at 6:13 AM, Andrew Palm
> <ajpalm@...<ajpalm%40gmail.com>>
> > > wrote:
> > > > Some more info is required I think.
> > > >
> > > > Robocopy rocks as a file copy tool. so....
> > > >
> > > > How much data are you trying to copy?
> > > >
> > > > How many files?
> > > >
> > > > Is it all local LAN or are you copying to a remote server over
> the WAN?
> > > >
> > > > Are you trying to keep the two locations in sync or just doing a
> one off
> > > > copy?
> > > >
> > > > Is the copy one way only? ie changes only happen at the source
> and you
> > > want
> > > > to copy to the dest server?
> > > >
> > > > Andrew
> > > >
> > > > On Thu, Aug 21, 2008 at 7:38 PM, Parag P. Doke
> <paragpdoke@...<paragpdoke%40gmail.com>>
>
> > > wrote:
> > > >
> > > >> Hello Everyone.
> > > >> Axel is a utility similar to wget that allows multi-part
> downloads. Here
> > > >> is
> > > >> the homepage for it:
> > > >> http://axel.alioth.debian.org/
> > > >> Does anyone know of a similar free tool, to copy data ?
> > > >>
> > > >> I tried the robocopy and xxcopy (free version) for copying a
> few files
> > > >> over
> > > >> a Windows share...one of the files being around 300 MB in
> size...but
> > > found
> > > >> the performance too low.
> > > >>
> > > >> [As an alternative, I could request the remote system
> administrator to
> > > >> host
> > > >> the files over http...but just wanted to see if a multi-part copy
> > > program
> > > >> exists already.]
> > > >>
> > > >> Thanks in advance for all your inputs,
> > > >> --
> > > >> Parag P. Doke
> > > >> http://paragpdoke.blogspot.com
> > > >>
> > > >> [Non-text portions of this message have been removed]
> > > >>
> > > >>
> > > >>
> > > >
> > > > [Non-text portions of this message have been removed]
> > > >
> > > >
> > >
> > > --
> > > Parag P. Doke
> > > http://paragpdoke.blogspot.com
> > >
> > >
> >
> >
> > [Non-text portions of this message have been removed]
> >
>
>  
>

[Non-text portions of this message have been removed]

[Andrew Palm-2]

Remove the /MIR and it will do a oneway copy...

Copying is restartable from the point it stopped, but it doesn't do delta
copies, which is what your after.

Enjoy your solution.

Andrew



On Sat, Sep 6, 2008 at 7:57 PM, Parag P. Doke <paragpdoke@...> wrote:

>   Hello Andrew.
> Thanks for the explanation about how robocopy helps to create mirrors.
> However, that is not really what I was looking for. In my case, the
> remote shared folder gets a new build created (in a folder) and my
> intent is to copy over only the newly created folder to my local
> machine (regardless of what happens to folders / files created in
> other locations inside the source).
>
> When I ran the robocopy command as suggested by you (including /l), it
> did list the contents it was about to copy over. Without the /l, it
> started to copy data.....but one file at a time. Also there was no
> indication if the file copy operation of the current file was split
> into chunks and then data re-joined on the destination. In short, I
> found axel downloading a lot faster than robocopy (over http).
>
> Thanks again.
>
> Hi Brian.
> I believe I've answered most of the questions in an earlier mail on
> this thread. Pasting it below:
> -----
>
> 1) Total size of data is somewhere around 350 MB.
> 2) Total # of files: roughly 50 (contained in 3 folders). Size of one
> file is to the tune of 330 MB. Rest total up to around 20 MB.
> 3) Pardon my lack of knowledge...but I'm copying over
> \\MachineName\SharedFolder. The system "MachineName" is located in the
> US while I'm located in India (Pune). I'm not sure whether to call it
> LAN or WAN.
> 4) Yes, I'm trying to keep 2 locations in sync. The copy process is
> triggered each morning.
> [Update: With the explanation from Andrew, let me correct
> myself....I'm not trying to keep 2 folders in sync....and not trying
> to create a mirror / replica.]
> 5) Data is copied only one way.
> -----
> There is no known issue with the equipment.
>
> Thank you.
>
> Hi Michael.
> The source of the data is a computer that I don't have administrative
> rights on. So I was trying for something that needs least time/effort
> for setup, for the source administrator.
>
> Though it will be possible to come up with a script that generates zip
> file(s) for the source data and run it as a scheduled task on the
> source, and then my local script gets the zip files here and extracts
> them is one possible solution, I went for the host data over http
> option (similar to the ftp one that you suggested).
>
> Thank you.
>
> For the time being, I have all my scheduled tasks / scripts in place
> and the copy takes something close to a few minutes....and I'm happy
> with the performance / time observed.
>
> Regards,
> Parag P. Doke
>
>
> On Sat, Sep 6, 2008 at 2:08 PM, Michael Burek <mike@...<mike%40mikeburek.com>>
> wrote:
> > @Brian Lasch:
> >
> > I'm pretty much under the impression that copying is limited to the disk
> > speed, or bus bandwidth. I really don't think the processor is involved
> > enough to make a difference. Especially at Gigahertz speeds. Using
> > firewire drives will take nearly all the processing off of the cpu and
> put
> > in onto the firewire interface card.
> >
> > Also, a hard drive has a single head, so a hard drive itself is most
> > efficient in a single thread sequential read. Having 2 threads read from
> 2
> > areas of the drive will cause seek time to enter the time equation and
> slow
> > things down. Having a big file, all in a row, will produce the fastest
> read
> > result. Even if drives had 2 separate 1 gigabyte buffers, those buffers
> > would have to take turns reading with only 1 drive head. And even with a
> > solid state drive or a flash drive, those still work faster with a
> > sequential read than a random read. But their random read is much faster
> > than a platter hard drive random read.
> >
> > PS, if you read the question, you'll get your answers to what the
> question
> > is about.
> >
> > -----
> > @Parag P. Doke
> >
> > What is the bandwidth available? If you max it out during the copying,
> then
> > no software can increase the speed. The only hope is to zip the files on
> > the source computer, and transfer the zip files, when your bandwidth is
> the
> > bottleneck. You'd have to zip each file separately and update the zip
> file
> > on the source computer when the original changes, so that you'd only copy
> > the updated files. Zipping into 1 large file would always bring over
> every
> > file inside that zip file - causing excess copying for static files.
> >
> > On the source computer, can an FTP server be run? FTP servers will allow
> > for multipart copies. But again, if you are maxing out your bandwidth,
> > multipart copying will not push more bits through the pipes than simple
> > copying. Although your Axel program already does this through the http
> > server on the source machine.
> >
> > -----
> >
> > [Non-text portions of this message have been removed]
> >
> >
>
> --
> Parag P. Doke
> http://paragpdoke.blogspot.com
>  
>

[Non-text portions of this message have been removed]