cygwin copy problems usb 2.0

[aldana]

hi,

i've got usb 2.0, when i copy through TotalCommander the copy speed is quite high. when i copy through cygwin shell it seems that it is transmitting data only with usb 1.x speed.
very spooky, because i thought that cygwin is calling windows drivers/api so it should be indirectly supporting usb2.0 ?

thanks.

[Gary R. Van Sickle]

> From: aldana
> Sent: Thursday, July 27, 2006 7:15 AM
> Subject: cygwin copy problems usb 2.0
>
>
> hi,
>
> i've got usb 2.0, when i copy through TotalCommander the copy
> speed is quite high.

I have no idea what a TotalCommander is, but from the context here I assume
it's a native Windows app that has copying abilities which are highly
optimized for speed.

> when i copy through cygwin shell it
> seems that it is transmitting data only with usb 1.x speed.
> very spooky, because i thought that cygwin is calling windows
> drivers/api so it should be indirectly supporting usb2.0 ?
>

I can assure you with 99.44% confidence that Cygwin is not somehow
downgrading your USB connections from 2.0 to 1.x.  Such decisions get made
at a much much lower level than user-mode apps operate at.  In fact, I doubt
it's even possible for this to happen in kernel-mode code - I think this all
happens at the hub/device hardware/SIE level.

Cygwin's cp is slower than a native copy, from anywhere to anywhere else,
for a number of reasons.  The main one, last I looked (admittedly years
ago), was a fallout of portability.  Cp opens file and copies them an
(f)read/(f)write buffer at a time, while your Windows native program almost
certainly simply calls CopyFile() (a Win32 API), bypassing a ton of library
code.

--
Gary R. Van Sickle
 


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

[aldana]

total commander is a clone of norton commander. something like midnight commander on linux.

i thought the same: choice between usb 1.x and 2.0 is done far lower level that cygwin can really influence it. but the difference of speed made me consipicious.

example, copying a single file (.tar.gz) of size ~70MB:
with total commander it takes 20 sec.
with cygwin it takes 4 minutes (!), which is plain too long to include it in my script.

i know through its abstracton layer cygwin must be slower. but regarding that slowliness could it be another reason?

thanks again!

[Larry Hall (Cygwin)]

aldana wrote:

> total commander is a clone of norton commander. something like midnight
> commander on linux.
>
> i thought the same: choice between usb 1.x and 2.0 is done far lower level
> that cygwin can really influence it. but the difference of speed made me
> consipicious.
>
> example, copying a single file (.tar.gz) of size ~70MB:
> with total commander it takes 20 sec.
> with cygwin it takes 4 minutes (!), which is plain too long to include it in
> my script.
>
> i know through its abstracton layer cygwin must be slower. but regarding
> that slowliness could it be another reason?

There could be.  But why speculate?  Write yourself a simple utility that uses
CopyFile() directly and see if that makes up the difference.


--
Larry Hall                              http://www.rfk.com
RFK Partners, Inc.                      [phone] - RFK Office
216 Dalton Rd.                          [phone] - FAX
Holliston, MA 01746

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

[aldana]

when running a little program using CopyFile() under cygwin it is about as quick as totalcommander. so it must be the abstraction layer of cygwin which makes copying veeeery slow...

[Larry Hall (Cygwin)]

aldana wrote:
> when running a little program using CopyFile() under cygwin it is about as
> quick as totalcommander. so it must be the abstraction layer of cygwin which
> makes copying veeeery slow...


Not necessarily.  To draw that conclusion, you would want to compare the
implementation of 'cp' compiled natively ('gcc -mno-cygwin' for example)
with Cygwin's and your little program.  Only if the natively compiled
version performed much closer to your program using CopyFile() could you
conclude that Cygwin is to blame.

--
Larry Hall                              http://www.rfk.com
RFK Partners, Inc.                      [phone] - RFK Office
216 Dalton Rd.                          [phone] - FAX
Holliston, MA 01746

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

[Christopher Faylor-2]

On Thu, Jul 27, 2006 at 04:17:14PM -0400, Larry Hall (Cygwin) wrote:

>aldana wrote:
>>when running a little program using CopyFile() under cygwin it is about
>>as quick as totalcommander.  so it must be the abstraction layer of
>>cygwin which makes copying veeeery slow...
>
>Not necessarily.  To draw that conclusion, you would want to compare
>the implementation of 'cp' compiled natively ('gcc -mno-cygwin' for
>example) with Cygwin's and your little program.  Only if the natively
>compiled version performed much closer to your program using CopyFile()
>could you conclude that Cygwin is to blame.

Of course, cygwin and gcc don't use CopyFile at all, so this is really
just a read/write loop.

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

[aldana]

isn't there a possibitly that cygwin provides a quicker cp-implementation?
i mean 4 minutes for a copy of 70MB to a memstick (instead of CopyFile() 20 sec.) is not really good performance.  
i guess there is a reason for that...

[Corinna Vinschen-2]

On Jul 27 13:48, aldana wrote:
>
> isn't there a possibitly that cygwin provides a quicker cp-implementation?
> i mean 4 minutes for a copy of 70MB to a memstick (instead of CopyFile() 20
> sec.) is not really good performance.  
> i guess there is a reason for that...

Right, how did you know?  The reason is that cp is a portable
implementation using simple reads and writes to perform the copy.
There's no such thing as a CopyFile routine on POSIX systems.


Corinna

--
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

[Larry Hall (Cygwin)]

aldana wrote:
> isn't there a possibitly that cygwin provides a quicker cp-implementation?
> i mean 4 minutes for a copy of 70MB to a memstick (instead of CopyFile() 20
> sec.) is not really good performance.  
> i guess there is a reason for that...

Chris has already given you the answer but feel free to look at the source
of 'cp' and offer your portable solution.

--
Larry Hall                              http://www.rfk.com
RFK Partners, Inc.                      [phone] - RFK Office
216 Dalton Rd.                          [phone] - FAX
Holliston, MA 01746

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

[Christopher Faylor-2]

On Thu, Jul 27, 2006 at 11:11:07PM +0200, Corinna Vinschen wrote:
>On Jul 27 13:48, aldana wrote:
>>isn't there a possibitly that cygwin provides a quicker
>>cp-implementation?  i mean 4 minutes for a copy of 70MB to a memstick
>>(instead of CopyFile() 20 sec.) is not really good performance.  i
>>guess there is a reason for that...
>
>Right, how did you know?  The reason is that cp is a portable
>implementation using simple reads and writes to perform the copy.
>There's no such thing as a CopyFile routine on POSIX systems.

A few weeks ago there was a guy in libc-alpha mailing list complaining
that glibc's API wasn't as rich and powerful as what is found on Windows.

As far as I know he's still alive.

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

[Jan Schormann]

You wrote on Thursday, July 27, 2006 7:40 PM:

> ... which is plain too long to include it in my script.

Have you tried "copy" instead of "cp"? It's plain Windows
and it's a command line tool. I don't know whether it uses
CopyFile(), but it might be worth a try, right?

Cheers,
        Jan.

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

[Markus Schönhaber-4]

Christopher Faylor wrote:
> A few weeks ago there was a guy in libc-alpha mailing list complaining
> that glibc's API wasn't as rich and powerful as what is found on Windows.
>
> As far as I know he's still alive.

You made my day!

Regards
  mks

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

[aldana]

yes that's the most obvious solution for my problem.

[Reini Urban]

Christopher Faylor schrieb:

> On Thu, Jul 27, 2006 at 11:11:07PM +0200, Corinna Vinschen wrote:
>> On Jul 27 13:48, aldana wrote:
>>> isn't there a possibitly that cygwin provides a quicker
>>> cp-implementation?  i mean 4 minutes for a copy of 70MB to a memstick
>>> (instead of CopyFile() 20 sec.) is not really good performance.  i
>>> guess there is a reason for that...
>> Right, how did you know?  The reason is that cp is a portable
>> implementation using simple reads and writes to perform the copy.
>> There's no such thing as a CopyFile routine on POSIX systems.
>
> A few weeks ago there was a guy in libc-alpha mailing list complaining
> that glibc's API wasn't as rich and powerful as what is found on Windows.
>
> As far as I know he's still alive.

Well, this brave guy has a point. :)

I'm really seeing the non-optimized cygwin cp behaviour causing bad
reputation, which could be easily patched and maybe even accepted
upstream. Who knows. Eric what do think? Would it be worthful to think
about?

My USB transfers are also way to slow, because we are copying our hourly
radio shows (80 MB) onto the stick every week. Everyone is angry. Linux
USB drivers are very unstable, so windows has a good reputation there.

BTW: Windows Explorer is even faster than Total Commander. We tested it.
--
Reini

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

[Christopher Faylor-2]

On Thu, Aug 03, 2006 at 02:11:00AM +0200, Reini Urban wrote:

>Christopher Faylor schrieb:
>>On Thu, Jul 27, 2006 at 11:11:07PM +0200, Corinna Vinschen wrote:
>>>On Jul 27 13:48, aldana wrote:
>>>>isn't there a possibitly that cygwin provides a quicker
>>>>cp-implementation?  i mean 4 minutes for a copy of 70MB to a memstick
>>>>(instead of CopyFile() 20 sec.) is not really good performance.  i
>>>>guess there is a reason for that...
>>>Right, how did you know?  The reason is that cp is a portable
>>>implementation using simple reads and writes to perform the copy.
>>>There's no such thing as a CopyFile routine on POSIX systems.
>>
>>A few weeks ago there was a guy in libc-alpha mailing list complaining
>>that glibc's API wasn't as rich and powerful as what is found on Windows.
>>
>>As far as I know he's still alive.
>
>Well, this brave guy has a point. :)

He wasn't brave.  He was stupid.  He didn't understand what glibc was
supposed to be providing and he wouldn't understand it even when it was
explained to him.

>I'm really seeing the non-optimized cygwin cp behaviour causing bad
>reputation, which could be easily patched and maybe even accepted
>upstream. Who knows. Eric what do think? Would it be worthful to think
>about?

If this is what you want then you should look into a non-cygwin
solution.  There are a couple of projects which provide GNU tools for
Windows without resorting to something like the Cygwin DLL.

Also, don't you see something wrong with the mindset of "Windows Fast.
Cygwin Slow.  So, must use straight Windows functions." without even
bothering to do any research into what is causing the slowness?  How do
you, or anyone who cares about this know that this "problem", know that
it isn't correctable without resorting to patching cp?

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

[Eric Blake-2]

> >I'm really seeing the non-optimized cygwin cp behaviour causing bad
> >reputation, which could be easily patched and maybe even accepted
> >upstream. Who knows. Eric what do think? Would it be worthful to think
> >about?

I don't really want to maintain a Windows API patch, and doubt that
it would be accepted upstream.  Now if there were something more
POSIX-y that we could do to speed things up, such as posix_fadvise,
which cygwin could translate into whatever Windows API hooks that
would improve the situation, then that would be the way to go.

>
> If this is what you want then you should look into a non-cygwin
> solution.  There are a couple of projects which provide GNU tools for
> Windows without resorting to something like the Cygwin DLL.

Agreed.  My other big worry is that I have no control over whether
using straight Windows API will violate other POSIX assumptions,  thus
making cp (and mv) noncompliant.  I am not a fan of mixing cygwin and
non-cygwin APIs when it can be helped.

That said, if someone else comes up with a potential patch, I will
certainly review it.  But it is not my highest priority right now.

--
Eric Blake


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

[Reini Urban]

Christopher Faylor schrieb:

> On Thu, Aug 03, 2006 at 02:11:00AM +0200, Reini Urban wrote:
>> Christopher Faylor schrieb:
>>> On Thu, Jul 27, 2006 at 11:11:07PM +0200, Corinna Vinschen wrote:
>>>> On Jul 27 13:48, aldana wrote:
>>>>> isn't there a possibitly that cygwin provides a quicker
>>>>> cp-implementation?  i mean 4 minutes for a copy of 70MB to a memstick
>>>>> (instead of CopyFile() 20 sec.) is not really good performance.  i
>>>>> guess there is a reason for that...
>>>> Right, how did you know?  The reason is that cp is a portable
>>>> implementation using simple reads and writes to perform the copy.
>>>> There's no such thing as a CopyFile routine on POSIX systems.
>>> A few weeks ago there was a guy in libc-alpha mailing list complaining
>>> that glibc's API wasn't as rich and powerful as what is found on Windows.

...
>> I'm really seeing the non-optimized cygwin cp behaviour causing bad
>> reputation, which could be easily patched and maybe even accepted
>> upstream. Who knows. Eric what do think? Would it be worthful to think
>> about?
>
> If this is what you want then you should look into a non-cygwin
> solution.  There are a couple of projects which provide GNU tools for
> Windows without resorting to something like the Cygwin DLL.

It's not for me, I know how to call copy or xcopy or how to install
mingw's cp, which doesn't help btw.
It's for the project reputation. Think of buildtimes.

> Also, don't you see something wrong with the mindset of "Windows Fast.
> Cygwin Slow.  So, must use straight Windows functions." without even
> bothering to do any research into what is causing the slowness?  

I didn't say that and I see the same problem as you.

> How do you, or anyone who cares about this know that this "problem", know that
> it isn't correctable without resorting to patching cp?

I have no idea, but it should be investigated. If it's just the buffer
size or some waits in the driver communication.

We need not to optimize away every POSIX utility with Windows API's.
But for cp vs CopyFile would make sense. If we are on Windows supported
Filesystems, which costs nothing to get known (we have no mount calling
fs-drivers yet), why not call CopyFile() for files bigger than the
buffer size? Small files would gain nothing because of the cost of path
conversion.
But it's really up to the coreutils maintainer or some of us to come up
with a patch.
--
Reini

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

[Christopher Faylor-2]

On Thu, Aug 03, 2006 at 09:54:13AM +0200, Reini Urban wrote:

>Christopher Faylor schrieb:
>>On Thu, Aug 03, 2006 at 02:11:00AM +0200, Reini Urban wrote:
>>>Christopher Faylor schrieb:
>>>>On Thu, Jul 27, 2006 at 11:11:07PM +0200, Corinna Vinschen wrote:
>>>>>On Jul 27 13:48, aldana wrote:
>>>>>>isn't there a possibitly that cygwin provides a quicker
>>>>>>cp-implementation?  i mean 4 minutes for a copy of 70MB to a memstick
>>>>>>(instead of CopyFile() 20 sec.) is not really good performance.  i
>>>>>>guess there is a reason for that...
>>>>>Right, how did you know?  The reason is that cp is a portable
>>>>>implementation using simple reads and writes to perform the copy.
>>>>>There's no such thing as a CopyFile routine on POSIX systems.
>>>>A few weeks ago there was a guy in libc-alpha mailing list complaining
>>>>that glibc's API wasn't as rich and powerful as what is found on Windows.
>...
>>>I'm really seeing the non-optimized cygwin cp behaviour causing bad
>>>reputation, which could be easily patched and maybe even accepted
>>>upstream. Who knows. Eric what do think? Would it be worthful to think
>>>about?
>>
>>If this is what you want then you should look into a non-cygwin
>>solution.  There are a couple of projects which provide GNU tools for
>>Windows without resorting to something like the Cygwin DLL.
>
>It's not for me, I know how to call copy or xcopy or how to install
>mingw's cp, which doesn't help btw.
>It's for the project reputation. Think of buildtimes.

That logic could be used for bash, too.   Maybe Eric should spend a few
months modifying bash to avoid fork and use CreateProcess.  I'm sure it
would be faster.  It's a slippery slope and I really don't want to start
down that path for Cygwin.

>>Also, don't you see something wrong with the mindset of "Windows Fast.
>>Cygwin Slow.  So, must use straight Windows functions." without even
>>bothering to do any research into what is causing the slowness?  
>
>I didn't say that and I see the same problem as you.

No.  You don't.  You see a problem that would be solved by someone
modifying cp to use the Win32 API.

>>How do you, or anyone who cares about this know that this "problem",
>>know that it isn't correctable without resorting to patching cp?
>
>I have no idea, but it should be investigated.  If it's just the buffer
>size or some waits in the driver communication.

So, as usual in an open source project, if it bothers you that much then
investigate it.

>But it's really up to the coreutils maintainer or some of us to come up
>with a patch.

I would submit that if one of the project leaders thinks it is a
terrible idea, then the packae maintainer should seriously take that
into consideration.

Again, if this problem is really important to someone then maybe they
should take time to understand what is causing the slowness before
suggesting solutions.

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

[Corinna Vinschen-2]

On Aug  3 01:54, Eric Blake wrote:
> > >I'm really seeing the non-optimized cygwin cp behaviour causing bad
> > >reputation, which could be easily patched and maybe even accepted
> > >upstream. Who knows. Eric what do think? Would it be worthful to think
> > >about?
>
> I don't really want to maintain a Windows API patch, and doubt that

Good.

> it would be accepted upstream.  Now if there were something more
> POSIX-y that we could do to speed things up, such as posix_fadvise,

posix_fadvise can't be implemented nicely, AFAICS.  The POSIX semantics
require an already opened file and the advice is given for an offset and
a length.  The Windows semantics only allow to give the advice for the
whole file, and only switching between FILE_SEQUENTIAL_ONLY or "normal",
using ZwSetInformationFile.  By re-opening the file using ZwOpenFile it
would also be possible to toggle the FILE_RANDOM_ACCESS flag.  Still,
it's always for the whole file, not for an area giving offset and length.


Corinna

--
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/