[1.7.0-50] scp progress counter flies through first 175 MB or so

[Warren Young]

Corinna Vinschen wrote:
> What's still not clear is why the ssh process takes so much CPU.

Too many buffer copies?  It takes a surprising amount of CPU power to
fill a gigabit pipe from userland.  Double or triple that workload with
unnecessary copies, and there goes your transfer rate, evaporating on
the wind.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Brian Ford]

On Tue, 30 Jun 2009, Warren Young wrote:

> Corinna Vinschen wrote:
> > What's still not clear is why the ssh process takes so much CPU.
>
> Too many buffer copies?  It takes a surprising amount of CPU power to
> fill a gigabit pipe from userland.  Double or triple that workload with
> unnecessary copies, and there goes your transfer rate, evaporating on
> the wind.

Even more so for context switches ;-):

http://www.cygwin.com/ml/cygwin-developers/2007-10/msg00040.html

although this performance penalty was removed from read/write and friends:

http://www.cygwin.com/ml/cygwin-developers/2007-12/msg00004.html

it is still present for send/recv and friends.

I'd be happy to make a donation or organize a fund raising effort if it
would help to find a work around for this long standing performance issue.

--
Brian Ford
Staff Realtime Software Engineer
VITAL - Visual Simulation Systems
FlightSafety International
the best safety device in any aircraft is a well-trained crew...

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Corinna Vinschen-2]

On Jun 30 14:23, Brian Ford wrote:
> On Tue, 30 Jun 2009, Warren Young wrote:
>
> > Corinna Vinschen wrote:
> > > What's still not clear is why the ssh process takes so much CPU.
> >
> > Too many buffer copies?  It takes a surprising amount of CPU power to
> > fill a gigabit pipe from userland.  Double or triple that workload with
> > unnecessary copies, and there goes your transfer rate, evaporating on
> > the wind.

The data is copied only once, from the application buffer to the
underlying WinSock socket buffer and vice versa.  There's no additional
copy of buffer data in Cygwin.

> Even more so for context switches ;-):
>
> http://www.cygwin.com/ml/cygwin-developers/2007-10/msg00040.html
>
> although this performance penalty was removed from read/write and friends:
>
> http://www.cygwin.com/ml/cygwin-developers/2007-12/msg00004.html
>
> it is still present for send/recv and friends.

I removed the sig_dispatch_pending calls from send/recv and friends.
Note, however, that this has no influence on the ssh/scp results we were
talking about in this thread.  ssh/scp are using read/write calls.

Anyway, using ssh/scp with the latest from CVS looks much better now.
It doesn't eat up all CPU anymore and the performance looks pretty
well as far as I can tell.


Corinna

--
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Chris Sutcliffe-2]

> Anyway, using ssh/scp with the latest from CVS looks much better now.
> It doesn't eat up all CPU anymore and the performance looks pretty
> well as far as I can tell.

Are these changes captured in the 2009/06/30 snapshot?

Chris

--
Chris Sutcliffe
http://emergedesktop.org

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Brian Ford]

On Wed, 1 Jul 2009, Corinna Vinschen wrote:

> On Jun 30 14:23, Brian Ford wrote:
> > Even more so for context switches ;-):
> >
> > http://www.cygwin.com/ml/cygwin-developers/2007-10/msg00040.html
> >
> > although this performance penalty was removed from read/write and friends:
> >
> > http://www.cygwin.com/ml/cygwin-developers/2007-12/msg00004.html
> >
> > it is still present for send/recv and friends.
>
> I removed the sig_dispatch_pending calls from send/recv and friends.

Thank you!  This should have removed the thread handshake overhead from
most all I/O calls now.  I just wish we could figure out a way to avoid
it everywhere unless it is actually necessary.  I know, PTC :-).

> Note, however, that this has no influence on the ssh/scp results we were
> talking about in this thread.  ssh/scp are using read/write calls.

Sorry, I should have investigated and bench marked the difference.

--
Brian Ford
Staff Realtime Software Engineer
VITAL - Visual Simulation Systems
FlightSafety International
the best safety device in any aircraft is a well-trained crew...

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Corinna Vinschen-2]

On Jul  1 10:23, Chris Sutcliffe wrote:
> > Anyway, using ssh/scp with the latest from CVS looks much better now.
> > It doesn't eat up all CPU anymore and the performance looks pretty
> > well as far as I can tell.
>
> Are these changes captured in the 2009/06/30 snapshot?

Mostly.  Just try it and report what happens for you.


Corinna

--
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Christopher Faylor-8]

On Wed, Jul 01, 2009 at 10:23:14AM -0400, Chris Sutcliffe wrote:
>> Anyway, using ssh/scp with the latest from CVS looks much better now.
>> It doesn't eat up all CPU anymore and the performance looks pretty
>> well as far as I can tell.
>
>Are these changes captured in the 2009/06/30 snapshot?

The 6/30 snapshot incorporates a medium-sized rewrite to select() which
allowed me to add a blocking wait in the code for pipes.  That doesn't
eliminate the possibility that select will chew through CPU time though.

I just redid the 7/1 snapshot to pull in Corinna's changes.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Warren Young]

Corinna Vinschen wrote:
>
> Just try it and report what happens for you.

I'm seeing similar results to your above benchmarks now, with 20090701:

        upload:   30 MB/s
        download: 47 MB/s

This on the Vista-64 machine that was seeing the original reported
problem of near-instant 100% on upload.

The 50% speed increase for download vs. upload bothered me until I tried
SecureCRT's sftp, which gives substantially the same numbers.  I guess
the difference is in the network or the disks on each end.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple