Parallel downloading of Cygwin packages

[Ron de Bruijn]

Hi,

I use the latest installer, but it only uses one mirror to download
packages from at the same time. I have the impression that the mirror I
selected only gives about 30KB/s, which makes installing Cygwin slow.
What I would like the installer to do is to download from all mirrors in
parallel different packages and in case one mirror doesn't respond that
it just drops that one and uses another one. The mirrors together should
be able to saturate my connection.

Best regards,

Ron de Bruijn



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Christopher Faylor-8]

On Fri, Oct 30, 2009 at 01:19:58PM +0100, Ron de Bruijn wrote:
>I use the latest installer, but it only uses one mirror to download
>packages from at the same time.  I have the impression that the mirror
>I selected only gives about 30KB/s, which makes installing Cygwin slow.
>What I would like the installer to do is to download from all mirrors
>in parallel different packages and in case one mirror doesn't respond
>that it just drops that one and uses another one.  The mirrors together
>should be able to saturate my connection.

I'd like the installer to use a more intuitive UI, be more capable of
running from the command line, and be able to handle RPMs.

FYI.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Paul McFerrin-2]

Don't you mean, "Stand in line? :)

Christopher Faylor wrote:

> On Fri, Oct 30, 2009 at 01:19:58PM +0100, Ron de Bruijn wrote:
>  
>> I use the latest installer, but it only uses one mirror to download
>> packages from at the same time.  I have the impression that the mirror
>> I selected only gives about 30KB/s, which makes installing Cygwin slow.
>> What I would like the installer to do is to download from all mirrors
>> in parallel different packages and in case one mirror doesn't respond
>> that it just drops that one and uses another one.  The mirrors together
>> should be able to saturate my connection.
>>    
>
> I'd like the installer to use a more intuitive UI, be more capable of
> running from the command line, and be able to handle RPMs.
>
> FYI.
>
> cgf
>
> --
> Problem reports:       http://cygwin.com/problems.html
> FAQ:                   http://cygwin.com/faq/
> Documentation:         http://cygwin.com/docs.html
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>
>
>  

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Chris Cormie-2]

Christopher Faylor wrote:
> On Fri, Oct 30, 2009 at 01:19:58PM +0100, Ron de Bruijn wrote:
>> I use the latest installer, but it only uses one mirror to download
>> packages from at the same time.  I have the impression that the mirror
>> I selected only gives about 30KB/s, which makes installing Cygwin slow.
>> What I would like the installer to do is to download from all mirrors
>> in parallel different packages and in case one mirror doesn't respond
>> that it just drops that one and uses another one.  The mirrors together
>> should be able to saturate my connection.

With respect Ron, why not select a fast mirror if your current mirror is
slow? A single fast mirror is always going to be faster than multiple
mirrors, so there is little reason for the setup.exe team to support
multiple mirrors, or so it seems to me.

> I'd like the installer to use a more intuitive UI, be more capable of
> running from the command line, and be able to handle RPMs.
http://code.google.com/p/cyg-apt/
(Command line installer project.)

Doesn't handle RPMs, only Cygwin packages. Does have a apt-get style
interface. I'm thinking RPM support is a bit beside the point? A binary
RPM from a different distro won't work, packages usually need porting to
  Cygwin -- that's all part of the Cygwin package system.

If you only need to install specific packages, setup.exe has you covered:
$./setup-1.7.exe -q -P joe

Cheers,
Chris


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Christopher Faylor-8]

On Sat, Oct 31, 2009 at 01:28:34PM +1100, Chris Cormie wrote:
>Christopher Faylor wrote:
>> I'd like the installer to use a more intuitive UI, be more capable of
>> running from the command line, and be able to handle RPMs.
>http://code.google.com/p/cyg-apt/
>(Command line installer project.)
>
>Doesn't handle RPMs, only Cygwin packages. Does have a apt-get style
>interface. I'm thinking RPM support is a bit beside the point?

Nope.  A real package manager would be very nice.

>A binary RPM from a different distro won't work, packages usually need
>porting to
>  Cygwin -- that's all part of the Cygwin package system.
>
>If you only need to install specific packages, setup.exe has you covered:
>$./setup-1.7.exe -q -P joe

I'm very well aware of setup's capabilities.  I am one of the authors
after all.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Ralph Hempel]

Chris Cormie wrote:
> Christopher Faylor wrote:

>> I'd like the installer to use a more intuitive UI, be more capable of
>> running from the command line, and be able to handle RPMs.
> http://code.google.com/p/cyg-apt/
> (Command line installer project.)
>
> Doesn't handle RPMs, only Cygwin packages. Does have a apt-get style
> interface. I'm thinking RPM support is a bit beside the point? A binary
> RPM from a different distro won't work, packages usually need porting to
>  Cygwin -- that's all part of the Cygwin package system.

Chris Cormie, meet Christopher Faylor - Master of the understated
ironic wisecrack.

Ralph

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Chris Cormie-2]

Christopher Faylor wrote:

> On Sat, Oct 31, 2009 at 01:28:34PM +1100, Chris Cormie wrote:
>> Christopher Faylor wrote:
>>> I'd like the installer to use a more intuitive UI, be more capable of
>>> running from the command line, and be able to handle RPMs.
>> http://code.google.com/p/cyg-apt/
>> (Command line installer project.)
>>
>> Doesn't handle RPMs, only Cygwin packages. Does have a apt-get style
>> interface. I'm thinking RPM support is a bit beside the point?
>
> Nope.  A real package manager would be very nice.

Ah, so you aren't a user expecting other distros RPMs to "just work"
with Cygwin, there are specific package management functions in a
RPM-like system you want? Or do you think Cygwin would benefit from
using a standard package format for ported packages over the custom
format we currently use?

>> A binary RPM from a different distro won't work, packages usually need
>> porting to
>>  Cygwin -- that's all part of the Cygwin package system.
>>
>> If you only need to install specific packages, setup.exe has you covered:
>> $./setup-1.7.exe -q -P joe
>
> I'm very well aware of setup's capabilities.  I am one of the authors
> after all.

Hi, nice to meet you. I was mentioning setup's CLI because I just
mentioned cyg-apt and didn't want to give the maillist the impression
setup had no CLI.

Cheers,
Chris.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Christopher Faylor-8]

On Sat, Oct 31, 2009 at 03:36:33PM +1100, Chris Cormie wrote:

>Christopher Faylor wrote:
>> On Sat, Oct 31, 2009 at 01:28:34PM +1100, Chris Cormie wrote:
>>> Christopher Faylor wrote:
>>>> I'd like the installer to use a more intuitive UI, be more capable of
>>>> running from the command line, and be able to handle RPMs.
>>> http://code.google.com/p/cyg-apt/
>>> (Command line installer project.)
>>>
>>> Doesn't handle RPMs, only Cygwin packages. Does have a apt-get style
>>> interface. I'm thinking RPM support is a bit beside the point?
>>
>> Nope.  A real package manager would be very nice.
>
>Ah, so you aren't a user expecting other distros RPMs to "just work"
>with Cygwin,

Do I think that a linux RPM would work on Cygwin?  No, of course not.  I
wrote the "What Isn't Cygwin" words on the Cygwin web page.

>there are specific package management functions in a RPM-like system
>you want?  Or do you think Cygwin would benefit from using a standard
>package format for ported packages over the custom format we currently
>use?

Just in case you're not getting this:
http://cygwin.com/who.html

But, yes, I think that the project would benefit from a standard package
manager and, since Cygwin is owned by Red Hat and currently hosted on a
Red Hat-owned site, I think it makes sense for the package manager to be
rpm based.

There are barriers to implementing rpm in Cygwin, the most frequently
mentioned being the fact that a cygwin process can't easily replace
cygwin1.dll or any other running executables.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Chris Cormie-2]

> There are barriers to implementing rpm in Cygwin, the most frequently
> mentioned being the fact that a cygwin process can't easily replace
> cygwin1.dll or any other running executables.

If the in-Cygwin package manager can't update core Cygwin, that's not a
deal breaker IMO: there is a fair bit of useful app package management
that can be done without changing the core.

As for running executables -- well, perhaps this is more of my ignorance
:) but couldn't a well behaved package close down running services in
the preremove script? (In the case the package was being upgraded.)

Best Regards,
Chris

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Larry Hall (Cygwin)]

On 11/01/2009 10:25 PM, Chris Cormie wrote:
>> There are barriers to implementing rpm in Cygwin, the most frequently
>> mentioned being the fact that a cygwin process can't easily replace
>> cygwin1.dll or any other running executables.
>
> If the in-Cygwin package manager can't update core Cygwin, that's not a
> deal breaker IMO: there is a fair bit of useful app package management
> that can be done without changing the core.

Sure but the value of package manager that can't do this mitigates its general
usefulness and adds to the "confusion" factor for people
installing/updating/removing,
etc.  Given the number of postings on this list which stem from confusion about
the operation of 'setup.exe', the importance of this factor shouldn't be
underestimated.

> As for running executables -- well, perhaps this is more of my ignorance
> :) but couldn't a well behaved package close down running services in
> the preremove script? (In the case the package was being upgraded.)

Sure but it is another issue where lack of symmetry is likely to cause problems.
Those used to using rpm or other package managers on platforms where this is
not a strict requirement for upgrading packages will likely be confused or
annoyed
by the Cygwin's requirement for this.  This, I believe, is less of a crucial
issue
than the one above.

--
Larry Hall                              http://www.rfk.com
RFK Partners, Inc.                      [phone] - RFK Office
216 Dalton Rd.                          [phone] - FAX
Holliston, MA 01746

_____________________________________________________________________

A: Yes.
 > Q: Are you sure?
 >> A: Because it reverses the logical flow of conversation.
 >>> Q: Why is top posting annoying in email?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Ron de Bruijn]

Hi Chris,

Chris Cormie schreef:

> Christopher Faylor wrote:
>> On Fri, Oct 30, 2009 at 01:19:58PM +0100, Ron de Bruijn wrote:
>>> I use the latest installer, but it only uses one mirror to download
>>> packages from at the same time.  I have the impression that the mirror
>>> I selected only gives about 30KB/s, which makes installing Cygwin slow.
>>> What I would like the installer to do is to download from all mirrors
>>> in parallel different packages and in case one mirror doesn't respond
>>> that it just drops that one and uses another one.  The mirrors together
>>> should be able to saturate my connection.
>
> With respect Ron, why not select a fast mirror if your current mirror is
> slow? A single fast mirror is always going to be faster than multiple
> mirrors, so there is little reason for the setup.exe team to support
> multiple mirrors, or so it seems to me.

A single fast mirror is only going to be faster than multiple mirrors
for unrealistic values of "fast". The reason for me not picking a fast
mirror is simple; the only way to find out the speed of all the mirrors
on this particular network is trying them all, because there is a proxy
which only sends the files to my machine when they are completely done
downloading. If the list of required packages is documented in a dead
simple format (which it probably is) and the list of mirrors is always
in some stable location, then writing something which downloads
everything in parallel should be less than 50 lines of code. I am not
saying that anyone is obliged to do so, I am just saying it isn't a huge
problem.

Best regards,
  Ron de Bruijn





--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Andy Koppe]

2009/11/2 Ron de Bruijn:
> A single fast mirror is only going to be faster than multiple mirrors for
> unrealistic values of "fast".

Well, I regularly get 1 MByte/s from nearby mirrors. Call me
undemanding, but I think that's a realistic value for "fast".

> The reason for me not picking a fast mirror is
> simple; the only way to find out the speed of all the mirrors on this
> particular network is trying them all, because there is a proxy which only
> sends the files to my machine when they are completely done downloading. If
> the list of required packages is documented in a dead simple format (which
> it probably is) and the list of mirrors is always in some stable location,
> then writing something which downloads everything in parallel should be less
> than 50 lines of code.

*splutter*. You obviously haven't seen setup.exe's source code, and
you're forgetting that any such solution also needs to be integrated
into the UI, most interestingly, the progress screen, which is
currently designed for downloading one package at a time. Parallel
downloading would be nice to have, but it's nowhere near the top of
the list of potential setup.exe improvements.

Meanwhile, you could always implement your scheme as a script that
downloads into a local directory and installs from there.

Andy

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Christopher Faylor-8]

On Mon, Nov 02, 2009 at 02:25:03PM +1100, Chris Cormie wrote:
>> There are barriers to implementing rpm in Cygwin, the most frequently
>> mentioned being the fact that a cygwin process can't easily replace
>> cygwin1.dll or any other running executables.
>
>If the in-Cygwin package manager can't update core Cygwin, that's not a
>deal breaker IMO: there is a fair bit of useful app package management
>that can be done without changing the core.

Maybe it's not clear but your personal needs are not really the issue
here.  There are many people for whom not being able to update the
cygwin DLL is a big deal.

>As for running executables -- well, perhaps this is more of my ignorance
>:) but couldn't a well behaved package close down running services in
>the preremove script? (In the case the package was being upgraded.)

This subject has been talked about quite a few times.  The mailing list
archives should have lots of information if you want to educate
yourself.

I will close my participation in this thread by pointing out that Cygwin
1.7 will come with a version of setup.exe that allows updating and
installation of packages from the command line.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[paul.hermeneutic]

On Mon, Nov 2, 2009 at 06:46, Andy Koppe <andy.koppe@...> wrote:

>
> *splutter*. You obviously haven't seen setup.exe's source code, and
> you're forgetting that any such solution also needs to be integrated
> into the UI, most interestingly, the progress screen, which is
> currently designed for downloading one package at a time. Parallel
> downloading would be nice to have, but it's nowhere near the top of
> the list of potential setup.exe improvements.
>
> Meanwhile, you could always implement your scheme as a script that
> downloads into a local directory and installs from there.

It sounds like what some want is a bittorrent-like operation.

Could a load balancer running on cygwin.com perform this distribution
better than embedding the logic into setup.exe?

-pdw

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Christopher Faylor-8]

On Mon, Nov 02, 2009 at 11:14:04AM -0600, paul.hermeneutic@... wrote:

>On Mon, Nov 2, 2009 at 06:46, Andy Koppe <andy.koppe@...> wrote:
>>
>> *splutter*. You obviously haven't seen setup.exe's source code, and
>> you're forgetting that any such solution also needs to be integrated
>> into the UI, most interestingly, the progress screen, which is
>> currently designed for downloading one package at a time. Parallel
>> downloading would be nice to have, but it's nowhere near the top of
>> the list of potential setup.exe improvements.
>>
>> Meanwhile, you could always implement your scheme as a script that
>> downloads into a local directory and installs from there.
>
>It sounds like what some want is a bittorrent-like operation.
>
>Could a load balancer running on cygwin.com perform this distribution
>better than embedding the logic into setup.exe?

No.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[Chris Cormie-2]

Christopher Faylor wrote:

> On Mon, Nov 02, 2009 at 02:25:03PM +1100, Chris Cormie wrote:
>>> There are barriers to implementing rpm in Cygwin, the most frequently
>>> mentioned being the fact that a cygwin process can't easily replace
>>> cygwin1.dll or any other running executables.
>> If the in-Cygwin package manager can't update core Cygwin, that's not a
>> deal breaker IMO: there is a fair bit of useful app package management
>> that can be done without changing the core.
>
> Maybe it's not clear but your personal needs are not really the issue
> here.

Huh? Why would you think that? I have my own solution for this issue.
Personal needs never entered into it.

Regards,
Chris.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

[paul.hermeneutic]

On Mon, Nov 2, 2009 at 22:39, Chris Cormie <cjcormie@...> wrote:

> Christopher Faylor wrote:
>>
>> On Mon, Nov 02, 2009 at 02:25:03PM +1100, Chris Cormie wrote:
>>>>
>>>> There are barriers to implementing rpm in Cygwin, the most frequently
>>>> mentioned being the fact that a cygwin process can't easily replace
>>>> cygwin1.dll or any other running executables.
>>>
>>> If the in-Cygwin package manager can't update core Cygwin, that's not a
>>> deal breaker IMO: there is a fair bit of useful app package management that
>>> can be done without changing the core.
>>
>> Maybe it's not clear but your personal needs are not really the issue
>> here.

I thought that the standard Windows procedure was to create a run-once
to do the replacement after the machine is rebooted.  Would that not
work here?

Yes, it would be much nicer to do things better than Microsoft, but
that is the current structure.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple