Proposal: concurrent remote-requests

[Marc Lustig]

Hi all,

we have configured about 25 remote-repos for our public-artifacts managed repo.
In certain cases, black and white lists don't help and a request is proxied to all the 20 remote-repos _sequentially_. Even thou we have configured a short timeout of 5 secs, this takes 125 secs in case the artifacts doesn't exist in any remote-repo - per artifact!

So I was wondering if it would make sense to send requests to all of the remote-repos _concurrently_.
The first thread that find the artifacts could cause all the other threads to cancel the http-request.
The total request time would reduce from 100 secs++ to merely 5 secs. Tremendous win or?

Has this been discussed before?
Is there an argument against this strategy?

The implementation could be based on a thread-pool, or rather a pool of thread-pools.

greetings
Marc

[brettporter]

On 15/10/2009, at 12:06 AM, Marc Lustig wrote:

>
> Hi all,
>
> we have configured about 25 remote-repos for our public-artifacts  
> managed
> repo.
> In certain cases, black and white lists don't help and a request is  
> proxied
> to all the 20 remote-repos _sequentially_. Even thou we have  
> configured a
> short timeout of 5 secs, this takes 125 secs in case the artifacts  
> doesn't
> exist in any remote-repo - per artifact!
>
> So I was wondering if it would make sense to send requests to all of  
> the
> remote-repos _concurrently_.
> The first thread that find the artifacts could cause all the other  
> threads
> to cancel the http-request.
> The total request time would reduce from 100 secs++ to merely 5 secs.
> Tremendous win or?
>
> Has this been discussed before?

I think this is a pretty unusual case. I don't quite understand why  
you are hitting the timeout limit on the remote repo - if they are up  
they should be fast. Also, "first that finds" is different to the  
current rule since it's first that appears in the list. I worry that  
in this set up you're not entirely sure which repository the artifacts  
are meant to be coming from, so maybe it points to another problem.

> Is there an argument against this strategy?

Particularly if we turned on streaming of the proxied download to the  
client (which is intended) - we couldn't do so if they were pooled  
like this, unless we accepted the "first found rule".

That said, this might speed up requests with a long list of proxies,  
even if they are functioning properly. So it might be reasonable as an  
optional capability. One thing to consider would be doing a HEAD  
request instead of a GET for all the remotes first to select where to  
download from, then execute the GET from the desired one.

- Brett

[Marc Lustig]

thanks Brett for the input.
I can confirm that using black and white lists the case is rather rare when all remote-repos are searched sequentially and the artifact is not found in the end. However it is typical for some scenarios e. g. when you enable the source-jars to get downloaded for a project. From 40 deps, maybe 5 will have source-jars available. In that way a simple mvn-goal takes 30 minutes or more.

I mentioned the timeout just to have a maximum value.  Of course usually the requests don't run in a timeout (except when the repo is down) - the average response time is maybe 3-4 secs (for our installation).

Also it is clear that the first-serve concept conflicts with the existing concept of an (ordered) list of repos that is searched for.
Can we not assume that artifacts with a given spec. are identical from whatever repo they come, provided the hash is matching?

Btw., this brings up another idea: could the ASF possibly grant "official" certificates for remote-repos?
In that way, Archiva could distinguish between trusted and non-trusted repos.
For companies, this would be a compelling feature! I (working for insurances and banks) often hear the argument "of boy - they are downloading software from some obscure server from russia". Having the label "Certified Maven Repository" would surely make those noises more silent :-)
The ASF could release a rule-set that the Maven-repo must conform to in order to get the "certified" label.
Or even better, the ASF could offer a VMware-image that includes all the software ready to run the Maven-repo - including some logic to verify that known artifacts are mirrored correctly. A total control of repos is not possible, of course. But the contract between Archiva and the remote repo could be tightened pretty much.


Back to the concurrent requests idea: sending the HEAD request before the actual GET is surely a good idea. Archiva could decide to which repo to send the GET based on the shortest response-time.
Anyway, this feature needs more brainstorming...

brettporter wrote:

On 15/10/2009, at 12:06 AM, Marc Lustig wrote:

>
> Hi all,
>
> we have configured about 25 remote-repos for our public-artifacts  
> managed
> repo.
> In certain cases, black and white lists don't help and a request is  
> proxied
> to all the 20 remote-repos _sequentially_. Even thou we have  
> configured a
> short timeout of 5 secs, this takes 125 secs in case the artifacts  
> doesn't
> exist in any remote-repo - per artifact!
>
> So I was wondering if it would make sense to send requests to all of  
> the
> remote-repos _concurrently_.
> The first thread that find the artifacts could cause all the other  
> threads
> to cancel the http-request.
> The total request time would reduce from 100 secs++ to merely 5 secs.
> Tremendous win or?
>
> Has this been discussed before?

I think this is a pretty unusual case. I don't quite understand why  
you are hitting the timeout limit on the remote repo - if they are up  
they should be fast. Also, "first that finds" is different to the  
current rule since it's first that appears in the list. I worry that  
in this set up you're not entirely sure which repository the artifacts  
are meant to be coming from, so maybe it points to another problem.

> Is there an argument against this strategy?

Particularly if we turned on streaming of the proxied download to the  
client (which is intended) - we couldn't do so if they were pooled  
like this, unless we accepted the "first found rule".

That said, this might speed up requests with a long list of proxies,  
even if they are functioning properly. So it might be reasonable as an  
optional capability. One thing to consider would be doing a HEAD  
request instead of a GET for all the remotes first to select where to  
download from, then execute the GET from the desired one.

- Brett

[brettporter]

On 15/10/2009, at 7:00 PM, Marc Lustig wrote:

> For companies, this would be a compelling feature! I (working for  
> insurances
> and banks) often hear the argument "of boy - they are downloading  
> software
> from some obscure server from russia". Having the label "Certified  
> Maven
> Repository" would surely make those noises more silent :-)
> The ASF could release a rule-set that the Maven-repo must conform to  
> in
> order to get the "certified" label.

This isn't really in the ASF's mission to provide. Everyone is going  
to have their own rules for what is certified - there are varying  
levels of trust, even if you verify it comes from the project itself  
(for example, see Eclipse's IP verification process).

In this case you are better off having a dedicated group of people  
approving third party artifacts to arrive into Archiva for use by  
others, and limiting proxy access outside. You can obviously do this  
manually in Archiva now, though ideally you want a "quarantine" area  
where they can be retrieved and await approval with a decent workflow  
for moving them into an accessible repository.

- Brett