maven-metadata.xml aggregation behavior

View: New views
6 Messages — Rating Filter:   Alert me  

maven-metadata.xml aggregation behavior

by Lorand :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

HI!

I'm a bit confused, so please help me understand the intention...

A little bit of background:
  • we have one simple repository (a'la repo1.maven.org) for older releases -- let's call this one OLDR
    • in simple folder/file structure
    • with generated maven-metadata.xml
    • serviced by Apache HTTPD
  • we have an Artifactory instance used by all users -- lets call this NEWR
    • we have a Remote Repository defined which points to OLDR
Now. If I have in OLDR: com.test:foo versions 1.1, 1.2, 1.3. 2.0 (with correct maven-metadata.xml). In NEWR, I have: com.test:foo versions 2.1, 3.0, 3.1. - all produced with maven release plugin.

What if I ask for: http://NEWR/repo/com/test/foo/maven-metadata.xml what should I get?

Thanks,

Lóránd Somogyi
http://lorands.com
http://photols.com

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

Re: maven-metadata.xml aggregation behavior

by Yoav Landman-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Merged maven-metadata.xml containing all versions reachable from the virtual repository 'http://NEWR/repo' - from both NEWR and OLDR (assuming the user making the request has enough permissions to populate artifacts in the OLDR cache).

On Tue, Oct 20, 2009 at 10:07 AM, Lóránd Somogyi <lorand.somogyi@...> wrote:
HI!

I'm a bit confused, so please help me understand the intention...

A little bit of background:
  • we have one simple repository (a'la repo1.maven.org) for older releases -- let's call this one OLDR
    • in simple folder/file structure
    • with generated maven-metadata.xml
    • serviced by Apache HTTPD
  • we have an Artifactory instance used by all users -- lets call this NEWR
    • we have a Remote Repository defined which points to OLDR
Now. If I have in OLDR: com.test:foo versions 1.1, 1.2, 1.3. 2.0 (with correct maven-metadata.xml). In NEWR, I have: com.test:foo versions 2.1, 3.0, 3.1. - all produced with maven release plugin.

What if I ask for: http://NEWR/repo/com/test/foo/maven-metadata.xml what should I get?

Thanks,

Lóránd Somogyi
http://lorands.com
http://photols.com

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users



------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

Re: maven-metadata.xml aggregation behavior

by Lorand :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Thx, Yoav!

In that case there is a bug in 2.1 (may affect other versions too):

The case:
  1. NOone accessed com.test:foo at all in the past (from NEWR)
  2. there is NO com.test:foo in NEWR
  3. there are com.test:foo versions with metadata in OLDR
  4. release of com.test:foo happens (to NEWR)
  5. accessing http://NEWR/repo/com/test/foo/maven-metadata.xml returns ONLY the NEW foo version
Fix by hand:
  1. Access http://NEWR/oldr/com/test/foo/maven-metadata.xml  -- assuming ID of Remote Repo to OLDR is oldr
  2. Now accessing http://NEWR/repo/com/test/foo/maven-metadata.xml returns ALL version
It looks like if there is no information about other metadata new release shades all other metadata.

It would be nice if there would be a way to force "recalculation" of metadata.

Regards,

Lóránd Somogyi
http://lorands.com
http://photols.com


On Tue, Oct 20, 2009 at 3:44 PM, Yoav Landman <yoavl@...> wrote:
Merged maven-metadata.xml containing all versions reachable from the virtual repository 'http://NEWR/repo' - from both NEWR and OLDR (assuming the user making the request has enough permissions to populate artifacts in the OLDR cache).

On Tue, Oct 20, 2009 at 10:07 AM, Lóránd Somogyi <lorand.somogyi@...> wrote:
HI!

I'm a bit confused, so please help me understand the intention...

A little bit of background:
  • we have one simple repository (a'la repo1.maven.org) for older releases -- let's call this one OLDR
    • in simple folder/file structure
    • with generated maven-metadata.xml
    • serviced by Apache HTTPD
  • we have an Artifactory instance used by all users -- lets call this NEWR
    • we have a Remote Repository defined which points to OLDR
Now. If I have in OLDR: com.test:foo versions 1.1, 1.2, 1.3. 2.0 (with correct maven-metadata.xml). In NEWR, I have: com.test:foo versions 2.1, 3.0, 3.1. - all produced with maven release plugin.

What if I ask for: http://NEWR/repo/com/test/foo/maven-metadata.xml what should I get?

Thanks,

Lóránd Somogyi
http://lorands.com
http://photols.com

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users



------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users



------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

Re: maven-metadata.xml aggregation behavior

by Lorand :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Hi!

Any reaction to this?
Shell I fill an issue to jira?

Regards,

Lóránd Somogyi
http://lorands.com
http://photols.com


On Tue, Oct 20, 2009 at 3:40 PM, Lóránd Somogyi <lorand.somogyi@...> wrote:
Thx, Yoav!

In that case there is a bug in 2.1 (may affect other versions too):

The case:
  1. NOone accessed com.test:foo at all in the past (from NEWR)
  2. there is NO com.test:foo in NEWR
  3. there are com.test:foo versions with metadata in OLDR
  4. release of com.test:foo happens (to NEWR)
  5. accessing http://NEWR/repo/com/test/foo/maven-metadata.xml returns ONLY the NEW foo version
Fix by hand:
  1. Access http://NEWR/oldr/com/test/foo/maven-metadata.xml  -- assuming ID of Remote Repo to OLDR is oldr
  2. Now accessing http://NEWR/repo/com/test/foo/maven-metadata.xml returns ALL version
It looks like if there is no information about other metadata new release shades all other metadata.

It would be nice if there would be a way to force "recalculation" of metadata.

Regards,
On Tue, Oct 20, 2009 at 3:44 PM, Yoav Landman <yoavl@...> wrote:
Merged maven-metadata.xml containing all versions reachable from the virtual repository 'http://NEWR/repo' - from both NEWR and OLDR (assuming the user making the request has enough permissions to populate artifacts in the OLDR cache).

On Tue, Oct 20, 2009 at 10:07 AM, Lóránd Somogyi <lorand.somogyi@...> wrote:
HI!

I'm a bit confused, so please help me understand the intention...

A little bit of background:
  • we have one simple repository (a'la repo1.maven.org) for older releases -- let's call this one OLDR
    • in simple folder/file structure
    • with generated maven-metadata.xml
    • serviced by Apache HTTPD
  • we have an Artifactory instance used by all users -- lets call this NEWR
    • we have a Remote Repository defined which points to OLDR
Now. If I have in OLDR: com.test:foo versions 1.1, 1.2, 1.3. 2.0 (with correct maven-metadata.xml). In NEWR, I have: com.test:foo versions 2.1, 3.0, 3.1. - all produced with maven release plugin.

What if I ask for: http://NEWR/repo/com/test/foo/maven-metadata.xml what should I get?

Thanks,

Lóránd Somogyi
http://lorands.com
http://photols.com

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users



------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users




------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

Re: maven-metadata.xml aggregation behavior

by Yoav Landman-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Artifactory's behavior is to never check for remote metadata for
release artifacts hosted in local repositories. The reason is simple -
typically what you host locally is your own artifacts, so you do not
want to make requests to remote repositories and scan for the
existence of other versions of artifacts that you specifically
deployed locally. We do, however, merge the metadata with caches since
it is cheap and doesn't require the unnecessary network traffic. This
is why it works if you deliberately cause the remote metadata to be
locally cached by requesting it directly from the remote repository.

Now, I can see how this optimization may be problematic if you wish to
get all possible remote versions of artifacts you have both locally
and also in other repositories in your organization that you proxy.
What we can do is make this type of metadata requests possible and let
users control the behavior and prevent inefficient queries to all
remote repositories by using proper include/exclude patterns.

I created http://issues.jfrog.org/jira/browse/RTFACT-2311.

Yoav

On Wed, Nov 4, 2009 at 4:01 PM, Lóránd Somogyi <lorand.somogyi@...> wrote:

> Hi!
>
> Any reaction to this?
> Shell I fill an issue to jira?
>
> Regards,
>
> Lóránd Somogyi
> http://lorands.com
> http://photols.com
>
>
> On Tue, Oct 20, 2009 at 3:40 PM, Lóránd Somogyi <lorand.somogyi@...>
> wrote:
>>
>> Thx, Yoav!
>>
>> In that case there is a bug in 2.1 (may affect other versions too):
>>
>> The case:
>>
>> NOone accessed com.test:foo at all in the past (from NEWR)
>> there is NO com.test:foo in NEWR
>> there are com.test:foo versions with metadata in OLDR
>> release of com.test:foo happens (to NEWR)
>> accessing http://NEWR/repo/com/test/foo/maven-metadata.xml returns ONLY
>> the NEW foo version
>>
>> Fix by hand:
>>
>> Access http://NEWR/oldr/com/test/foo/maven-metadata.xml  -- assuming ID of
>> Remote Repo to OLDR is oldr
>> Now accessing http://NEWR/repo/com/test/foo/maven-metadata.xml returns ALL
>> version
>>
>> It looks like if there is no information about other metadata new release
>> shades all other metadata.
>>
>> It would be nice if there would be a way to force "recalculation" of
>> metadata.
>>
>> Regards,
>>
>> Lóránd Somogyi
>> http://lorands.com
>> http://photols.com
>>
>>
>> On Tue, Oct 20, 2009 at 3:44 PM, Yoav Landman <yoavl@...> wrote:
>>>
>>> Merged maven-metadata.xml containing all versions reachable from the
>>> virtual repository 'http://NEWR/repo' - from both NEWR and OLDR (assuming
>>> the user making the request has enough permissions to populate artifacts in
>>> the OLDR cache).
>>>
>>> On Tue, Oct 20, 2009 at 10:07 AM, Lóránd Somogyi
>>> <lorand.somogyi@...> wrote:
>>>>
>>>> HI!
>>>>
>>>> I'm a bit confused, so please help me understand the intention...
>>>>
>>>> A little bit of background:
>>>>
>>>> we have one simple repository (a'la repo1.maven.org) for older releases
>>>> -- let's call this one OLDR
>>>>
>>>> in simple folder/file structure
>>>> with generated maven-metadata.xml
>>>> serviced by Apache HTTPD
>>>>
>>>> we have an Artifactory instance used by all users -- lets call this NEWR
>>>>
>>>> we have a Remote Repository defined which points to OLDR
>>>>
>>>> Now. If I have in OLDR: com.test:foo versions 1.1, 1.2, 1.3. 2.0 (with
>>>> correct maven-metadata.xml). In NEWR, I have: com.test:foo versions 2.1,
>>>> 3.0, 3.1. - all produced with maven release plugin.
>>>>
>>>> What if I ask for: http://NEWR/repo/com/test/foo/maven-metadata.xml what
>>>> should I get?
>>>>
>>>> Thanks,
>>>>
>>>> Lóránd Somogyi
>>>> http://lorands.com
>>>> http://photols.com
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
>>>> is the only developer event you need to attend this year. Jumpstart your
>>>> developing skills, take BlackBerry mobile applications to market and
>>>> stay
>>>> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
>>>> http://p.sf.net/sfu/devconference
>>>> _______________________________________________
>>>> Artifactory-users mailing list
>>>> Artifactory-users@...
>>>> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
>>> is the only developer event you need to attend this year. Jumpstart your
>>> developing skills, take BlackBerry mobile applications to market and stay
>>> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
>>> http://p.sf.net/sfu/devconference
>>> _______________________________________________
>>> Artifactory-users mailing list
>>> Artifactory-users@...
>>> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>>>
>>
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus
> on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Artifactory-users mailing list
> Artifactory-users@...
> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>
>

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

Re: maven-metadata.xml aggregation behavior

by Lorand :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Great to hear that!
I see it's already done and will be included in 2.1.4.

Lóránd


On Thu, Nov 5, 2009 at 11:58 AM, Yoav Landman <yoavl@...> wrote:
Artifactory's behavior is to never check for remote metadata for
release artifacts hosted in local repositories. The reason is simple -
typically what you host locally is your own artifacts, so you do not
want to make requests to remote repositories and scan for the
existence of other versions of artifacts that you specifically
deployed locally. We do, however, merge the metadata with caches since
it is cheap and doesn't require the unnecessary network traffic. This
is why it works if you deliberately cause the remote metadata to be
locally cached by requesting it directly from the remote repository.

Now, I can see how this optimization may be problematic if you wish to
get all possible remote versions of artifacts you have both locally
and also in other repositories in your organization that you proxy.
What we can do is make this type of metadata requests possible and let
users control the behavior and prevent inefficient queries to all
remote repositories by using proper include/exclude patterns.

I created http://issues.jfrog.org/jira/browse/RTFACT-2311.

Yoav

On Wed, Nov 4, 2009 at 4:01 PM, Lóránd Somogyi <lorand.somogyi@...> wrote:
> Hi!
>
> Any reaction to this?
> Shell I fill an issue to jira?
>
> Regards,
>
> Lóránd Somogyi
> http://lorands.com
> http://photols.com
>
>
> On Tue, Oct 20, 2009 at 3:40 PM, Lóránd Somogyi <lorand.somogyi@...>
> wrote:
>>
>> Thx, Yoav!
>>
>> In that case there is a bug in 2.1 (may affect other versions too):
>>
>> The case:
>>
>> NOone accessed com.test:foo at all in the past (from NEWR)
>> there is NO com.test:foo in NEWR
>> there are com.test:foo versions with metadata in OLDR
>> release of com.test:foo happens (to NEWR)
>> accessing http://NEWR/repo/com/test/foo/maven-metadata.xml returns ONLY
>> the NEW foo version
>>
>> Fix by hand:
>>
>> Access http://NEWR/oldr/com/test/foo/maven-metadata.xml  -- assuming ID of
>> Remote Repo to OLDR is oldr
>> Now accessing http://NEWR/repo/com/test/foo/maven-metadata.xml returns ALL
>> version
>>
>> It looks like if there is no information about other metadata new release
>> shades all other metadata.
>>
>> It would be nice if there would be a way to force "recalculation" of
>> metadata.
>>
>> Regards,
>>
>> Lóránd Somogyi
>> http://lorands.com
>> http://photols.com
>>
>>
>> On Tue, Oct 20, 2009 at 3:44 PM, Yoav Landman <yoavl@...> wrote:
>>>
>>> Merged maven-metadata.xml containing all versions reachable from the
>>> virtual repository 'http://NEWR/repo' - from both NEWR and OLDR (assuming
>>> the user making the request has enough permissions to populate artifacts in
>>> the OLDR cache).
>>>
>>> On Tue, Oct 20, 2009 at 10:07 AM, Lóránd Somogyi
>>> <lorand.somogyi@...> wrote:
>>>>
>>>> HI!
>>>>
>>>> I'm a bit confused, so please help me understand the intention...
>>>>
>>>> A little bit of background:
>>>>
>>>> we have one simple repository (a'la repo1.maven.org) for older releases
>>>> -- let's call this one OLDR
>>>>
>>>> in simple folder/file structure
>>>> with generated maven-metadata.xml
>>>> serviced by Apache HTTPD
>>>>
>>>> we have an Artifactory instance used by all users -- lets call this NEWR
>>>>
>>>> we have a Remote Repository defined which points to OLDR
>>>>
>>>> Now. If I have in OLDR: com.test:foo versions 1.1, 1.2, 1.3. 2.0 (with
>>>> correct maven-metadata.xml). In NEWR, I have: com.test:foo versions 2.1,
>>>> 3.0, 3.1. - all produced with maven release plugin.
>>>>
>>>> What if I ask for: http://NEWR/repo/com/test/foo/maven-metadata.xml what
>>>> should I get?
>>>>
>>>> Thanks,
>>>>
>>>> Lóránd Somogyi
>>>> http://lorands.com
>>>> http://photols.com
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
>>>> is the only developer event you need to attend this year. Jumpstart your
>>>> developing skills, take BlackBerry mobile applications to market and
>>>> stay
>>>> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
>>>> http://p.sf.net/sfu/devconference
>>>> _______________________________________________
>>>> Artifactory-users mailing list
>>>> Artifactory-users@...
>>>> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
>>> is the only developer event you need to attend this year. Jumpstart your
>>> developing skills, take BlackBerry mobile applications to market and stay
>>> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
>>> http://p.sf.net/sfu/devconference
>>> _______________________________________________
>>> Artifactory-users mailing list
>>> Artifactory-users@...
>>> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>>>
>>
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus
> on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Artifactory-users mailing list
> Artifactory-users@...
> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>
>

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users