Ivy resolving wildcard revisions w/ Artifactory

[Carl Quinn]

I am having trouble similar to what topher1120 describes in his Aug 10, 2009 post, but maybe even more fundamental.

I am having a hard time getting Ivy to resolve artifacts in Artifactory when the dependency spec uses wildcards or ranges in the revision (version). When the revisions are concrete, resolution works fine, but when they are not, the artifacts are just not found.

I have a bunch of 3rd party artifacts in the repository with corresponding Ivy files. Like:

  commons-collections/commons-collections/2.1.1/commons-collections-2.1.1.jar
  commons-collections/commons-collections/2.1.1/ivy-2.1.1.xml
  commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar
  commons-collections/commons-collections/3.2.1/ivy-3.2.1.xml

I have tried both ibiblio and url resolvers, with pretty much every combination of flags.

I guess I don't quite have a complete understanding of how this is supposed to work. That is, how can Ivy query/enumerate the available versions of the module using a URL?

When I throw these artifacts and module descriptors onto the disk and use a filesystem resolver, it works fine.

Any ideas? Suggestions for good reading would be appreciated too. It's hard to find good info on Ivy + Artifactory together.

[Yoav Landman-3]

Hi,

For HTTP-based repositories it looks like Ivy is doing HTML pattern-matching to figure out the available versions. This is done by requesting the path of the group, then Ivy is looking for suitable href tags in the markup of the response.

We have tried (programmaticaly) to get the list of versions with both Ivy 2.0.0 and 2.1.0-rc2, and found out it does indeed break in 2.0.0, but works fine in 2.1.0-rc2 (tested with the trunk of Artifactory 2.1.0, but I expect the behavior to be the same with 2.0.8). This is probably related to http://issues.apache.org/jira/browse/IVY-959.

So, if you are using Ivy 2.0.0, we expect Ivy 2.1.0-rc1 and above to be able to successfully resolve dynamic versions with Artifactory.

Hope it helps,

Yoav

On Fri, Sep 11, 2009 at 7:08 AM, cquinn <carl.quinn@...> wrote:

I am having trouble similar to what topher1120 describes in his Aug 10, 2009
post, but maybe even more fundamental.

I am having a hard time getting Ivy to resolve artifacts in Artifactory when
the dependency spec uses wildcards or ranges in the revision (version). When
the revisions are concrete, resolution works fine, but when they are not,
the artifacts are just not found.

I have a bunch of 3rd party artifacts in the repository with corresponding
Ivy files. Like:


commons-collections/commons-collections/2.1.1/commons-collections-2.1.1.jar
 commons-collections/commons-collections/2.1.1/ivy-2.1.1.xml

commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar
 commons-collections/commons-collections/3.2.1/ivy-3.2.1.xml

I have tried both ibiblio and url resolvers, with pretty much every
combination of flags.

I guess I don't quite have a complete understanding of how this is supposed
to work. That is, how can Ivy query/enumerate the available versions of the
module using a URL?

When I throw these artifacts and module descriptors onto the disk and use a
filesystem resolver, it works fine.

Any ideas? Suggestions for good reading would be appreciated too. It's hard
to find good info on Ivy + Artifactory together.
--
View this message in context: http://www.nabble.com/Ivy-resolving-wildcard-revisions-w--Artifactory-tp25394756p25394756.html
Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

[Carl Quinn]

I am using Ivy 2.1.0-rc2 with Artifactory 2.0.7. Do you think Artifactory 2.0.8 would be any different? Or should I even try out the 2.1.0 trunk?

Also, what Ivy resolvers & settings did you find successful with your programatic tests?

I am a little surprised that there isn't a more rest-full (non-html) way for Ivy to talk to the repository.

--carl

Yoav Landman-3 wrote:

Hi,

For HTTP-based repositories it looks like Ivy is doing HTML pattern-matching
to figure out the available versions. This is done by requesting the path of
the group, then Ivy is looking for suitable href tags in the markup of the
response.

We have tried (programmaticaly) to get the list of versions with both Ivy
2.0.0 and 2.1.0-rc2, and found out it does indeed break in 2.0.0, but works
fine in 2.1.0-rc2 (tested with the trunk of Artifactory 2.1.0, but I expect
the behavior to be the same with 2.0.8). This is probably related to
http://issues.apache.org/jira/browse/IVY-959.

So, if you are using Ivy 2.0.0, we expect Ivy 2.1.0-rc1 and above to be able
to successfully resolve dynamic versions with Artifactory.

Hope it helps,

Yoav

[Yoav Landman-3]

On Sun, Sep 13, 2009 at 12:40 AM, Carl Quinn <carl.quinn@...> wrote:

I am using Ivy 2.1.0-rc2 with Artifactory 2.0.7. Do you think Artifactory
2.0.8 would be any different? Or should I even try out the 2.1.0 trunk?

I guess my first question should have been what Artifactory version you are using :)
 

Also, what Ivy resolvers & settings did you find successful with your
programatic tests?

We tested programatically resolving the correct list of versions (assuming that when this work the Ivy client will take care of the rest). We used the URL resolver in m2-compatible mode and default Ivy settings. We tried both virtual repo URLs and local repo URLs as the repo root.
We already tested both 2.0.7 and 2.0.8 with the same test case and it worked fine.
 

I am a little surprised that there isn't a more rest-full (non-html) way for
Ivy to talk to the repository.

Actually, the ibiblio resolver uses the maven-metadata.xml by default to get the list of versions, which can be seen as a more restful way.

--carl

 

Since getting the list of versions worked fine in all our tests, we assumed that once Ivy is getting back a correct list of versions from the repository something else goes wrong, so we also ran a full standalone Ivy test, with the following settings:

<ivysettings>
    <settings defaultResolver="url"/>
    <resolvers>
        <url name="url" m2compatible="true">
          <artifact pattern="${test.repoRoot}/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
        </url>
    </resolvers>
</ivysettings>

We used test.repoRoot = http://server:8081/artifactory/repo

And ran something like:
java -jar ivy.jar -settings <the-above-ivysettings-file> -dependency mysql connector-java [5.1.0,5.2.0]

And that also worked as expected, resolving the correct artifact version.

Do you have a particular version pattern that fails for you that we can try out?

Yoav

Yoav Landman-3 wrote:
>
> Hi,
>
> For HTTP-based repositories it looks like Ivy is doing HTML
> pattern-matching
> to figure out the available versions. This is done by requesting the path
> of
> the group, then Ivy is looking for suitable href tags in the markup of the
> response.
>
> We have tried (programmaticaly) to get the list of versions with both Ivy
> 2.0.0 and 2.1.0-rc2, and found out it does indeed break in 2.0.0, but
> works
> fine in 2.1.0-rc2 (tested with the trunk of Artifactory 2.1.0, but I
> expect
> the behavior to be the same with 2.0.8). This is probably related to
> http://issues.apache.org/jira/browse/IVY-959.
>
> So, if you are using Ivy 2.0.0, we expect Ivy 2.1.0-rc1 and above to be
> able
> to successfully resolve dynamic versions with Artifactory.
>
> Hope it helps,
>
> Yoav
>
>

--
View this message in context: http://www.nabble.com/Ivy-resolving-wildcard-revisions-w--Artifactory-tp25394756p25418461.html

Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

[Carl Quinn]

Yoav Landman-3 wrote:

We tested programatically resolving the correct list of versions (assuming
that when this work the Ivy client will take care of the rest). We used the
URL resolver in m2-compatible mode and default Ivy settings. We tried both
virtual repo URLs and local repo URLs as the repo root.
We already tested both 2.0.7 and 2.0.8 with the same test case and it worked
fine.

OK, very good. Then I should be able to achieve the same results with my current installation.

Yoav Landman-3 wrote:

Actually, the ibiblio resolver uses the maven-metadata.xml by default to get
the list of versions, which can be seen as a more restful way.

I see. That is a nicer way to get the aggregated metadata than scraping the html, but it seems that maven-metadata.xml must be generated externally to Artifactory and uploaded. Am I wrong? Will it be synthesized on the fly if it is requested?
 

Yoav Landman-3 wrote:

Since getting the list of versions worked fine in all our tests, we assumed
that once Ivy is getting back a correct list of versions from the repository
something else goes wrong, so we also ran a full standalone Ivy test, with
the following settings:

<ivysettings>
    <settings defaultResolver="url"/>
    <resolvers>
        <url name="url" m2compatible="true">
          <artifact
pattern="${test.repoRoot}/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"
/>
        </url>
    </resolvers>
</ivysettings>

We used test.repoRoot = http://server:8081/artifactory/repo

And ran something like:
java -jar ivy.jar -settings <the-above-ivysettings-file> -dependency mysql
connector-java [5.1.0,5.2.0]

And that also worked as expected, resolving the correct artifact version.

Do you have a particular version pattern that fails for you that we can try
out?

I will try out your test-mechanism to figure out where I am breaking. The patterns that I am having trouble with mostly seem to have +'s in them, and exclusive range end values. Like:

        :: commons-collections#commons-collections;3.2.1+: not found
        :: commons-collections#commons-collections;3.2.+: not found
        :: commons-logging#commons-logging;1.1.+: not found
        :: commons-logging#commons-logging;1.+: not found
        :: commons-collections#commons-collections;[2.+,3.2.+): not found
        :: commons-logging#commons-logging;[1.1,2.0): not found

When commons-collections has versions 2.1.1 and 3.2.1 in the repo, and commons-logging has v. 1.1.1, which should match all the above patterns. I think. They do, at least, when I resolve with a filesystem resolver:

   [3.2.1] commons-collections#commons-collections;[2.+,3.2.+)
   [1.1.1] commons-logging#commons-logging;[1.1,2.0)

thanks,
--carl

[freddy33]

Great, we will try your patterns soon.
For the maven-metadata.xml, you don't have to worry since Artifactory is generating it depending on what is deployed locally and remotely. When you request a maven-metadata.xml on a virtual repository of Artifactory, it returns the aggregated list of versions from the local and remote repositories contained in this virtual repository.

Conclusion: Don't need to do anything :)

On Wed, Sep 16, 2009 at 12:06 AM, Carl Quinn <carl.quinn@...> wrote:

Yoav Landman-3 wrote:
>
> We tested programatically resolving the correct list of versions (assuming
> that when this work the Ivy client will take care of the rest). We used
> the
> URL resolver in m2-compatible mode and default Ivy settings. We tried both
> virtual repo URLs and local repo URLs as the repo root.
> We already tested both 2.0.7 and 2.0.8 with the same test case and it
> worked
> fine.
>

OK, very good. Then I should be able to achieve the same results with my
current installation.

Yoav Landman-3 wrote:
>
> Actually, the ibiblio resolver uses the maven-metadata.xml by default to
> get
> the list of versions, which can be seen as a more restful way.
>

I see. That is a nicer way to get the aggregated metadata than scraping the
html, but it seems that maven-metadata.xml must be generated externally to
Artifactory and uploaded. Am I wrong? Will it be synthesized on the fly if
it is requested?

Yoav Landman-3 wrote:
>
> Since getting the list of versions worked fine in all our tests, we
> assumed
> that once Ivy is getting back a correct list of versions from the
> repository
> something else goes wrong, so we also ran a full standalone Ivy test, with
> the following settings:
>
> <ivysettings>
>     <settings defaultResolver="url"/>
>     <resolvers>
>         <url name="url" m2compatible="true">
>           <artifact
> pattern="${test.repoRoot}/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"
> />
>         </url>
>     </resolvers>
> </ivysettings>
>
> We used test.repoRoot = http://server:8081/artifactory/repo
>
> And ran something like:
> java -jar ivy.jar -settings <the-above-ivysettings-file> -dependency mysql
> connector-java [5.1.0,5.2.0]
>
> And that also worked as expected, resolving the correct artifact version.
>
> Do you have a particular version pattern that fails for you that we can
> try
> out?
>

I will try out your test-mechanism to figure out where I am breaking. The
patterns that I am having trouble with mostly seem to have +'s in them, and
exclusive range end values. Like:

       :: commons-collections#commons-collections;3.2.1+: not found
       :: commons-collections#commons-collections;3.2.+: not found
       :: commons-logging#commons-logging;1.1.+: not found
       :: commons-logging#commons-logging;1.+: not found
       :: commons-collections#commons-collections;[2.+,3.2.+): not found
       :: commons-logging#commons-logging;[1.1,2.0): not found

When commons-collections has versions 2.1.1 and 3.2.1 in the repo, and
commons-logging has v. 1.1.1, which should match all the above patterns. I
think. They do, at least, when I resolve with a filesystem resolver:

  [3.2.1] commons-collections#commons-collections;[2.+,3.2.+)
  [1.1.1] commons-logging#commons-logging;[1.1,2.0)

thanks,
--carl

--
View this message in context: http://www.nabble.com/Ivy-resolving-wildcard-revisions-w--Artifactory-tp25394756p25461826.html

Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------

Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf

_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

--
JFrog Ltd
5 Habonim st., P.O.Box 8187
Netanya, Israel 42504.
Tel: [phone]    
Fax: [phone]
http://www.jfrog.org/
http://freddy33.blogspot.com/
http://nothingisinfinite.blogspot.com/

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

[Carl Quinn]

I see the same problems with wildcards using the command-line technique:

With an ivysettings.xml like:

<ivysettings>
    <settings defaultResolver="url"/><!-- "ext" -->
    <resolvers>
        <url name="url" m2compatible="true">
            <artifact pattern="http://artifacts.netflix.com/ext-release-local/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
        </url>
        <filesystem name="ext" m2compatible="false" descriptor="optional" checkconsistency="false">
            <artifact pattern="${ext.libs}/[artifact]-[revision].[ext]" />
        </filesystem>
    </resolvers>
</ivysettings>

And ${ext.libs} point to a dir full of jars and -ivy.xml files like: commons-collections-3.2.1.jar.

Command:
  java -jar ivy-2.1.0-rc2.jar -settings ivysettings.xml -dependency commons-collections commons-collections '[3.2,4.0)'

Yields:
                module not found: commons-collections#commons-collections;[3.2,4.0)

But using the "ext" resolver yields:

        found commons-collections#commons-collections;3.2.1 in ext
        [3.2.1] commons-collections#commons-collections;[3.2,4.0)

[Carl Quinn]

Hi Fred!

I think the problem is with my iptables/Tomcat setup. This trace points it out:

        using url to list all in http://artifacts/nfrepo/commons-collections/commons-collections/
                found 0 resources
        found revs: []

I'm playing some games with iptables on that box to make Artifactory appear on port 80, but then it thinks (knows) it's on 8080 and the returned urls are under artifacts:8080/ which doesn't route. I think that's my trouble.

  HTTP response status: 404 url=http://artifacts:8080/nfrepo/commons-collections/commons-collections/
  CLIENT ERROR: Not Found url=http://artifacts:8080/nfrepo/commons-collections/commons-collections/
  problem while listing resources in http://artifacts:8080/nfrepo/commons-collections/commons-collections/ with url:
    java.io.IOException The HTTP response code for http://artifacts:8080/nfrepo/commons-  collections/commons-collections/ did not indicate a success. See log for more detail.

Thanks for your pointers!

[Yoav Landman-3]

The bright side is, we now have the infra for running Ivy-specific tests as part of our integration tests :)

Please let us know if you find out otherwise.

Thanks,

Yoav

On Wed, Sep 16, 2009 at 1:01 AM, Carl Quinn <carl.quinn@...> wrote:

Hi Fred!

I think the problem is with my iptables/Tomcat setup. This trace points it
out:

       using url to list all in
http://artifacts/nfrepo/commons-collections/commons-collections/
               found 0 resources
       found revs: []

I'm playing some games with iptables on that box to make Artifactory appear
on port 80, but then it thinks (knows) it's on 8080 and the returned urls
are under artifacts:8080/ which doesn't route. I think that's my trouble.

 HTTP response status: 404
url=http://artifacts:8080/nfrepo/commons-collections/commons-collections/
 CLIENT ERROR: Not Found
url=http://artifacts:8080/nfrepo/commons-collections/commons-collections/
 problem while listing resources in
http://artifacts:8080/nfrepo/commons-collections/commons-collections/ with
url:
   java.io.IOException The HTTP response code for
http://artifacts:8080/nfrepo/commons-  collections/commons-collections/ did
not indicate a success. See log for more detail.

Thanks for your pointers!

--
View this message in context: http://www.nabble.com/Ivy-resolving-wildcard-revisions-w--Artifactory-tp25394756p25462631.html

Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------
Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
Artifactory-users mailing list
Artifactory-users@...
https://lists.sourceforge.net/lists/listinfo/artifactory-users