Re: Dependency management

[dmos]

On Wed, Jul 8, 2009 at 11:28 AM, Phil Hagelberg<phil@...> wrote:

>
> I've been noodling on the problem of dependency management for a while
> now. It's definitely a pain point for projects with more than a couple
> dependencies. Currently our approach has been to use maven, but that
> involves a fair amount of arcane knowledge as well as writing a bunch of
> XML, which isn't a lot of fun. But there's been a community consensus
> that any dependency tool needs to be able to leverage all the valuable
> stuff out there that's currently in maven repositories.
>
> I've put together a proof-of-concept dependency manager called
> Corkscrew. It uses Maven under the hood as well as offering dependencies
> on source packages stored in git or subversion. A simple project.clj
> file lays out the dependencies and other project info:
>
>    {:name "my-sample"
>     :version "1.0"
>     :dependencies [["tagsoup" "1.2" "org.ccil.cowan.tagsoup"]
>                    ;; group defaults to name
>                    ["rome" "0.9"]]
>     :source-dependencies [["clojure-contrib" "r663" :svn
>                            "http://clojure-contrib.googlecode.com/svn/trunk"]
>                           ["enlive" "95b2558943f50bb9962fe7d500ede353f1b578f0"
>                            :git "git://github.com/cgrand/enlive.git"]]}
>
> You can install it with:
>
>  $ git clone git://github.com/technomancy/corkscrew.git
>  $ cd corkscrew
>  $ ./install
>  $ cp bin/corkscrew /somewhere/on/your/path
>
> Create a project.clj file in your project root based on the one
> above. Then you can use "corkscrew deps" to set everything up for
> you. At that point just make sure your classpath includes
> target/dependency/ and you should be good to go.
>
> It's simple, but it solves the main problems that I've been having with
> more complicated projects. I'd love to get some opinions on it.
>
> The biggest issue right now is that it runs Maven as a subprocess rather
> than using the Java API in the same VM because I can't make head or tail
> of the Maven Java API (it uses plexus.core), but shelling out works as a
> proof-of-concept even if it's tacky.
>
> -Phil
>

Thanks for bringing that up, as it's been bugging me for a some time
now (and violently ejected me out of lurk mode on the list). What
follows is a pretty long essay on the topic, I hope you stay with me,
and take part in the debate, because I think the availability of a
good buildtool/dependency manager is absolutely crucial to use Clojure
in larger projects, and as such for adoption on a wider scale.

I have basically the same problem as you in that Clojure is lacking a
proper buildtool (i.e. one that can replace Maven). I used Maven for a
long time and lived with it's strengths and weaknesses. It's far from
perfect, but Maven does still give you quite a lot:
* a dependency resolution and publishing model for generated artifacts
* a build based on conventions (can be overridden, but Maven pushes
you hard to do things in a de-facto standard way)
* a defined execution model (the phases thing. Not always nice, and
not always sufficient, but at least something)
* a defined API to hook into with your plugins (API design is so-so,
and documentation is IMHO abysmal, but ...)
* a ton of reports and plugins to extend the model (not always
perfect, and configuration is not nice, but they usually do what they
advertise)
* more?

On the other hand, some things Maven are always a pain in the back:
figuring out how to configure plugins, writing plugins because it's
the only way to do something non-standard in Maven, and
debugging/finding out what the heck is going on, if some plugin is not
doing what it should do... That's the arcane knowledge part.

But recently I had the need for a more flexible buildtool that would
allow me to build non-standard JVM based languages in a non-standard
fashion and still play nice. Case in point was that I had to marry two
different dependency resolution systems (Maven style, and
Jython/Python style), and then build a mix of code in
Scala/Java/Jython. The experience was none too pretty with Maven.

Most of the buildtools out there for JVM related stuff don't try to
compete with Maven and all it's plugins and reports, but they start by
replacing/wrapping Ant (and usually Ivy for dependency management).
That description applies to Buildr, the Groovy Ant Builder, Gant,
Grape (which is a wrapper for Ivy), SBT (Simple Build Tool - Scala),
Scalr (Scala), Lancet, and probably some more that I missed. The only
notable exception I could dig up was Gradle.

Gradle (http://gradle.org/) turns out to be very interesting. Here are
some propaganda points (not exhaustive):
* Although it currently uses mainly a Groovy DSL, the Gradle core is
written in Java, and there's an explicit goal on the roadmap to
support DSLs written in other JVM languages on top of it as to make it
an attractive choice for everybody. For Scala there's a patch in the
Bugtracker to add a Scala DSL on top. Other languages I'm unaware of.
* Convention based builds. They have taken care that they appeal to
Maven developers (all the conventions of the Java Plugin are the same
ones as in Maven), while still leaving it flexible through
configuration that every project layout can be built.
* Hierarchical, dependency based builds.
* A build is executed in two major phases (configuration and
execution), and tasks are allowed to hook into both (a task can make
it's execution dependent on whether another task is scheduled to run
*after* itself (try that with Maven))
* A build graph containing all the tasks for execution can be accessed
and manipulated programmatically.
* Gradle defines it's own task API, but provides an Ant wrapper to
give you an easy migration route or extension path (similar to Gant,
or Lancet in Clojure). Custom tasks can be easily coded up, although
you need custom things less often than with Maven and Co., because a
lot can be solved with a flexible build script (it's all code).
* The build can be extended from within a project. You don't need a
plugin project that you're only going to use in one project anyway.
There's a separate source tree in your project home that hosts
extension to your build.
* Dependency resolution is done with Ivy. But they took care to clean
the Ivy model a bit up so that it provides better support for
convention based builds.
* They have an integration layer with Maven. I don't know how far it
actually goes, but I know that you can call through to Maven for some
things (MavenPlugin).

Non-technical
* The mailinglist is very active, and receptive to input that comes
from the outside. They have github mirrors, pull requests can be
issued easily.
* The project is well setup: toplevel domain, liberal license (Apache
v2), plethora of SCM can be used (master is on subversion, mirrors on
github (git), and launchpad (bzr)) issue tracker (JIRA), wiki
(Confluence), public roadmap, continuous integration (bamboo &
teamcity, platform specific builds), selfhosting (gradle is built by
gradle), documentation is autogenerated during the build (including
the userguide HTML and PDFs).
* The guys working on Gradle have been around the build tool circus
for a long time apparently (five committers, among them: the founder
of JBoss IDE, and the author of Gant).
* The team is dedicated enough to evangelize their product largescale,
including talks on conferences around the globe, and at least one of
the participants runs a consultancy providing advice on buildsystems.
* Reasonably complete documentation (there's arguably room for
improvement, especially in structure, but you can find almost anything
in the userguide).
* They are willing to change stuff in how it works internally if
there's technical merit.
* It's developing fast, and I have the feeling that it's getting
increasingly powerful and flexible.
* It's not very complicated IMHO. It's probably slightly more
complicated than ant + sugar coating, but they Gradle tagline is:
"make the impossible possible, the possible easy, and the easy
elegant."  (Moshé Feldenkrais), and I think they are definitely within
scope.


Ok, now the question why am I pushing this so hard? Because have been
annoyed at the exact same things in every language that I tried for
the JVM. Maven is not very good but it's usually the best thing you
can get. And although there's a lot of competition for build tools,
nobody seems to try to tackle what Maven solved, and only goes for a
simpler subset of the problems. Usually it goes something like: "Our
language is more productive than Java and we don't like Maven, because
it's a pain to use. Ant is nice but uses XML so we can't use it out of
the box. We can't really wrap Maven, so let's wrap Ant, and since we
are lacking dependency management in that case, let's also wrap Ivy,
add a little bit of sugar on top, and voila, new shiny build system."
Except that everybody now has to learn a gazillion new (and different)
tools if he wants to build stuff easily in a certain language, and
beware if you try to do something cross language (a problem that will
become more common with alternative languages for the JVM on the
rise).

I would very much like to see that people don't suffer the NIH
syndrome everywhere. Usually people forget that every alternative
technology implemented has an associated cost that everyone looking
for a solution has to pay, because I have to consciously discard the
tool as insufficient, and that means I have to acquire enough
knowledge about each tool to make an informed decision (the paradox of
choice). I cannot even imagine how much time has been wasted by
halfbaked (and partially or fully abandoned) projects.

If the miracle would happen and the different, rapidly growing
language communities on the JVM finally could commit themselves to
support and push a shared buildtool, then that tool would have enough
steam to grow into a decent alternative for Maven. It might even
become the preferred way of build at least non-Java projects, who
knows. If everybody is trying to get their own share of followers and
only solves what is immediately visible for the projects used by
themselves and their followers, then we will never get anywhere.

And please note that I'm not asking to start from scratch. Gradle is
there and has momentum already. I'm just hoping that I'm not the only
one that sees a benefit in joining forces with stuff that already
exists. If people in the Clojure community start to throw your weight
behind something that seems to be on the right track, can still be
influenced and made better.

To get started with Gradle, you would have to:
* write a DSL that can talk to Gradle Java API.
* write a ClojurePlugin (probably based on the JavaPlugin) for Gradle,
if you have specialized wishes (such as that src/main/clojure should
be the default convention for Clojure projects, or that you can
specify which version of Clojure should be used for compilation etc.).

</propaganda>

Sorry for the long mail/rant, but it's been too painful too many
times. And a sincere apology to Phil, who just scratched an itch for
himself, and was interested in sharing it with the larger Clojure
community. This mail is not an attack on your work, or approach. It
just triggered something that I have stumbled over too many times and
I would love to see it finally resolved. I think that Gradle could be
used to give you all you envisioned for Corkscrew, but in the process,
you will gain so much more.

Disclaimer: I'm (currently) not actively developing Gradle itself, but
i was impressed with what's there and where it's going. I also took
the liberty to crosspost this mail to the Gradle dev list (CC'd) to
invite a broader participation.

I'm really hoping that we could find some consensus on the topic.

These are my probably 20$ or so...

Daniel
(probably by now unofficial Gradle Evangelist)

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[hdockter]

Hi Daniel,

the Gradle dev team has synced their holidays :) I'm back in my office
on Friday and I'm looking forward to replying to your email then.

- Hans

On Wed, 08 Jul 2009 12:51 +0700, "Daniel" <dan.in.a.bottle@...>
wrote:

> On Wed, Jul 8, 2009 at 11:28 AM, Phil Hagelberg<phil@...> wrote:
> >
> > I've been noodling on the problem of dependency management for a while
> > now. It's definitely a pain point for projects with more than a couple
> > dependencies. Currently our approach has been to use maven, but that
> > involves a fair amount of arcane knowledge as well as writing a bunch of
> > XML, which isn't a lot of fun. But there's been a community consensus
> > that any dependency tool needs to be able to leverage all the valuable
> > stuff out there that's currently in maven repositories.
> >
> > I've put together a proof-of-concept dependency manager called
> > Corkscrew. It uses Maven under the hood as well as offering dependencies
> > on source packages stored in git or subversion. A simple project.clj
> > file lays out the dependencies and other project info:
> >
> >    {:name "my-sample"
> >     :version "1.0"
> >     :dependencies [["tagsoup" "1.2" "org.ccil.cowan.tagsoup"]
> >                    ;; group defaults to name
> >                    ["rome" "0.9"]]
> >     :source-dependencies [["clojure-contrib" "r663" :svn
> >                            "http://clojure-contrib.googlecode.com/svn/trunk"]
> >                           ["enlive" "95b2558943f50bb9962fe7d500ede353f1b578f0"
> >                            :git "git://github.com/cgrand/enlive.git"]]}
> >
> > You can install it with:
> >
> >  $ git clone git://github.com/technomancy/corkscrew.git
> >  $ cd corkscrew
> >  $ ./install
> >  $ cp bin/corkscrew /somewhere/on/your/path
> >
> > Create a project.clj file in your project root based on the one
> > above. Then you can use "corkscrew deps" to set everything up for
> > you. At that point just make sure your classpath includes
> > target/dependency/ and you should be good to go.
> >
> > It's simple, but it solves the main problems that I've been having with
> > more complicated projects. I'd love to get some opinions on it.
> >
> > The biggest issue right now is that it runs Maven as a subprocess rather
> > than using the Java API in the same VM because I can't make head or tail
> > of the Maven Java API (it uses plexus.core), but shelling out works as a
> > proof-of-concept even if it's tacky.
> >
> > -Phil
> >
>
> Thanks for bringing that up, as it's been bugging me for a some time
> now (and violently ejected me out of lurk mode on the list). What
> follows is a pretty long essay on the topic, I hope you stay with me,
> and take part in the debate, because I think the availability of a
> good buildtool/dependency manager is absolutely crucial to use Clojure
> in larger projects, and as such for adoption on a wider scale.
>
> I have basically the same problem as you in that Clojure is lacking a
> proper buildtool (i.e. one that can replace Maven). I used Maven for a
> long time and lived with it's strengths and weaknesses. It's far from
> perfect, but Maven does still give you quite a lot:
> * a dependency resolution and publishing model for generated artifacts
> * a build based on conventions (can be overridden, but Maven pushes
> you hard to do things in a de-facto standard way)
> * a defined execution model (the phases thing. Not always nice, and
> not always sufficient, but at least something)
> * a defined API to hook into with your plugins (API design is so-so,
> and documentation is IMHO abysmal, but ...)
> * a ton of reports and plugins to extend the model (not always
> perfect, and configuration is not nice, but they usually do what they
> advertise)
> * more?
>
> On the other hand, some things Maven are always a pain in the back:
> figuring out how to configure plugins, writing plugins because it's
> the only way to do something non-standard in Maven, and
> debugging/finding out what the heck is going on, if some plugin is not
> doing what it should do... That's the arcane knowledge part.
>
> But recently I had the need for a more flexible buildtool that would
> allow me to build non-standard JVM based languages in a non-standard
> fashion and still play nice. Case in point was that I had to marry two
> different dependency resolution systems (Maven style, and
> Jython/Python style), and then build a mix of code in
> Scala/Java/Jython. The experience was none too pretty with Maven.
>
> Most of the buildtools out there for JVM related stuff don't try to
> compete with Maven and all it's plugins and reports, but they start by
> replacing/wrapping Ant (and usually Ivy for dependency management).
> That description applies to Buildr, the Groovy Ant Builder, Gant,
> Grape (which is a wrapper for Ivy), SBT (Simple Build Tool - Scala),
> Scalr (Scala), Lancet, and probably some more that I missed. The only
> notable exception I could dig up was Gradle.
>
> Gradle (http://gradle.org/) turns out to be very interesting. Here are
> some propaganda points (not exhaustive):
> * Although it currently uses mainly a Groovy DSL, the Gradle core is
> written in Java, and there's an explicit goal on the roadmap to
> support DSLs written in other JVM languages on top of it as to make it
> an attractive choice for everybody. For Scala there's a patch in the
> Bugtracker to add a Scala DSL on top. Other languages I'm unaware of.
> * Convention based builds. They have taken care that they appeal to
> Maven developers (all the conventions of the Java Plugin are the same
> ones as in Maven), while still leaving it flexible through
> configuration that every project layout can be built.
> * Hierarchical, dependency based builds.
> * A build is executed in two major phases (configuration and
> execution), and tasks are allowed to hook into both (a task can make
> it's execution dependent on whether another task is scheduled to run
> *after* itself (try that with Maven))
> * A build graph containing all the tasks for execution can be accessed
> and manipulated programmatically.
> * Gradle defines it's own task API, but provides an Ant wrapper to
> give you an easy migration route or extension path (similar to Gant,
> or Lancet in Clojure). Custom tasks can be easily coded up, although
> you need custom things less often than with Maven and Co., because a
> lot can be solved with a flexible build script (it's all code).
> * The build can be extended from within a project. You don't need a
> plugin project that you're only going to use in one project anyway.
> There's a separate source tree in your project home that hosts
> extension to your build.
> * Dependency resolution is done with Ivy. But they took care to clean
> the Ivy model a bit up so that it provides better support for
> convention based builds.
> * They have an integration layer with Maven. I don't know how far it
> actually goes, but I know that you can call through to Maven for some
> things (MavenPlugin).
>
> Non-technical
> * The mailinglist is very active, and receptive to input that comes
> from the outside. They have github mirrors, pull requests can be
> issued easily.
> * The project is well setup: toplevel domain, liberal license (Apache
> v2), plethora of SCM can be used (master is on subversion, mirrors on
> github (git), and launchpad (bzr)) issue tracker (JIRA), wiki
> (Confluence), public roadmap, continuous integration (bamboo &
> teamcity, platform specific builds), selfhosting (gradle is built by
> gradle), documentation is autogenerated during the build (including
> the userguide HTML and PDFs).
> * The guys working on Gradle have been around the build tool circus
> for a long time apparently (five committers, among them: the founder
> of JBoss IDE, and the author of Gant).
> * The team is dedicated enough to evangelize their product largescale,
> including talks on conferences around the globe, and at least one of
> the participants runs a consultancy providing advice on buildsystems.
> * Reasonably complete documentation (there's arguably room for
> improvement, especially in structure, but you can find almost anything
> in the userguide).
> * They are willing to change stuff in how it works internally if
> there's technical merit.
> * It's developing fast, and I have the feeling that it's getting
> increasingly powerful and flexible.
> * It's not very complicated IMHO. It's probably slightly more
> complicated than ant + sugar coating, but they Gradle tagline is:
> "make the impossible possible, the possible easy, and the easy
> elegant."  (Moshé Feldenkrais), and I think they are definitely within
> scope.
>
>
> Ok, now the question why am I pushing this so hard? Because have been
> annoyed at the exact same things in every language that I tried for
> the JVM. Maven is not very good but it's usually the best thing you
> can get. And although there's a lot of competition for build tools,
> nobody seems to try to tackle what Maven solved, and only goes for a
> simpler subset of the problems. Usually it goes something like: "Our
> language is more productive than Java and we don't like Maven, because
> it's a pain to use. Ant is nice but uses XML so we can't use it out of
> the box. We can't really wrap Maven, so let's wrap Ant, and since we
> are lacking dependency management in that case, let's also wrap Ivy,
> add a little bit of sugar on top, and voila, new shiny build system."
> Except that everybody now has to learn a gazillion new (and different)
> tools if he wants to build stuff easily in a certain language, and
> beware if you try to do something cross language (a problem that will
> become more common with alternative languages for the JVM on the
> rise).
>
> I would very much like to see that people don't suffer the NIH
> syndrome everywhere. Usually people forget that every alternative
> technology implemented has an associated cost that everyone looking
> for a solution has to pay, because I have to consciously discard the
> tool as insufficient, and that means I have to acquire enough
> knowledge about each tool to make an informed decision (the paradox of
> choice). I cannot even imagine how much time has been wasted by
> halfbaked (and partially or fully abandoned) projects.
>
> If the miracle would happen and the different, rapidly growing
> language communities on the JVM finally could commit themselves to
> support and push a shared buildtool, then that tool would have enough
> steam to grow into a decent alternative for Maven. It might even
> become the preferred way of build at least non-Java projects, who
> knows. If everybody is trying to get their own share of followers and
> only solves what is immediately visible for the projects used by
> themselves and their followers, then we will never get anywhere.
>
> And please note that I'm not asking to start from scratch. Gradle is
> there and has momentum already. I'm just hoping that I'm not the only
> one that sees a benefit in joining forces with stuff that already
> exists. If people in the Clojure community start to throw your weight
> behind something that seems to be on the right track, can still be
> influenced and made better.
>
> To get started with Gradle, you would have to:
> * write a DSL that can talk to Gradle Java API.
> * write a ClojurePlugin (probably based on the JavaPlugin) for Gradle,
> if you have specialized wishes (such as that src/main/clojure should
> be the default convention for Clojure projects, or that you can
> specify which version of Clojure should be used for compilation etc.).
>
> </propaganda>
>
> Sorry for the long mail/rant, but it's been too painful too many
> times. And a sincere apology to Phil, who just scratched an itch for
> himself, and was interested in sharing it with the larger Clojure
> community. This mail is not an attack on your work, or approach. It
> just triggered something that I have stumbled over too many times and
> I would love to see it finally resolved. I think that Gradle could be
> used to give you all you envisioned for Corkscrew, but in the process,
> you will gain so much more.
>
> Disclaimer: I'm (currently) not actively developing Gradle itself, but
> i was impressed with what's there and where it's going. I also took
> the liberty to crosspost this mail to the Gradle dev list (CC'd) to
> invite a broader participation.
>
> I'm really hoping that we could find some consensus on the topic.
>
> These are my probably 20$ or so...
>
> Daniel
> (probably by now unofficial Gradle Evangelist)
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[hdockter]

Apologies for the late response.

On Jul 8, 2009, at 11:51 AM, Daniel wrote:

> On Wed, Jul 8, 2009 at 11:28 AM, Phil Hagelberg<phil@...>  
> wrote:
>>
>> I've been noodling on the problem of dependency management for a  
>> while
>> now. It's definitely a pain point for projects with more than a  
>> couple
>> dependencies. Currently our approach has been to use maven, but that
>> involves a fair amount of arcane knowledge as well as writing a  
>> bunch of
>> XML, which isn't a lot of fun. But there's been a community consensus
>> that any dependency tool needs to be able to leverage all the  
>> valuable
>> stuff out there that's currently in maven repositories.
>>
>> I've put together a proof-of-concept dependency manager called
>> Corkscrew. It uses Maven under the hood as well as offering  
>> dependencies
>> on source packages stored in git or subversion. A simple project.clj
>> file lays out the dependencies and other project info:
>>
>>   {:name "my-sample"
>>    :version "1.0"
>>    :dependencies [["tagsoup" "1.2" "org.ccil.cowan.tagsoup"]
>>                   ;; group defaults to name
>>                   ["rome" "0.9"]]
>>    :source-dependencies [["clojure-contrib" "r663" :svn
>>                           "http://clojure-contrib.googlecode.com/svn/trunk 
>> "]
>>                          ["enlive"  
>> "95b2558943f50bb9962fe7d500ede353f1b578f0"
>>                           :git "git://github.com/cgrand/
>> enlive.git"]]}
>>
>> You can install it with:
>>
>> $ git clone git://github.com/technomancy/corkscrew.git
>> $ cd corkscrew
>> $ ./install
>> $ cp bin/corkscrew /somewhere/on/your/path
>>
>> Create a project.clj file in your project root based on the one
>> above. Then you can use "corkscrew deps" to set everything up for
>> you. At that point just make sure your classpath includes
>> target/dependency/ and you should be good to go.
>>
>> It's simple, but it solves the main problems that I've been having  
>> with
>> more complicated projects. I'd love to get some opinions on it.
>>
>> The biggest issue right now is that it runs Maven as a subprocess  
>> rather
>> than using the Java API in the same VM because I can't make head or  
>> tail
>> of the Maven Java API (it uses plexus.core), but shelling out works  
>> as a
>> proof-of-concept even if it's tacky.
>>
>> -Phil
>>
>
> Thanks for bringing that up, as it's been bugging me for a some time
> now (and violently ejected me out of lurk mode on the list). What
> follows is a pretty long essay on the topic, I hope you stay with me,
> and take part in the debate, because I think the availability of a
> good buildtool/dependency manager is absolutely crucial to use Clojure
> in larger projects, and as such for adoption on a wider scale.
>
> I have basically the same problem as you in that Clojure is lacking a
> proper buildtool (i.e. one that can replace Maven). I used Maven for a
> long time and lived with it's strengths and weaknesses. It's far from
> perfect, but Maven does still give you quite a lot:
> * a dependency resolution and publishing model for generated artifacts
> * a build based on conventions (can be overridden, but Maven pushes
> you hard to do things in a de-facto standard way)
> * a defined execution model (the phases thing. Not always nice, and
> not always sufficient, but at least something)
> * a defined API to hook into with your plugins (API design is so-so,
> and documentation is IMHO abysmal, but ...)
> * a ton of reports and plugins to extend the model (not always
> perfect, and configuration is not nice, but they usually do what they
> advertise)
> * more?
>
> On the other hand, some things Maven are always a pain in the back:
> figuring out how to configure plugins, writing plugins because it's
> the only way to do something non-standard in Maven, and
> debugging/finding out what the heck is going on, if some plugin is not
> doing what it should do... That's the arcane knowledge part.
>
> But recently I had the need for a more flexible buildtool that would
> allow me to build non-standard JVM based languages in a non-standard
> fashion and still play nice. Case in point was that I had to marry two
> different dependency resolution systems (Maven style, and
> Jython/Python style), and then build a mix of code in
> Scala/Java/Jython. The experience was none too pretty with Maven.
>
> Most of the buildtools out there for JVM related stuff don't try to
> compete with Maven and all it's plugins and reports, but they start by
> replacing/wrapping Ant (and usually Ivy for dependency management).
> That description applies to Buildr, the Groovy Ant Builder, Gant,
> Grape (which is a wrapper for Ivy), SBT (Simple Build Tool - Scala),
> Scalr (Scala), Lancet, and probably some more that I missed. The only
> notable exception I could dig up was Gradle.
>
> Gradle (http://gradle.org/) turns out to be very interesting. Here are
> some propaganda points (not exhaustive):
> * Although it currently uses mainly a Groovy DSL, the Gradle core is
> written in Java, and there's an explicit goal on the roadmap to
> support DSLs written in other JVM languages on top of it as to make it
> an attractive choice for everybody. For Scala there's a patch in the
> Bugtracker to add a Scala DSL on top. Other languages I'm unaware of.
> * Convention based builds. They have taken care that they appeal to
> Maven developers (all the conventions of the Java Plugin are the same
> ones as in Maven), while still leaving it flexible through
> configuration that every project layout can be built.
> * Hierarchical, dependency based builds.
> * A build is executed in two major phases (configuration and
> execution), and tasks are allowed to hook into both (a task can make
> it's execution dependent on whether another task is scheduled to run
> *after* itself (try that with Maven))
> * A build graph containing all the tasks for execution can be accessed
> and manipulated programmatically.
> * Gradle defines it's own task API, but provides an Ant wrapper to
> give you an easy migration route or extension path (similar to Gant,
> or Lancet in Clojure). Custom tasks can be easily coded up, although
> you need custom things less often than with Maven and Co., because a
> lot can be solved with a flexible build script (it's all code).

In Gradle 0.7 we have introduce deep integration with existing ant  
builds. AFAIK no other JVM build tool (except Ant :)) is providing  
this. See the user's guide for details.

> * The build can be extended from within a project. You don't need a
> plugin project that you're only going to use in one project anyway.
> There's a separate source tree in your project home that hosts
> extension to your build.
> * Dependency resolution is done with Ivy. But they took care to clean
> the Ivy model a bit up so that it provides better support for
> convention based builds.
> * They have an integration layer with Maven. I don't know how far it
> actually goes, but I know that you can call through to Maven for some
> things (MavenPlugin).

So far the integration is repository based. Gradle can generate a  
pom.xml for you and install/deploy your artifacts together with the  
pom to your local/remote repositories. No other integration is offered  
yet. It is very easy to write a pass-through script to loosely  
integrate existing Maven builds. Once Maven 3.0 is out, we plan to use  
the new embedded functionality of Maven to provide deep integration  
with existing Maven builds. But the details for this have yet to be  
figured out.

>
> Non-technical
> * The mailinglist is very active, and receptive to input that comes
> from the outside. They have github mirrors, pull requests can be
> issued easily.
> * The project is well setup: toplevel domain, liberal license (Apache
> v2), plethora of SCM can be used (master is on subversion, mirrors on
> github (git), and launchpad (bzr)) issue tracker (JIRA), wiki
> (Confluence), public roadmap, continuous integration (bamboo &
> teamcity, platform specific builds), selfhosting (gradle is built by
> gradle), documentation is autogenerated during the build (including
> the userguide HTML and PDFs).
> * The guys working on Gradle have been around the build tool circus
> for a long time apparently (five committers, among them: the founder
> of JBoss IDE, and the author of Gant).
> * The team is dedicated enough to evangelize their product largescale,
> including talks on conferences around the globe, and at least one of
> the participants runs a consultancy providing advice on buildsystems.
> * Reasonably complete documentation (there's arguably room for
> improvement, especially in structure, but you can find almost anything
> in the userguide).
> * They are willing to change stuff in how it works internally if
> there's technical merit.
> * It's developing fast, and I have the feeling that it's getting
> increasingly powerful and flexible.
> * It's not very complicated IMHO. It's probably slightly more
> complicated than ant + sugar coating, but they Gradle tagline is:
> "make the impossible possible, the possible easy, and the easy
> elegant."  (Moshé Feldenkrais), and I think they are definitely within
> scope.
>
>
> Ok, now the question why am I pushing this so hard? Because have been
> annoyed at the exact same things in every language that I tried for
> the JVM. Maven is not very good but it's usually the best thing you
> can get. And although there's a lot of competition for build tools,
> nobody seems to try to tackle what Maven solved, and only goes for a
> simpler subset of the problems. Usually it goes something like: "Our
> language is more productive than Java and we don't like Maven, because
> it's a pain to use. Ant is nice but uses XML so we can't use it out of
> the box. We can't really wrap Maven, so let's wrap Ant, and since we
> are lacking dependency management in that case, let's also wrap Ivy,
> add a little bit of sugar on top, and voila, new shiny build system."
> Except that everybody now has to learn a gazillion new (and different)
> tools if he wants to build stuff easily in a certain language, and
> beware if you try to do something cross language (a problem that will
> become more common with alternative languages for the JVM on the
> rise).
>
> I would very much like to see that people don't suffer the NIH
> syndrome everywhere. Usually people forget that every alternative
> technology implemented has an associated cost that everyone looking
> for a solution has to pay, because I have to consciously discard the
> tool as insufficient, and that means I have to acquire enough
> knowledge about each tool to make an informed decision (the paradox of
> choice). I cannot even imagine how much time has been wasted by
> halfbaked (and partially or fully abandoned) projects.
>
> If the miracle would happen and the different, rapidly growing
> language communities on the JVM finally could commit themselves to
> support and push a shared buildtool, then that tool would have enough
> steam to grow into a decent alternative for Maven. It might even
> become the preferred way of build at least non-Java projects, who
> knows. If everybody is trying to get their own share of followers and
> only solves what is immediately visible for the projects used by
> themselves and their followers, then we will never get anywhere.
>
> And please note that I'm not asking to start from scratch. Gradle is
> there and has momentum already. I'm just hoping that I'm not the only
> one that sees a benefit in joining forces with stuff that already
> exists. If people in the Clojure community start to throw your weight
> behind something that seems to be on the right track, can still be
> influenced and made better.
>
> To get started with Gradle, you would have to:
> * write a DSL that can talk to Gradle Java API.
> * write a ClojurePlugin (probably based on the JavaPlugin) for Gradle,
> if you have specialized wishes (such as that src/main/clojure should
> be the default convention for Clojure projects, or that you can
> specify which version of Clojure should be used for compilation etc.).

In Gradle trunk we have now a Scala plugin which will be part of  
Gradle 0.8.

Regarding a Clojure DSL. We wanted to avoid speculative generality. So  
there is no plugin-your-DSL-engine mechanism yet. Clojure would be the  
first alternative DSL so I guess we would have to make a few  
modification in the core to make everything work. But we would be very  
happy to do this.

>
> </propaganda>
>
> Sorry for the long mail/rant, but it's been too painful too many
> times. And a sincere apology to Phil, who just scratched an itch for
> himself, and was interested in sharing it with the larger Clojure
> community. This mail is not an attack on your work, or approach. It
> just triggered something that I have stumbled over too many times and
> I would love to see it finally resolved. I think that Gradle could be
> used to give you all you envisioned for Corkscrew, but in the process,
> you will gain so much more.
>
> Disclaimer: I'm (currently) not actively developing Gradle itself, but
> i was impressed with what's there and where it's going. I also took
> the liberty to crosspost this mail to the Gradle dev list (CC'd) to
> invite a broader participation.
>
> I'm really hoping that we could find some consensus on the topic.
>
> These are my probably 20$ or so...
>
> Daniel
> (probably by now unofficial Gradle Evangelist)

Thanks a lot David for taking such an in-depth look at Gradle.

- Hans

--
Hans Dockter
Gradle Project Manager
http://www.gradle.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email