Performance of Gradle vs. Maven

[Paul Gier]

We're currently using Maven for a lot of our projects at JBoss and I've been
researching Gradle a bit to see if it can solve some problems in our builds.  I
set up one of our simple builds to use Gradle just to check it's performance vs.
Maven, and I was kind of surprised to see that Gradle seemed somewhat slower for
the build.

The project is a basic java/jar project and I am using the gradle "java" plugin.
  In Maven this is equivalent to a "jar" packaging.  So running a clean build

gradle clean libs ~ 12s
mvn clean package ~ 8s

gradle libs ~ 10 - 20s (This one varied quite a bit while testing)
mvn package ~ 5s

Maven builds seem significantly slower in general than Ant builds, so I was
hoping that Gradle would be somewhere in between.  So anyway, my question is
mainly (1) whether performance is a known problem with Gradle and it's being
worked on, (2) maybe Gradle just has a slow startup time and with bigger
projects it's not an issue, (3) maybe I'm doing/not doing something that is
making the build slower.

I obviously didn't do any detailed test, but I did notice that in my small build
the steps that seem to take the most time are the initial startup, the test
task, and the jar task.

How do I tell Gradle to fork once for the tests instead of the default per test
forking?  Also I remember reading something in the user guide about pre
compiling the build script, how do I do that?

Thanks much!




---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[RenŽé Gröschke]

Hi Paul,

Am 10.06.2009 um 19:50 schrieb Paul Gier:

>
> We're currently using Maven for a lot of our projects at JBoss and  
> I've been researching Gradle a bit to see if it can solve some  
> problems in our builds.  I set up one of our simple builds to use  
> Gradle just to check it's performance vs. Maven, and I was kind of  
> surprised to see that Gradle seemed somewhat slower for the build.
>
> The project is a basic java/jar project and I am using the gradle  
> "java" plugin.  In Maven this is equivalent to a "jar" packaging.  
> So running a clean build
>
> gradle clean libs ~ 12s
> mvn clean package ~ 8s
>
> gradle libs ~ 10 - 20s (This one varied quite a bit while testing)
> mvn package ~ 5s
>
> Maven builds seem significantly slower in general than Ant builds,  
> so I was hoping that Gradle would be somewhere in between.  So  
> anyway, my question is mainly (1) whether performance is a known  
> problem with Gradle and it's being worked on, (2) maybe Gradle just  
> has a slow startup time and with bigger projects it's not an issue,  
> (3) maybe I'm doing/not doing something that is making the build  
> slower.

The performance of gradle was discussed some days ago at the gradle  
user list. Indeed it seems that the startup time of gradle is greater  
than in maven and ant. Take a look at the discussion at the  
mailinglist archive:
http://www.nabble.com/Gradle-performance-td23806096.html for detailed  
informations.

> I obviously didn't do any detailed test, but I did notice that in my  
> small build the steps that seem to take the most time are the  
> initial startup, the test task, and the jar task.
>
> How do I tell Gradle to fork once for the tests instead of the  
> default per test forking?  Also I remember reading something in the  
> user guide about pre compiling the build script, how do I do that?
>
> Thanks much!
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>   http://xircles.codehaus.org/manage_email
>

regards,
René



-----------------------------------
René Gröschke
rene@...
http://www.breskeby.com





---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Paul Gier]

Rene Groeschke wrote:

> Hi Paul,
>
> Am 10.06.2009 um 19:50 schrieb Paul Gier:
>
>>
>> We're currently using Maven for a lot of our projects at JBoss and
>> I've been researching Gradle a bit to see if it can solve some
>> problems in our builds.  I set up one of our simple builds to use
>> Gradle just to check it's performance vs. Maven, and I was kind of
>> surprised to see that Gradle seemed somewhat slower for the build.
>>
>> The project is a basic java/jar project and I am using the gradle
>> "java" plugin.  In Maven this is equivalent to a "jar" packaging.  So
>> running a clean build
>>
>> gradle clean libs ~ 12s
>> mvn clean package ~ 8s
>>
>> gradle libs ~ 10 - 20s (This one varied quite a bit while testing)
>> mvn package ~ 5s
>>
>> Maven builds seem significantly slower in general than Ant builds, so
>> I was hoping that Gradle would be somewhere in between.  So anyway, my
>> question is mainly (1) whether performance is a known problem with
>> Gradle and it's being worked on, (2) maybe Gradle just has a slow
>> startup time and with bigger projects it's not an issue, (3) maybe I'm
>> doing/not doing something that is making the build slower.
>
> The performance of gradle was discussed some days ago at the gradle user
> list. Indeed it seems that the startup time of gradle is greater than in
> maven and ant. Take a look at the discussion at the mailinglist archive:
> http://www.nabble.com/Gradle-performance-td23806096.html for detailed
> informations.
>
>

Ok, thanks for the info.  It looks like most of my questions are answered in
that discussion.  Sorry to bring this up again on the dev list.

>> I obviously didn't do any detailed test, but I did notice that in my
>> small build the steps that seem to take the most time are the initial
>> startup, the test task, and the jar task.
>>
>> How do I tell Gradle to fork once for the tests instead of the default
>> per test forking?  Also I remember reading something in the user guide
>> about pre compiling the build script, how do I do that?
>>
>> Thanks much!
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>   http://xircles.codehaus.org/manage_email
>>
>
>
>
> regards,
> René
>
>
>
> -----------------------------------
> René Gröschke
> rene@...
> http://www.breskeby.com
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[hdockter]

On Jun 10, 2009, at 7:50 PM, Paul Gier wrote:

>
> We're currently using Maven for a lot of our projects at JBoss and  
> I've been researching Gradle a bit to see if it can solve some  
> problems in our builds.  I set up one of our simple builds to use  
> Gradle just to check it's performance vs. Maven, and I was kind of  
> surprised to see that Gradle seemed somewhat slower for the build.
>
> The project is a basic java/jar project and I am using the gradle  
> "java" plugin.  In Maven this is equivalent to a "jar" packaging.  
> So running a clean build
>
> gradle clean libs ~ 12s
> mvn clean package ~ 8s
>
> gradle libs ~ 10 - 20s (This one varied quite a bit while testing)
> mvn package ~ 5s
>
> Maven builds seem significantly slower in general than Ant builds,  
> so I was hoping that Gradle would be somewhere in between.  So  
> anyway, my question is mainly (1) whether performance is a known  
> problem with Gradle and it's being worked on,

The performance of a build tool has a couple of aspects. One is its  
execution efficiency. The other is its smartness in not doing  
unnecessary actions.

I don't think there are major usability issues with Gradle's execution  
efficiency. On the other hand we haven't focused on optimizing that in  
the last releases. I was doing some profiling recently and found a  
significant inefficiency in some areas were we use reflection. The fix  
is now in trunk and has a very nice impact on the performance.

I use the Apache Abdera project (which is a 28 projects multi-project  
build) as my guinea pig project for comparisons.

gradle clean libs -Dskip.test takes 15 seconds on my machine. (Gradle  
trunk)
mvn clean package -o takes 20 seconds. (Maven 2.1)

gradle libs -Dskip.test takes 7 seconds on my machine. (Gradle trunk)
mvn package -o takes 9 seconds. (Maven 2.1)

So this is looking very good I think. There are more optimizations  
that can be done to improve execution efficiency. We definitely need  
to figure out why the testing takes so much longer than with Ant. Its  
funny that Maven has the same problem.

Another area were we want to improve Gradle in not doing unnecessary  
stuff. Having a good change detection and act on this accordingly,  
etc ....

> (2) maybe Gradle just has a slow startup time and with bigger  
> projects it's not an issue, (3) maybe I'm doing/not doing something  
> that is making the build slower.
>
> I obviously didn't do any detailed test, but I did notice that in my  
> small build the steps that seem to take the most time are the  
> initial startup, the test task, and the jar task.
>
> How do I tell Gradle to fork once for the tests instead of the  
> default per test forking?

test {
     options.fork(forkMode: ForkMode.ONCE)
}

or

test.options.fork(forkMode: ForkMode.ONCE)

> Also I remember reading something in the user guide about pre  
> compiling the build script, how do I do that?

This is done automatically for you. The precompiled script can be  
found in the project's .gradle directory.

- Hans

--
Hans Dockter
Gradle Project Manager
http://www.gradle.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Peter Niederwieser]

hdockter wrote:

test.options.fork(forkMode: ForkMode.ONCE)

IMHO, ForkMode.ONCE should be the default. ForkMode.PER_TEST may be a good choice for integration tests, but for unit tests it's overkill (trying too hard to be on the safe side). I don't know about Ant, but Maven also uses "once" by default.

Cheers,
Peter

[Adam Murdoch-2]

Peter Niederwieser wrote:

hdockter wrote:
  

test.options.fork(forkMode: ForkMode.ONCE)

    

IMHO, ForkMode.ONCE should be the default.

+1

 ForkMode.PER_TEST may be a good
choice for integration tests, but for unit tests it's overkill (trying too
hard to be on the safe side). I don't know about Ant, but Maven also uses
"once" by default.

Cheers,
Peter

  

[Russel Winder-4]

On Thu, 2009-06-11 at 08:54 +0200, Hans Dockter wrote:
[ . . . ]
> > gradle clean libs ~ 12s
> > mvn clean package ~ 8s
> >
> > gradle libs ~ 10 - 20s (This one varied quite a bit while testing)
> > mvn package ~ 5s


The following is not an experiment as such just a summary of a few data
points (so more than just one run taken at random).

For my ADS library (which is pure Java with TestNG) on my 3GHz P4 with
NFS mounted filestore, I find:

"gradle clean" of a fully built project ~ 4--6s
"mvn clean" of a fully built project ~ 4--5s

"gradle clean" of an already clean project ~ 4--4.5s
"mvn clean" of an already clean project ~ 1--2s

"gradle -i libs" from a cleaned stated ~ 15--23s
"mvn package" from a cleaned state ~ 13--15s

"gradle -i libs" of an already built project ~ 8.5--10.5s
"mvn package" of an already built project ~ 6--7s

This is the time reported by the application and is clearly only
partially related to the time it actually takes from issuing the command
to getting the next prompt.  A full experiment would measure reported
time and actual operating system recorded time.

So overall the impression that Gradle is a bit slower than Maven is
backed up, at least for this project in its current state.

I have long ago deleted the Ant and Gant builds so I cannot compare
those.

[ . . . ]

> The performance of a build tool has a couple of aspects. One is its  
> execution efficiency. The other is its smartness in not doing  
> unnecessary actions.

True but in what situations is this an issue?  Given the indicators
above, Gradle is still only just keeping up with Maven for a
straightforward, pure Java project of relatively small size.  
 
> I don't think there are major usability issues with Gradle's execution  
> efficiency. On the other hand we haven't focused on optimizing that in  
> the last releases. I was doing some profiling recently and found a  
> significant inefficiency in some areas were we use reflection. The fix  
> is now in trunk and has a very nice impact on the performance.

Gradle's performance is definitely in the "not bad" category, it is
perfectly usable.  However, it does appear to be slower than Maven for
the cases I use it for, which is sad.

> I use the Apache Abdera project (which is a 28 projects multi-project  
> build) as my guinea pig project for comparisons.
>
> gradle clean libs -Dskip.test takes 15 seconds on my machine. (Gradle  
> trunk)
> mvn clean package -o takes 20 seconds. (Maven 2.1)
>
> gradle libs -Dskip.test takes 7 seconds on my machine. (Gradle trunk)
> mvn package -o takes 9 seconds. (Maven 2.1)

I don't know whether the -Dskip.test actually makes a difference to the
Gradle run or the -o to the Maven run, but it is clear that the Gradle
run is not directly analogous to the Maven run so this data point could
be undermined.

> So this is looking very good I think. There are more optimizations  
> that can be done to improve execution efficiency. We definitely need  
> to figure out why the testing takes so much longer than with Ant. Its  
> funny that Maven has the same problem.
>
> Another area were we want to improve Gradle in not doing unnecessary  
> stuff. Having a good change detection and act on this accordingly,  
> etc ....

I must reiterate that I don't think Gradle has a performance problem per
se, but I do think it would be better if we can get Gradle performing
faster than Maven in real use projects and workflows -- as opposed to
benchmarking.  It would be good to be faster in benchmarking as well of
course :-)

> > (2) maybe Gradle just has a slow startup time and with bigger  
> > projects it's not an issue, (3) maybe I'm doing/not doing something  
> > that is making the build slower.
> >
> > I obviously didn't do any detailed test, but I did notice that in my  
> > small build the steps that seem to take the most time are the  
> > initial startup, the test task, and the jar task.

I suspect there is a Groovy startup cost.  I have no data points to back
up this opinion, but at every turn Groovy execution tends to show a
startup cost over Java execution.

> > How do I tell Gradle to fork once for the tests instead of the  
> > default per test forking?
>
> test {
>      options.fork(forkMode: ForkMode.ONCE)
> }
>
> or
>
> test.options.fork(forkMode: ForkMode.ONCE)

Is this true for TestNG tests or just JUnit tests?

I suspect not since my project adding this line leads to:

Cause: No signature of method: groovy.xml.MarkupBuilder.fork() is
applicable for argument types: (java.util.LinkedHashMap) values:
{["forkMode":once]}

Moral of this story is I guess that Gradle has to stop assuming JUnit is
being used.
[ . . . ]
--
Russel.
=============================================================================
Dr Russel Winder      Partner
                                            xmpp: russel@...
Concertant LLP        t: [phone], [phone]
41 Buckmaster Road,   f: [phone]   voip: sip:russel.winder@...
London SW11 1EN, UK   m: [phone]   skype: russel_winder

[Russel Winder-4]

On Fri, 2009-06-12 at 07:55 +0100, Russel Winder wrote:
[ . . . ]
> I have long ago deleted the Ant and Gant builds so I cannot compare
> those.

For the Gant project (mixed Java and Groovy, using JUnit, compiled on P4
with NFS mounted filestore) I can compare an Ant, Gant and Gradle builds
(note this is reported time not real time, so 0s doesn't really mean
0s :-):

Cleaning a clean project:
        Ant ~ 0--1s
        Gant ~ 3.1--3.2s
        Gradle ~ 5.1--5.3s

Cleaning from built:
        Ant ~ 2--5s
        Gant ~ 5.5--6.5s
        Gradle ~ 6.5--8.5s

Building from clean, no tests:
        Ant ~ 10--11s
        Gant ~ 10.5--11.5s
        Gradle ~ 15.5--20s

Building from built, no tests:
        Ant ~ 1s
        Gant ~ 3.4--3.5s
        Gradle ~ 7.5s

I took the tests out as they take basically the same amount of time in
all cases and one of them takes a whole 32s!

What I think this shows is that there is a Groovy penalty over Java and
that Gradle is a bit slower than Ant/Gant.  The extra time taken by
Gradle is really worrisome, but it is a bit irritating -- it provides a
bit of a break on rapid turnaround.

[ . . . ]

--
Russel.
=============================================================================
Dr Russel Winder      Partner
                                            xmpp: russel@...
Concertant LLP        t: [phone], [phone]
41 Buckmaster Road,   f: [phone]   voip: sip:russel.winder@...
London SW11 1EN, UK   m: [phone]   skype: russel_winder

[Russel Winder-4]

On Thu, 2009-06-11 at 08:54 +0200, Hans Dockter wrote:
[ . . . ]

> > How do I tell Gradle to fork once for the tests instead of the  
> > default per test forking?
>
> test {
>      options.fork(forkMode: ForkMode.ONCE)
> }
>
> or
>
> test.options.fork(forkMode: ForkMode.ONCE)

Uuuurrrr.... for the Gant build I am seeing that Gradle uses once mode
by default and that I have to use

test.options.fork ( forkMode : ForkMode.PER_TEST )

to get per test mode.  I am using Gradle Trunk though.

--
Russel.
=============================================================================
Dr Russel Winder      Partner
                                            xmpp: russel@...
Concertant LLP        t: [phone], [phone]
41 Buckmaster Road,   f: [phone]   voip: sip:russel.winder@...
London SW11 1EN, UK   m: [phone]   skype: russel_winder

[hdockter]

On Jun 12, 2009, at 8:55 AM, Russel Winder wrote:

> On Thu, 2009-06-11 at 08:54 +0200, Hans Dockter wrote:
> [ . . . ]
>>> gradle clean libs ~ 12s
>>> mvn clean package ~ 8s
>>>
>>> gradle libs ~ 10 - 20s (This one varied quite a bit while testing)
>>> mvn package ~ 5s
>
>
> The following is not an experiment as such just a summary of a few  
> data
> points (so more than just one run taken at random).
>
> For my ADS library (which is pure Java with TestNG) on my 3GHz P4 with
> NFS mounted filestore, I find:
>
> "gradle clean" of a fully built project ~ 4--6s
> "mvn clean" of a fully built project ~ 4--5s
>
> "gradle clean" of an already clean project ~ 4--4.5s
> "mvn clean" of an already clean project ~ 1--2s
>
> "gradle -i libs" from a cleaned stated ~ 15--23s
> "mvn package" from a cleaned state ~ 13--15s
>
> "gradle -i libs" of an already built project ~ 8.5--10.5s
> "mvn package" of an already built project ~ 6--7s
>
> This is the time reported by the application and is clearly only
> partially related to the time it actually takes from issuing the  
> command
> to getting the next prompt.  A full experiment would measure reported
> time and actual operating system recorded time.
>
> So overall the impression that Gradle is a bit slower than Maven is
> backed up, at least for this project in its current state.
>
> I have long ago deleted the Ant and Gant builds so I cannot compare
> those.
>
> [ . . . ]
>
>> The performance of a build tool has a couple of aspects. One is its
>> execution efficiency. The other is its smartness in not doing
>> unnecessary actions.
>
> True but in what situations is this an issue?  Given the indicators
> above, Gradle is still only just keeping up with Maven for a
> straightforward, pure Java project of relatively small size.

Those numbers above don't say anything about how well Gradle scales in  
comparison with Maven regarding multi-project builds.

>
>> I don't think there are major usability issues with Gradle's  
>> execution
>> efficiency. On the other hand we haven't focused on optimizing that  
>> in
>> the last releases. I was doing some profiling recently and found a
>> significant inefficiency in some areas were we use reflection. The  
>> fix
>> is now in trunk and has a very nice impact on the performance.
>
> Gradle's performance is definitely in the "not bad" category, it is
> perfectly usable.  However, it does appear to be slower than Maven for
> the cases I use it for, which is sad.
>
>> I use the Apache Abdera project (which is a 28 projects multi-project
>> build) as my guinea pig project for comparisons.
>>
>> gradle clean libs -Dskip.test takes 15 seconds on my machine. (Gradle
>> trunk)
>> mvn clean package -o takes 20 seconds. (Maven 2.1)
>>
>> gradle libs -Dskip.test takes 7 seconds on my machine. (Gradle trunk)
>> mvn package -o takes 9 seconds. (Maven 2.1)
>
> I don't know whether the -Dskip.test actually makes a difference to  
> the
> Gradle run or the -o to the Maven run, but it is clear that the Gradle
> run is not directly analogous to the Maven run so this data point  
> could
> be undermined.

I did a mistake above. The maven command I was actually using is:

mvn -Dmaven.test.skip=true clean package -o

The -o prevents Maven from doing any snapshot lookup thus improves its  
performance.

So those runs are analogous.

>
>> So this is looking very good I think. There are more optimizations
>> that can be done to improve execution efficiency. We definitely need
>> to figure out why the testing takes so much longer than with Ant. Its
>> funny that Maven has the same problem.
>>
>> Another area were we want to improve Gradle in not doing unnecessary
>> stuff. Having a good change detection and act on this accordingly,
>> etc ....
>
> I must reiterate that I don't think Gradle has a performance problem  
> per
> se, but I do think it would be better if we can get Gradle performing
> faster than Maven in real use projects and workflows -- as opposed to
> benchmarking.  It would be good to be faster in benchmarking as well  
> of
> course :-)
>
>>> (2) maybe Gradle just has a slow startup time and with bigger
>>> projects it's not an issue, (3) maybe I'm doing/not doing something
>>> that is making the build slower.
>>>
>>> I obviously didn't do any detailed test, but I did notice that in my
>>> small build the steps that seem to take the most time are the
>>> initial startup, the test task, and the jar task.
>
> I suspect there is a Groovy startup cost.  I have no data points to  
> back
> up this opinion, but at every turn Groovy execution tends to show a
> startup cost over Java execution.
>
>>> How do I tell Gradle to fork once for the tests instead of the
>>> default per test forking?
>>
>> test {
>>     options.fork(forkMode: ForkMode.ONCE)
>> }
>>
>> or
>>
>> test.options.fork(forkMode: ForkMode.ONCE)
>
> Is this true for TestNG tests or just JUnit tests?
>
> I suspect not since my project adding this line leads to:
>
> Cause: No signature of method: groovy.xml.MarkupBuilder.fork() is
> applicable for argument types: (java.util.LinkedHashMap) values:
> {["forkMode":once]}

Jira please :)

Thanks

- Hans

--
Hans Dockter
Gradle Project Manager
http://www.gradle.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[hdockter]

On Jun 12, 2009, at 9:50 AM, Russel Winder wrote:

> On Thu, 2009-06-11 at 08:54 +0200, Hans Dockter wrote:
> [ . . . ]
>>> How do I tell Gradle to fork once for the tests instead of the
>>> default per test forking?
>>
>> test {
>>     options.fork(forkMode: ForkMode.ONCE)
>> }
>>
>> or
>>
>> test.options.fork(forkMode: ForkMode.ONCE)
>
> Uuuurrrr.... for the Gant build I am seeing that Gradle uses once mode
> by default and that I have to use
>
> test.options.fork ( forkMode : ForkMode.PER_TEST )
>
> to get per test mode.  I am using Gradle Trunk though.

For Groovy projects we use ONCE and for Java projects we use PER_TEST  
as default.

- Hans

--
Hans Dockter
Gradle Project Manager
http://www.gradle.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[hdockter]

On Jun 11, 2009, at 1:11 PM, Peter Niederwieser wrote:

>
>
> hdockter wrote:
>>
>> test.options.fork(forkMode: ForkMode.ONCE)
>>
>
> IMHO, ForkMode.ONCE should be the default. ForkMode.PER_TEST may be  
> a good
> choice for integration tests, but for unit tests it's overkill  
> (trying too
> hard to be on the safe side). I don't know about Ant, but Maven also  
> uses
> "once" by default.

Ant uses per test. Gradle actually uses per test for Java projects and  
once for Groovy projects. At least that's what its supposed to do.

- Hans

--
Hans Dockter
Gradle Project Manager
http://www.gradle.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email