Gradle performance

[Peter Niederwieser]

Hi,

first of all my congratulations to Gradle 0.6! I'm happy to say that compiling and running Spock specifications now works just fine. Also I'm impressed with the much improved build script DSL.

I have a HelloSpock project that contains a single Groovy class (a Spock specification). What puzzles me a bit is that 'gradle test' takes 10 seconds to complete. By comparison, 'ant test' takes 3 seconds, and 'mvn test' 8 seconds. This is on repeated invocation, i.e. all downloading and compilation has already taken place. Do you consider this acceptable? Would the difference be less pronounced for bigger projects?

Cheers,
Peter

PS: You can download the HelloSpock project from http://spock.googlecode.com/files/HelloSpockProject.zip

[Curtis Cooley-3]

On Sun, May 31, 2009 at 12:21 PM, Peter Niederwieser <pniederw@...> wrote:
> I have a HelloSpock project that contains a single Groovy class (a Spock
> specification). What puzzles me a bit is that 'gradle test' takes 10 seconds
> to complete. By comparison, 'ant test' takes 3 seconds, and 'mvn test' 8
> seconds. This is on repeated invocation, i.e. all downloading and
> compilation has already taken place. Do you consider this acceptable? Would
> the difference be less pronounced for bigger projects?

I think what you are seeing is the startup cost of Gradle, which in my
observations is higher than Ant's. When you get to 60 second
build/test runs then you should be seeing 53 second ant builds and 60
second Gradle builds. Whether that is acceptable is up to you.
--
Curtis Cooley
curtis.cooley@...
===============
Once a programmer had a problem. He thought he could solve it with a
regular expression. Now he has two problems.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Russel Winder-4]

On Sun, 2009-05-31 at 13:14 -0700, Curtis Cooley wrote:

> On Sun, May 31, 2009 at 12:21 PM, Peter Niederwieser <pniederw@...> wrote:
> > I have a HelloSpock project that contains a single Groovy class (a Spock
> > specification). What puzzles me a bit is that 'gradle test' takes 10 seconds
> > to complete. By comparison, 'ant test' takes 3 seconds, and 'mvn test' 8
> > seconds. This is on repeated invocation, i.e. all downloading and
> > compilation has already taken place. Do you consider this acceptable? Would
> > the difference be less pronounced for bigger projects?
>
> I think what you are seeing is the startup cost of Gradle, which in my
> observations is higher than Ant's. When you get to 60 second
> build/test runs then you should be seeing 53 second ant builds and 60
> second Gradle builds. Whether that is acceptable is up to you.

The problem is that those of use with smaller builds are a bit concerned
with the fact that Gradle is slower than Maven and definitely slower
than Ant.

No matter whether justified or not, whether it can be ameliorated or
not, small builds is where people make comparison judgements, and where
benchmarks are created.  There is also a tendency for detractors of a
technology to obsess about performance issues -- witness the arguments
about Groovy itself, and indeed the hassles SCons is getting.

I think there are more important technical evolutions lined up for 0.7,
and I think they should remain the priority.  However, I think
performance should be reviewed for 0.8.  There was a review previously
and this led to a major rewrite of large tracts of Gradle source from
Groovy to Java, and this successfully slashed run times.

Perhaps a new stage of profiling on a number of projects is in order to
find the "hot spot" and see if there are factors stopping the JIT from
doing its job.  Or perhaps the problem is a class written in Groovy that
needs to be rewritten in Java.  Alternatively there are just
infelicities in the Java that have not been picked up.

Gradle's performance is in the right scale so I don't think there is a
crisis here, so I don't think there needs to be any deep worrying.
However it would be good to get performance closer to Ant and definitely
better than Maven.

To progress this we need to set up an experiment where as many people as
possible execute their build with profiling so as to provide data.  I
can't volunteer to drive this, but if there is anyone out there with
strong profiling experience who can, that would be fantastic.
--
Russel.
=============================================================================
Dr Russel Winder      Partner
                                            xmpp: russel@...
Concertant LLP        t: [phone], [phone]
41 Buckmaster Road,   f: [phone]   voip: sip:russel.winder@...
London SW11 1EN, UK   m: [phone]   skype: russel_winder

[Peter Niederwieser]

I fully agree with Russel. I'm not really worried about Gradle's performance at the moment. I mainly wanted to provide some feedback because these results came as a surprise to me.

Cheers,
Peter

Russel Winder-4 wrote:

On Sun, 2009-05-31 at 13:14 -0700, Curtis Cooley wrote:
> On Sun, May 31, 2009 at 12:21 PM, Peter Niederwieser <[e-mail]> wrote:
> > I have a HelloSpock project that contains a single Groovy class (a Spock
> > specification). What puzzles me a bit is that 'gradle test' takes 10 seconds
> > to complete. By comparison, 'ant test' takes 3 seconds, and 'mvn test' 8
> > seconds. This is on repeated invocation, i.e. all downloading and
> > compilation has already taken place. Do you consider this acceptable? Would
> > the difference be less pronounced for bigger projects?
>
> I think what you are seeing is the startup cost of Gradle, which in my
> observations is higher than Ant's. When you get to 60 second
> build/test runs then you should be seeing 53 second ant builds and 60
> second Gradle builds. Whether that is acceptable is up to you.

The problem is that those of use with smaller builds are a bit concerned
with the fact that Gradle is slower than Maven and definitely slower
than Ant.

No matter whether justified or not, whether it can be ameliorated or
not, small builds is where people make comparison judgements, and where
benchmarks are created.  There is also a tendency for detractors of a
technology to obsess about performance issues -- witness the arguments
about Groovy itself, and indeed the hassles SCons is getting.

I think there are more important technical evolutions lined up for 0.7,
and I think they should remain the priority.  However, I think
performance should be reviewed for 0.8.  There was a review previously
and this led to a major rewrite of large tracts of Gradle source from
Groovy to Java, and this successfully slashed run times.

Perhaps a new stage of profiling on a number of projects is in order to
find the "hot spot" and see if there are factors stopping the JIT from
doing its job.  Or perhaps the problem is a class written in Groovy that
needs to be rewritten in Java.  Alternatively there are just
infelicities in the Java that have not been picked up.

Gradle's performance is in the right scale so I don't think there is a
crisis here, so I don't think there needs to be any deep worrying.
However it would be good to get performance closer to Ant and definitely
better than Maven.

To progress this we need to set up an experiment where as many people as
possible execute their build with profiling so as to provide data.  I
can't volunteer to drive this, but if there is anyone out there with
strong profiling experience who can, that would be fantastic.
--
Russel.
=============================================================================
Dr Russel Winder      Partner
                                            xmpp: [e-mail]
Concertant LLP        t: [phone], [phone]
41 Buckmaster Road,   f: [phone]   voip: sip:[e-mail]
London SW11 1EN, UK   m: [phone]   skype: russel_winder

 

[Helmut Denk]

hi peter,

something off-topic ...

i played a little bit with spock today and took
a look at the docs and sourcecode ... it looks
quite promising to me.

i would like to try it against grails-apps. have
you done that already ?

i tried easyb against grails via grails-easyb-plugin  
but that was not very satisfying and since the  grails-
1.1.1 upgrade the easyb-plugin has some more
issues.

gruesse

[Peter Niederwieser]

Hi Helmut,

at the moment I devote all my time and energy to the Spock core. That's why I have yet to investigate how Spock could be used to test Grails applications. If you could provide some feedback, ideas, or code, that would be great! Don't hesitate to post your questions or findings to the Spock forum: http://groups.google.com/group/spockframework (anyone can post, no Google account required)

Cheers,
Peter

[hdockter]

On May 31, 2009, at 9:21 PM, Peter Niederwieser wrote:

>
> Hi,
>
> first of all my congratulations to Gradle 0.6! I'm happy to say that
> compiling and running Spock specifications now works just fine. Also  
> I'm
> impressed with the much improved build script DSL.
>
> I have a HelloSpock project that contains a single Groovy class (a  
> Spock
> specification). What puzzles me a bit is that 'gradle test' takes 10  
> seconds
> to complete. By comparison, 'ant test' takes 3 seconds, and 'mvn  
> test' 8
> seconds.

What version of Maven are you using?

> This is on repeated invocation, i.e. all downloading and
> compilation has already taken place. Do you consider this  
> acceptable? Would
> the difference be less pronounced for bigger projects?

First of all I'm surprised by the seven seconds difference. It would  
be interesting to see which areas are mostly responsible for this  
(startup time, dependency management, testing, ...). We will have a  
look at this. I assume it is less pronounced in larger projects. But  
we have to examine this to be sure.

Having said this, there are many aspects that affect build script  
performance. Where we focus on in 0.7 is to become smarter in regard  
to change detection. For larger builds that might me much more  
important than the startup time, if the latter is the reason for the  
above behavior.

Thanks a lot for checking this out

- Hans

--
Hans Dockter
Gradle Project Manager
http://www.gradle.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[hdockter]

On Jun 3, 2009, at 4:23 PM, Hans Dockter wrote:

>
> On May 31, 2009, at 9:21 PM, Peter Niederwieser wrote:
>
>>
>> Hi,
>>
>> first of all my congratulations to Gradle 0.6! I'm happy to say that
>> compiling and running Spock specifications now works just fine.  
>> Also I'm
>> impressed with the much improved build script DSL.
>>
>> I have a HelloSpock project that contains a single Groovy class (a  
>> Spock
>> specification). What puzzles me a bit is that 'gradle test' takes  
>> 10 seconds
>> to complete. By comparison, 'ant test' takes 3 seconds, and 'mvn  
>> test' 8
>> seconds.
>
> What version of Maven are you using?
>
>> This is on repeated invocation, i.e. all downloading and
>> compilation has already taken place. Do you consider this  
>> acceptable? Would
>> the difference be less pronounced for bigger projects?
>
> First of all I'm surprised by the seven seconds difference. It would  
> be interesting to see which areas are mostly responsible for this  
> (startup time, dependency management, testing, ...). We will have a  
> look at this. I assume it is less pronounced in larger projects. But  
> we have to examine this to be sure.

After all it does not look that bad. It is not a fundamental issue.  
Here are some numbers:

Gradle
time gradle clean compile
real    0m3.188s
user    0m2.909s
sys     0m0.177s

Ant 1.7
time ant clean compile
real    0m2.204s
user    0m1.841s
sys     0m0.239s

Maven 2.0.9
time mvn clean compile
real    0m2.648s
user    0m2.351s
sys     0m0.189s

So you see that there is a small Groovy startup time cost but I think  
this is no problem.

The tests are taking so much longer for two reasons. One is that the  
compileTests always checks for the newest Spock snapshot in the remote  
repository. Which is different to the Maven behavior which checks by  
default once a day. Not using a snapshot this yields:

time gradle clean test
real    0m7.668s
user    0m6.839s
sys     0m0.564s

time ant clean test
real    0m3.405s
user    0m2.893s
sys     0m0.364s

time mvn clean test
real    0m7.752s
user    0m7.044s
sys     0m0.487s

Which means Gradle is as fast as Maven but significantly slower than  
Ant. Tom has added a lot of auto test detection stuff in 0.6 I'm  
wondering if this is related to that. Even just compiling the tests  
take surprisingly long.

I have filed a Jira: http://jira.codehaus.org/browse/GRADLE-514

- Hans

--
Hans Dockter
Gradle Project Manager
http://www.gradle.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[hdockter]

> The tests are taking so much longer for two reasons. One is that the  
> compileTests always checks for the newest Spock snapshot in the  
> remote repository. Which is different to the Maven behavior which  
> checks by default once a day.


You could add a cache resolver when a certain property is set. That  
way you would avoid the remote lookup. It would be nice if Gradle  
could provide some out of the box functionality for this. But this is  
not the case yet.

See also: http://markmail.org/message/lhrnb7lcqt527j4k

<snip>

- Hans

--
Hans Dockter
Gradle Project Manager
http://www.gradle.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Tom Eyckmans]

2009/6/3 Hans Dockter <mail@...>

On Jun 3, 2009, at 4:23 PM, Hans Dockter wrote:

On May 31, 2009, at 9:21 PM, Peter Niederwieser wrote:

Hi,

first of all my congratulations to Gradle 0.6! I'm happy to say that
compiling and running Spock specifications now works just fine. Also I'm
impressed with the much improved build script DSL.

I have a HelloSpock project that contains a single Groovy class (a Spock
specification). What puzzles me a bit is that 'gradle test' takes 10 seconds
to complete. By comparison, 'ant test' takes 3 seconds, and 'mvn test' 8
seconds.

What version of Maven are you using?

This is on repeated invocation, i.e. all downloading and
compilation has already taken place. Do you consider this acceptable? Would
the difference be less pronounced for bigger projects?

First of all I'm surprised by the seven seconds difference. It would be interesting to see which areas are mostly responsible for this (startup time, dependency management, testing, ...). We will have a look at this. I assume it is less pronounced in larger projects. But we have to examine this to be sure.

After all it does not look that bad. It is not a fundamental issue. Here are some numbers:

Gradle
time gradle clean compile
real    0m3.188s
user    0m2.909s
sys     0m0.177s

Ant 1.7
time ant clean compile
real    0m2.204s
user    0m1.841s
sys     0m0.239s

Maven 2.0.9
time mvn clean compile
real    0m2.648s
user    0m2.351s
sys     0m0.189s

So you see that there is a small Groovy startup time cost but I think this is no problem.

The tests are taking so much longer for two reasons. One is that the compileTests always checks for the newest Spock snapshot in the remote repository. Which is different to the Maven behavior which checks by default once a day. Not using a snapshot this yields:

time gradle clean test
real    0m7.668s
user    0m6.839s
sys     0m0.564s

time ant clean test
real    0m3.405s
user    0m2.893s
sys     0m0.364s

time mvn clean test
real    0m7.752s
user    0m7.044s
sys     0m0.487s

Which means Gradle is as fast as Maven but significantly slower than Ant. Tom has added a lot of auto test detection stuff in 0.6 I'm wondering if this is related to that.

It would surprise me if the test class detection would be the cause of this. 

For Gradle itself it only takes

- <5 secs to create the package <-> jars index

- and less than a second to detect all the tests (2000+ tests) .

 

Even just compiling the tests take surprisingly long.

Test class detection is done in the test task itself so this is done after test class compilation. Not sure what could be causing this I'll need to have a closer look.

I have filed a Jira: http://jira.codehaus.org/browse/GRADLE-514

- Hans

--
Hans Dockter
Gradle Project Manager
http://www.gradle.org


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

  http://xircles.codehaus.org/manage_email