Java Plugin Task Dependencies

[Steve Appling]

My confusion over how to determine the dependencies for optimizing tasks was not
actually a problem with the API to determine dependencies, but a
misunderstanding of how the dependencies were declared.  I do think, however,
that it may be a good idea to make some changes in the dependency relationships.

The current dependency chain established by the java plugin does not really
specify which tasks need as their input, the output of another task.  The
current dependency chain is just establishes a "lifecycle" of tasks that will be
executed in a particular order.  For example, processTestResources currently
depends on compile, not because the processTestResources task is really
dependent on compile, but just to fit the task in a lifecycle order.

I would rather see a task A really only depend on task B if task A will consume
the the output of task B in some manner.  I think that this does really help
with automatic optimization.  This also fits in nicely with Adam's ideas about a
dependency chain based off of input/output artifacts.

With that in mind, I would like to see the dependencies for the java plugin
tasks changed as shown in the attached image.

--
Steve Appling
Automated Logic Research Team


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Adam Murdoch-2]

Steve Appling wrote:

My confusion over how to determine the dependencies for optimizing tasks was not actually a problem with the API to determine dependencies, but a misunderstanding of how the dependencies were declared.  I do think, however, that it may be a good idea to make some changes in the dependency relationships.

The current dependency chain established by the java plugin does not really specify which tasks need as their input, the output of another task.  The current dependency chain is just establishes a "lifecycle" of tasks that will be executed in a particular order.  For example, processTestResources currently depends on compile, not because the processTestResources task is really dependent on compile, but just to fit the task in a lifecycle order.

I would rather see a task A really only depend on task B if task A will consume the the output of task B in some manner.  I think that this does really help with automatic optimization.  This also fits in nicely with Adam's ideas about a dependency chain based off of input/output artifacts.

With that in mind, I would like to see the dependencies for the java plugin tasks changed as shown in the attached image.

This is a good idea, with a few tweaks:

- jar should depend on compile as well as processResources.
- jar should probably not depend on test. I'm not sure what we should do with it. Perhaps libs or upload* should depend on test instead.
- we should get rid of init. There are plenty of other points in the build lifecycle where you can do this.

---

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
  

[John Murph]

On Wed, Jul 1, 2009 at 5:37 PM, Adam Murdoch <a@...> wrote:

- jar should depend on compile as well as processResources.

 

+1  (I'm not sure why I didn't think of this before Steve posted.)

- jar should probably not depend on test. I'm not sure what we should do with it. Perhaps libs or upload* should depend on test instead.

I still like the idea of "the tests should pass".  I'm not sure I know what that means with respect to the task dependencies, however.  My reason (so I don't speak for Steve) for leaving jar depending on tests was so that we didn't change existing behavior too much.

- we should get rid of init. There are plenty of other points in the build lifecycle where you can do this.

 

+1

--
John Murph
Automated Logic Research Team

[Steve Appling]

Adam Murdoch wrote:

>
>
> Steve Appling wrote:
>> My confusion over how to determine the dependencies for optimizing
>> tasks was not actually a problem with the API to determine
>> dependencies, but a misunderstanding of how the dependencies were
>> declared.  I do think, however, that it may be a good idea to make
>> some changes in the dependency relationships.
>>
>> The current dependency chain established by the java plugin does not
>> really specify which tasks need as their input, the output of another
>> task.  The current dependency chain is just establishes a "lifecycle"
>> of tasks that will be executed in a particular order.  For example,
>> processTestResources currently depends on compile, not because the
>> processTestResources task is really dependent on compile, but just to
>> fit the task in a lifecycle order.
>>
>> I would rather see a task A really only depend on task B if task A
>> will consume the the output of task B in some manner.  I think that
>> this does really help with automatic optimization.  This also fits in
>> nicely with Adam's ideas about a dependency chain based off of
>> input/output artifacts.
>>
>> With that in mind, I would like to see the dependencies for the java
>> plugin tasks changed as shown in the attached image.
>
> This is a good idea, with a few tweaks:
>
> - jar should depend on compile as well as processResources.

You're right, that's what I intended, but that line didn't make it from my white
board to the drawing :(

> - jar should probably not depend on test. I'm not sure what we should do
> with it. Perhaps libs or upload* should depend on test instead.
I'm not sure what your intent is here.  I think you still want to run tests as
part of making the artifact.  What would be accomplished by moving the test
dependency up to libs?  Currently the uploadDefaultInternal task used to resolve
dependencies on other projects is associated with the jar (not the libs) task,
so moving the dependency up to libs would keep tests from running in other
dependent projects.  Is this your intent?

Thinking through this is helping me understand some of your original ideas for
more of a I/O centered dependency declaration.  I actually see two different
types of dependencies here.  One is a producer/consumer type of dependency where
a task consumes the output of another task.  The other is just a task ordering
type of dependency.  For example, jar has a producer/consumer dependency on both
processResources and compile, but has an ordering dependency on test.  We simply
want the tests to run successfully before the jar is made, but there are no
outputs or side effects from the test task that are consumed by the jar task.
This distinction helps with the optimization.  You want to run a task if any of
it's producer/consumer dependencies didWork.  You don't really care if the
ordering dependencies didWork.

I will make a patch with the changes to dependency as shown in the diagram (with
Adam's correction).  I would, however, like to see more work on this other way
of expressing task dependencies in a future version (perhaps 0.8).  I can
visualize this a little better now - a ProducerConsumer dependency is more like
a new specialized type of configuration.

> - we should get rid of init. There are plenty of other points in the
> build lifecycle where you can do this.
Great.

--
Steve Appling
Automated Logic Research Team


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Adam Murdoch-2]

Steve Appling wrote:

>
>
> Adam Murdoch wrote:
>>
>>
>> Steve Appling wrote:
>>> My confusion over how to determine the dependencies for optimizing
>>> tasks was not actually a problem with the API to determine
>>> dependencies, but a misunderstanding of how the dependencies were
>>> declared.  I do think, however, that it may be a good idea to make
>>> some changes in the dependency relationships.
>>>
>>> The current dependency chain established by the java plugin does not
>>> really specify which tasks need as their input, the output of
>>> another task.  The current dependency chain is just establishes a
>>> "lifecycle" of tasks that will be executed in a particular order.  
>>> For example, processTestResources currently depends on compile, not
>>> because the processTestResources task is really dependent on
>>> compile, but just to fit the task in a lifecycle order.
>>>
>>> I would rather see a task A really only depend on task B if task A
>>> will consume the the output of task B in some manner.  I think that
>>> this does really help with automatic optimization.  This also fits
>>> in nicely with Adam's ideas about a dependency chain based off of
>>> input/output artifacts.
>>>
>>> With that in mind, I would like to see the dependencies for the java
>>> plugin tasks changed as shown in the attached image.
>>
>> This is a good idea, with a few tweaks:
>>
>> - jar should depend on compile as well as processResources.
> You're right, that's what I intended, but that line didn't make it
> from my white
> board to the drawing :(
>
>> - jar should probably not depend on test. I'm not sure what we should
>> do with it. Perhaps libs or upload* should depend on test instead.
> I'm not sure what your intent is here.  I think you still want to run
> tests as part of making the artifact.

The problem is that sometimes you do, and sometimes you don't. It really
depends on what you intend to do with the artifact after it's built. It
feels like the decision of which tests, if any, should be run belongs
with something else; something higher level than the jar task or the
java plugin. For example, you could model this using build types, where,
say, a pre-checkin or CI build would run the unit tests for all
projects, a local ad hoc build would not run any tests, and a developer
build would run the unit tests for those projects with modified source
plus their dependent projects.

Given that we currently don't have a good way to figure out the intended
use for the artifact, I think there might be some better places for the
dependency on 'test':
- On upload<conf>Internal. This way, the artifacts are tested before
they are available outside the project.
- Or, on upload<conf>. This way, the artifacts are tested before they
are available outside the build.
- Or, a new lifecycle task, called, say, 'build'.

> What would be accomplished by moving the test dependency up to libs?  
> Currently the uploadDefaultInternal task used to resolve dependencies
> on other projects is associated with the jar (not the libs) task, so
> moving the dependency up to libs would keep tests from running in
> other dependent projects.  Is this your intent?
>
> Thinking through this is helping me understand some of your original
> ideas for more of a I/O centered dependency declaration.  I actually
> see two different types of dependencies here.  One is a
> producer/consumer type of dependency where a task consumes the output
> of another task.  The other is just a task ordering type of
> dependency.  For example, jar has a producer/consumer dependency on
> both processResources and compile, but has an ordering dependency on
> test.  We simply want the tests to run successfully before the jar is
> made, but there are no outputs or side effects from the test task that
> are consumed by the jar task. This distinction helps with the
> optimization.  You want to run a task if any of it's producer/consumer
> dependencies didWork.  You don't really care if the ordering
> dependencies didWork.
>
> I will make a patch with the changes to dependency as shown in the
> diagram (with Adam's correction).  I would, however, like to see more
> work on this other way of expressing task dependencies in a future
> version (perhaps 0.8).

The task optimisation discussion so far has been quite abstract. I think
things will be a bit easier if we go concrete. So, I think the next
steps are to 1) get the Java/Groovy/War plugins to wire up the
onlyIf()/didWork()/change detection stuff, and 2) introduce the
SourceDir/ClassesDir/TestSuite concepts, and 3) then think more about
extracting some general patterns once we've seen how these shape up.


Adam


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email