Proposal after-the-fact: Experimental multithreading support

[dfabulich]

On Friday I was playing cowboy with my experimental thread support, but
here's a more formal proposal around parallel project support in Maven
3.0.

OUTSTANDING ISSUES

In my earlier revision 833566, I attempted to fix MNG-3004:

"Allow build lifecycle to execute projects in parallel"
http://jira.codehaus.org/browse/MNG-3004

The general consensus around this bug (which has 25 votes) is that
multithreading support can't work correctly right now because of MNG-2802:

"Concurrent-safe access to local Maven repository"
http://jira.codehaus.org/browse/MNG-2802

Several people have remarked in comments to MNG-3004 that it's appropriate
to try to fix MNG-3004 (parallel projects) without fixing MNG-2802
(thread-safe local repo).  In doing so, we'll allow users to optionally
enable building multiple projects simultaneously.  This is worth doing,
because it can be tested in the wild, and because for some users, it will
be immediately useful (e.g. especially if you use --offline mode).

I agree with these comments, and attempted to implement them in revision
833566.

MY PROPOSAL

I attempted to check in the code in time for 3.0-alpha-3 a few days ago.
That code was rolled back over the weekend and now lives in the MNG-3004
branch, because it broke integration tests.  All integration tests now
pass on my machine with the MNG-3004 branch, so I'd like to land it back
in trunk again and re-cut 3.0-alpha-3 with this additional feature.

As I stated on Friday, using the MNG-3004 branch, you can now do this:

   mvn install -Dmaven.threads.experimental=4

It works on my machine.  If it works for you great; if it doesn't, we can
figure out what's wrong together.

WHY ALPHA-3?

I'd like to land it back in alpha-3 because:
* alpha-2 was in February; I don't want to wait for even four months to
start testing this in the wild
* I did check it in before alpha-3 was released, but it was rolled back
quickly before the release vote was called
* Otherwise I'd want to try to release alpha-4 immediately after alpha-3,
which seems wasteful

A NOTE ABOUT DEFAULTS

Currently the code defaults to maven.threads.experimental=1.  This fires
up a single executor thread who does all work; the main thread joins to
wait for it to finish.  It's also possible to set
maven.threads.experimental=0, in which case everything is done on the main
thread, just like in alpha-2.

If there is consensus, I'd be happy to set the default value to 0, though
that increases the risk that the multithreaded code will get less coverage
in the wild.  I think leaving the value at 1 is a good compromise between
avoiding complex threading issues and exercising tricky code.

WHAT'S IN THE FUTURE?

Post alpha-3 I'm keen on implementing a settings.xml option allowing users
to configure their local repository layout.  This will allow users to
choose an alternate implementation that is thread-safe.

Additionally, I think that this feature needs more tooling to make
parallel projects understandable.  The logger should be enhanced to
identify which projects are logging which messages, and the final output
of Maven should report more clearly which projects had warning/error
messages, and how many such messages were reported.

What do you say?

-Dan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Wendy Smoak-3]

On Mon, Nov 9, 2009 at 8:02 PM, Dan Fabulich <dan@...> wrote:
> I attempted to check in the code in time for 3.0-alpha-3 a few days ago.
> That code was rolled back over the weekend and now lives in the MNG-3004
> branch, because it broke integration tests.  All integration tests now pass
> on my machine with the MNG-3004 branch, so I'd like to land it back in trunk
> again and re-cut 3.0-alpha-3 with this additional feature.

I think we should let the alpha-3 vote continue as-is with however
many months of changes have accumulated, and you can do alpha-4 with
this new feature.

--
Wendy

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Jason van Zyl-2]

On 2009-11-10, at 4:02 AM, Dan Fabulich wrote:

>
> On Friday I was playing cowboy with my experimental thread support,  
> but here's a more formal proposal around parallel project support in  
> Maven 3.0.
>
> OUTSTANDING ISSUES
>
> In my earlier revision 833566, I attempted to fix MNG-3004:
>
> "Allow build lifecycle to execute projects in parallel"
> http://jira.codehaus.org/browse/MNG-3004
>
> The general consensus around this bug (which has 25 votes) is that  
> multithreading support can't work correctly right now because of  
> MNG-2802:
>
> "Concurrent-safe access to local Maven repository"
> http://jira.codehaus.org/browse/MNG-2802
>
> Several people have remarked in comments to MNG-3004 that it's  
> appropriate to try to fix MNG-3004 (parallel projects) without  
> fixing MNG-2802 (thread-safe local repo).  In doing so, we'll allow  
> users to optionally enable building multiple projects  
> simultaneously.  This is worth doing, because it can be tested in  
> the wild, and because for some users, it will be immediately useful  
> (e.g. especially if you use --offline mode).
>
> I agree with these comments, and attempted to implement them in  
> revision 833566.
>
> MY PROPOSAL
>
> I attempted to check in the code in time for 3.0-alpha-3 a few days  
> ago. That code was rolled back over the weekend and now lives in the  
> MNG-3004 branch, because it broke integration tests.  All  
> integration tests now pass on my machine with the MNG-3004 branch,  
> so I'd like to land it back in trunk again and re-cut 3.0-alpha-3  
> with this additional feature.
>

-1

It's not going in that fast. Sorry, but passing all the existing ITs  
is the first step and then there is ITs for the feature you multi-
threading feature itself. We simply don't rely on existing ITs anymore  
to make sure things work, you must have ITs for this feature itself.  
The trunk is now critical for M2Eclipse and anyone else who is  
embedding and new features just aren't going in without adequate ITs.  
You're not experimenting on users. I'm not trying to detract from any  
of the work that you're doing but it's not going in over the course of  
a week, and especially not without ITs.

I also don't particularly think it's a good idea without actually  
thinking about the impact of not doing MNG-2802.

> As I stated on Friday, using the MNG-3004 branch, you can now do this:
>
>  mvn install -Dmaven.threads.experimental=4
>
> It works on my machine.  If it works for you great; if it doesn't,  
> we can figure out what's wrong together.
>
> WHY ALPHA-3?
>
> I'd like to land it back in alpha-3 because:
> * alpha-2 was in February; I don't want to wait for even four months  
> to start testing this in the wild
> * I did check it in before alpha-3 was released, but it was rolled  
> back quickly before the release vote was called
> * Otherwise I'd want to try to release alpha-4 immediately after  
> alpha-3, which seems wasteful
>

I think at this point we've fixing almost everything. So a weekly  
alpha would be great, and if you want to get involved with those even  
better. But this stuff is not going into the alpha-3 that has been  
staged now. We can schedule an alpha-4 for next Monday, provided the  
code is merged and has adequate ITs. I don't see any reason why we  
can't do weekly alphas.

> A NOTE ABOUT DEFAULTS
>
> Currently the code defaults to maven.threads.experimental=1.  This  
> fires up a single executor thread who does all work; the main thread  
> joins to wait for it to finish.  It's also possible to set  
> maven.threads.experimental=0, in which case everything is done on  
> the main thread, just like in alpha-2.
>
> If there is consensus, I'd be happy to set the default value to 0,  
> though that increases the risk that the multithreaded code will get  
> less coverage in the wild.  I think leaving the value at 1 is a good  
> compromise between avoiding complex threading issues and exercising  
> tricky code.
>
> WHAT'S IN THE FUTURE?
>
> Post alpha-3 I'm keen on implementing a settings.xml option allowing  
> users to configure their local repository layout.  This will allow  
> users to choose an alternate implementation that is thread-safe.

Why would the layout have anything to do with thread safety? It should  
just be threadsafe don't you think?

>
> Additionally, I think that this feature needs more tooling to make  
> parallel projects understandable.  The logger should be enhanced to  
> identify which projects are logging which messages, and the final  
> output of Maven should report more clearly which projects had  
> warning/error messages, and how many such messages were reported.
>
> What do you say?

All sounds good and the framework has been laid in the ITs where they  
can be run with an embedded version of Maven so you can test all sorts  
of things now where the same instance of Maven can be used across the  
ITs which will allow you to find many threading and share state  
problems. Really I would like to see this work fleshed out so that we  
can adequately know we're doing the right thing.

This is really much of what Benjamin and I have been doing is laying  
the foundation work so that folks can add features safely, but you  
just can't whack things into trunk anymore and have the *current* ITs  
be the only criterion for being adequate.

>
> -Dan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@...
> For additional commands, e-mail: dev-help@...
>

Thanks,

Jason

----------------------------------------------------------
Jason van Zyl
Founder,  Apache Maven
http://twitter.com/jvanzyl
----------------------------------------------------------


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[dfabulich]

Jason van Zyl wrote:

> It's not going in that fast.

Fair enough; that seems to be the consensus.

> Sorry, but passing all the existing ITs is the first step and then there
> is ITs for the feature you multi-threading feature itself.

I've got a couple of ITs for MNG-3004 parallel projects on my machine now
(just for --fail-never, --fail-fast, --fail-at-end); I'll check them into
trunk/core-it-suite when the MNG-3004 branch lands.  (Since I'm not sure
when that is I don't know what version range to apply yet.)

> [snip forward]

> I also don't particularly think it's a good idea without actually
> thinking about the impact of not doing MNG-2802.

I presume you don't mean just "thinking" about the impact, but testing the
impact with ITs.

I've already tried temporarily using multithreaded code by default and
running the ITs on my machine; I think that's a reasonable impact test.
Is there more testing you have in mind?  If so, what?

> [snip backward]

> You're not experimenting on users.

Just to clarify, do you object to the very idea of having experimental
features that are disabled by default?

I can pretty well guarantee the safety of the code when the feature is
off, but it will be a lot of work to guarantee that the feature is safe
when enabled.  (Indeed, we know it isn't safe right now, due to the
un-thread-safe local repo.)

But I think that's OK; a certain kind of user will want to type
"-Dmaven.threads.experimental=8" just to see what happens.  That kind of
user will want to be part of the experiment.

Does that sound OK?

>> WHAT'S IN THE FUTURE?
>>
>> Post alpha-3 I'm keen on implementing a settings.xml option allowing users
>> to configure their local repository layout.  This will allow users to
>> choose an alternate implementation that is thread-safe.
>
> Why would the layout have anything to do with thread safety? It should just
> be threadsafe don't you think?

There isn't much commentary in http://jira.codehaus.org/browse/MNG-2802 
but the bit of discussion that is there suggests that the best solution
for MNG-2802 would be to implement Brett's proposal from last year:
http://docs.codehaus.org/display/MAVEN/Local+repository+separation

That seems right to me.

It sounds like you're thinking that we shouldn't do this, but should
instead just fix the locking behavior in the existing layout?  Why so?

-Dan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[dfabulich]

Jason van Zyl wrote:

> The trunk is now critical for M2Eclipse and anyone else who is embedding
> and new features just aren't going in without adequate ITs.

m2eclipse depends on the Maven 3.0-SNAPSHOT trunk?

Does that mean that changes that we make in the Apache trunk can
automatically break people's Eclipse builds out in the wild, even without
an official release?

That doesn't seem good.  Can we break this dependency once alpha-3 comes
out?

-Dan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Jorg Heymans-3]

Hi Dan,

Would the multithreading feature make it easier or harder to implement
something like [1], ie distribute the different modules of a reactor
build across different machines ? Or is it completely unrelated you
think ?

Thanks,
Jorg Heymans

[1] http://n4.nabble.com/New-plugin-to-distribute-maven-jobs-steps-td585138.html#a585138

On Tue, Nov 10, 2009 at 4:02 AM, Dan Fabulich <dan@...> wrote:

>
> On Friday I was playing cowboy with my experimental thread support, but
> here's a more formal proposal around parallel project support in Maven 3.0.
>
> OUTSTANDING ISSUES
>
> In my earlier revision 833566, I attempted to fix MNG-3004:
>
> "Allow build lifecycle to execute projects in parallel"
> http://jira.codehaus.org/browse/MNG-3004
>
> The general consensus around this bug (which has 25 votes) is that
> multithreading support can't work correctly right now because of MNG-2802:
>
> "Concurrent-safe access to local Maven repository"
> http://jira.codehaus.org/browse/MNG-2802
>
> Several people have remarked in comments to MNG-3004 that it's appropriate
> to try to fix MNG-3004 (parallel projects) without fixing MNG-2802
> (thread-safe local repo).  In doing so, we'll allow users to optionally
> enable building multiple projects simultaneously.  This is worth doing,
> because it can be tested in the wild, and because for some users, it will be
> immediately useful (e.g. especially if you use --offline mode).
>
> I agree with these comments, and attempted to implement them in revision
> 833566.
>
> MY PROPOSAL
>
> I attempted to check in the code in time for 3.0-alpha-3 a few days ago.
> That code was rolled back over the weekend and now lives in the MNG-3004
> branch, because it broke integration tests.  All integration tests now pass
> on my machine with the MNG-3004 branch, so I'd like to land it back in trunk
> again and re-cut 3.0-alpha-3 with this additional feature.
>
> As I stated on Friday, using the MNG-3004 branch, you can now do this:
>
>  mvn install -Dmaven.threads.experimental=4
>
> It works on my machine.  If it works for you great; if it doesn't, we can
> figure out what's wrong together.
>
> WHY ALPHA-3?
>
> I'd like to land it back in alpha-3 because:
> * alpha-2 was in February; I don't want to wait for even four months to
> start testing this in the wild
> * I did check it in before alpha-3 was released, but it was rolled back
> quickly before the release vote was called
> * Otherwise I'd want to try to release alpha-4 immediately after alpha-3,
> which seems wasteful
>
> A NOTE ABOUT DEFAULTS
>
> Currently the code defaults to maven.threads.experimental=1.  This fires up
> a single executor thread who does all work; the main thread joins to wait
> for it to finish.  It's also possible to set maven.threads.experimental=0,
> in which case everything is done on the main thread, just like in alpha-2.
>
> If there is consensus, I'd be happy to set the default value to 0, though
> that increases the risk that the multithreaded code will get less coverage
> in the wild.  I think leaving the value at 1 is a good compromise between
> avoiding complex threading issues and exercising tricky code.
>
> WHAT'S IN THE FUTURE?
>
> Post alpha-3 I'm keen on implementing a settings.xml option allowing users
> to configure their local repository layout.  This will allow users to choose
> an alternate implementation that is thread-safe.
>
> Additionally, I think that this feature needs more tooling to make parallel
> projects understandable.  The logger should be enhanced to identify which
> projects are logging which messages, and the final output of Maven should
> report more clearly which projects had warning/error messages, and how many
> such messages were reported.
>
> What do you say?
>
> -Dan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@...
> For additional commands, e-mail: dev-help@...
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[dfabulich]

Jorg Heymans wrote:

> Would the multithreading feature make it easier or harder to implement
> something like [1], ie distribute the different modules of a reactor
> build across different machines ? Or is it completely unrelated you
> think ?

No harder, but not much easier.  With that said, I do have some ideas for
you.

Take a look at
http://svn.apache.org/repos/asf/maven/maven-3/branches/MNG-3004/maven-core/src/main/java/org/apache/maven/lifecycle/DefaultLifecycleExecutor.java

Currently the unit of work is a CallableBuilder.  In principle you could
imagine serializing it, launching it on a remote machine and gathering the
artifacts locally.  But its members include some objects that may be very
tricky to serialize, including MavenSession and MavenProject.  This seems
like a challenge.

Also, in the thread you linked, Kohsuke points out a related problem that
will make your job hard.  Today, Maven treats compiled artifacts in a
multi-module build very differently from artifacts built one at a time.

Suppose you have two projects X and Y where X depends on Y. X --> Y

If you use Maven to run a build in the Y directory, then launch a second
Maven process to run a build in the X directory, X will depend on Y.jar in
the local repository.

But if you do a reactor build from the root and build both projects at
once, X will depend directly on ../Y/target/classes; the artifact will be
resolved from the reactor.

This behavior is required so you can run "mvn compile" from the root; no
jar is created when you do this, so there's no way for X to depend on
Y.jar.  If you're running "mvn install" from the root, this behavior is
probably unnecessary.

As Kohsuke points out, you'd have to do something rather tricky for Maven
to resolve artifacts from the reactor when the artifacts are on another
machine!

But that leads me to think that the moral of the story is not to run a
multi-machine reactor build, but to simply build the isolated projects on
multiple machines.

Clearly you won't be able to run a "mvn compile" on box 1 and have box 2
consume those classes, so you'll need to do "mvn deploy", deploying those
artifacts to a repository that all of the boxes can see.  (You might use a
repository manager like Nexus or Archiva to handle this.)

It should be possible to orchestrate this by using Maven's
ProjectDependencyGraph.  You can get a copy of the project dependency
graph off of the MavenSession object in Maven 3.0.  Any Maven plugin can
get access to the MavenSession object via the ${session} plugin parameter
expression; just call session.getProjectDependencyGraph().

The ProjectDependencyGraph will give you a list of sorted projects, as
well as the getDownstreamProjects and getUpstreamProjects methods.  You
can start by launching all projects with no upstream projects on N
machines.  When each project succeeds, you can launch all downstream
projects whose upstream projects are finished.  (See the code linked above
for an example of this.)

(Note that you can't just launch all downstream projects, because they may
have unfinished upstream projects.  Suppose X depends on Y and Z, and Y
finishes first.  You can't launch X yet, because Z isn't done, so just
skip X.  When Z is done, all of X's dependencies are done, so now you can
launch X.)

Hope that helps!

-Dan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[BRIAN FOX-5]

On Tue, Nov 10, 2009 at 3:21 PM, Dan Fabulich <dan@...> wrote:
> Jason van Zyl wrote:
>
>> The trunk is now critical for M2Eclipse and anyone else who is embedding
>> and new features just aren't going in without adequate ITs.
>
> m2eclipse depends on the Maven 3.0-SNAPSHOT trunk?
>

Kinda

> Does that mean that changes that we make in the Apache trunk can
> automatically break people's Eclipse builds out in the wild, even without an
> official release?
>

No. Igor has to bundle them into M2e before it's directly used, so it
could break the next time we update. It's a pseudo-realtime
dependency, and definately wouldn't affect previous releases. Its
about keeping trunk stable.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[dfabulich]

Brian Fox wrote:

>> Does that mean that changes that we make in the Apache trunk can
>> automatically break people's Eclipse builds out in the wild, even without an
>> official release?
>>
>
> No. Igor has to bundle them into M2e before it's directly used, so it
> could break the next time we update. It's a pseudo-realtime
> dependency, and definately wouldn't affect previous releases. Its
> about keeping trunk stable.

Despite my recent blunder, I'm all in favor of keeping trunk stable.  But
Igor should probably just stick to 3.0-alpha-3 now, right?

To put it another way: it's about keeping trunk stable, AND it's about
releasing frequently. ;-)

-Dan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[Igor Fedorenko-4]

Unfortunately m2e will have to wait until alpha-4 due to PLX-444,
assuming alpha-3 is not respun that is. Do not break trunk just yet ;-)

--
Regards,
Igor

Dan Fabulich wrote:

> Brian Fox wrote:
>
>>> Does that mean that changes that we make in the Apache trunk can
>>> automatically break people's Eclipse builds out in the wild, even
>>> without an
>>> official release?
>>>
>>
>> No. Igor has to bundle them into M2e before it's directly used, so it
>> could break the next time we update. It's a pseudo-realtime
>> dependency, and definately wouldn't affect previous releases. Its
>> about keeping trunk stable.
>
> Despite my recent blunder, I'm all in favor of keeping trunk stable.  
> But Igor should probably just stick to 3.0-alpha-3 now, right?
>
> To put it another way: it's about keeping trunk stable, AND it's about
> releasing frequently. ;-)
>
> -Dan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@...
> For additional commands, e-mail: dev-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[dfabulich]

This is now an official proposal here:

http://cwiki.apache.org/confluence/display/MAVEN/Experimental+Multithreading+Support

(I didn't have karma to edit this page until a few minutes ago; you might
need to request karma as well, especially since this the first proposal
in this new location.)

-Dan

Dan Fabulich wrote:

>
> On Friday I was playing cowboy with my experimental thread support, but
> here's a more formal proposal around parallel project support in Maven 3.0.
>
> OUTSTANDING ISSUES
>
> In my earlier revision 833566, I attempted to fix MNG-3004:
>
> "Allow build lifecycle to execute projects in parallel"
> http://jira.codehaus.org/browse/MNG-3004
>
> The general consensus around this bug (which has 25 votes) is that
> multithreading support can't work correctly right now because of MNG-2802:
>
> "Concurrent-safe access to local Maven repository"
> http://jira.codehaus.org/browse/MNG-2802
>
> Several people have remarked in comments to MNG-3004 that it's appropriate to
> try to fix MNG-3004 (parallel projects) without fixing MNG-2802 (thread-safe
> local repo).  In doing so, we'll allow users to optionally enable building
> multiple projects simultaneously.  This is worth doing, because it can be
> tested in the wild, and because for some users, it will be immediately useful
> (e.g. especially if you use --offline mode).
>
> I agree with these comments, and attempted to implement them in revision
> 833566.
>
> MY PROPOSAL
>
> I attempted to check in the code in time for 3.0-alpha-3 a few days ago. That
> code was rolled back over the weekend and now lives in the MNG-3004 branch,
> because it broke integration tests.  All integration tests now pass on my
> machine with the MNG-3004 branch, so I'd like to land it back in trunk again
> and re-cut 3.0-alpha-3 with this additional feature.
>
> As I stated on Friday, using the MNG-3004 branch, you can now do this:
>
>  mvn install -Dmaven.threads.experimental=4
>
> It works on my machine.  If it works for you great; if it doesn't, we can
> figure out what's wrong together.
>
> WHY ALPHA-3?
>
> I'd like to land it back in alpha-3 because:
> * alpha-2 was in February; I don't want to wait for even four months to start
> testing this in the wild
> * I did check it in before alpha-3 was released, but it was rolled back
> quickly before the release vote was called
> * Otherwise I'd want to try to release alpha-4 immediately after alpha-3,
> which seems wasteful
>
> A NOTE ABOUT DEFAULTS
>
> Currently the code defaults to maven.threads.experimental=1.  This fires up a
> single executor thread who does all work; the main thread joins to wait for
> it to finish.  It's also possible to set maven.threads.experimental=0, in
> which case everything is done on the main thread, just like in alpha-2.
>
> If there is consensus, I'd be happy to set the default value to 0, though
> that increases the risk that the multithreaded code will get less coverage in
> the wild.  I think leaving the value at 1 is a good compromise between
> avoiding complex threading issues and exercising tricky code.
>
> WHAT'S IN THE FUTURE?
>
> Post alpha-3 I'm keen on implementing a settings.xml option allowing users to
> configure their local repository layout.  This will allow users to choose an
> alternate implementation that is thread-safe.
>
> Additionally, I think that this feature needs more tooling to make parallel
> projects understandable.  The logger should be enhanced to identify which
> projects are logging which messages, and the final output of Maven should
> report more clearly which projects had warning/error messages, and how many
> such messages were reported.
>
> What do you say?
>
> -Dan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@...
> For additional commands, e-mail: dev-help@...
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...

[BRIAN FOX-5]

This is more a plan to implement the existing multi-threading patch
than an actual proposal. Imo a proposal should describe the
solution...ie how will this multi-threading work, how will it decide
what to build and in what order, etc. I think essentially the details
you Ralph and I briefly discussed at ApacheCon.

On Sat, Nov 14, 2009 at 2:21 PM, Dan Fabulich <dan@...> wrote:

> This is now an official proposal here:
>
> http://cwiki.apache.org/confluence/display/MAVEN/Experimental+Multithreading+Support
>
> (I didn't have karma to edit this page until a few minutes ago; you might
> need to request karma as well, especially since this the first proposal in
> this new location.)
>
> -Dan
>
> Dan Fabulich wrote:
>
>>
>> On Friday I was playing cowboy with my experimental thread support, but
>> here's a more formal proposal around parallel project support in Maven 3.0.
>>
>> OUTSTANDING ISSUES
>>
>> In my earlier revision 833566, I attempted to fix MNG-3004:
>>
>> "Allow build lifecycle to execute projects in parallel"
>> http://jira.codehaus.org/browse/MNG-3004
>>
>> The general consensus around this bug (which has 25 votes) is that
>> multithreading support can't work correctly right now because of MNG-2802:
>>
>> "Concurrent-safe access to local Maven repository"
>> http://jira.codehaus.org/browse/MNG-2802
>>
>> Several people have remarked in comments to MNG-3004 that it's appropriate
>> to try to fix MNG-3004 (parallel projects) without fixing MNG-2802
>> (thread-safe local repo).  In doing so, we'll allow users to optionally
>> enable building multiple projects simultaneously.  This is worth doing,
>> because it can be tested in the wild, and because for some users, it will be
>> immediately useful (e.g. especially if you use --offline mode).
>>
>> I agree with these comments, and attempted to implement them in revision
>> 833566.
>>
>> MY PROPOSAL
>>
>> I attempted to check in the code in time for 3.0-alpha-3 a few days ago.
>> That code was rolled back over the weekend and now lives in the MNG-3004
>> branch, because it broke integration tests.  All integration tests now pass
>> on my machine with the MNG-3004 branch, so I'd like to land it back in trunk
>> again and re-cut 3.0-alpha-3 with this additional feature.
>>
>> As I stated on Friday, using the MNG-3004 branch, you can now do this:
>>
>>  mvn install -Dmaven.threads.experimental=4
>>
>> It works on my machine.  If it works for you great; if it doesn't, we can
>> figure out what's wrong together.
>>
>> WHY ALPHA-3?
>>
>> I'd like to land it back in alpha-3 because:
>> * alpha-2 was in February; I don't want to wait for even four months to
>> start testing this in the wild
>> * I did check it in before alpha-3 was released, but it was rolled back
>> quickly before the release vote was called
>> * Otherwise I'd want to try to release alpha-4 immediately after alpha-3,
>> which seems wasteful
>>
>> A NOTE ABOUT DEFAULTS
>>
>> Currently the code defaults to maven.threads.experimental=1.  This fires
>> up a single executor thread who does all work; the main thread joins to wait
>> for it to finish.  It's also possible to set maven.threads.experimental=0,
>> in which case everything is done on the main thread, just like in alpha-2.
>>
>> If there is consensus, I'd be happy to set the default value to 0, though
>> that increases the risk that the multithreaded code will get less coverage
>> in the wild.  I think leaving the value at 1 is a good compromise between
>> avoiding complex threading issues and exercising tricky code.
>>
>> WHAT'S IN THE FUTURE?
>>
>> Post alpha-3 I'm keen on implementing a settings.xml option allowing users
>> to configure their local repository layout.  This will allow users to choose
>> an alternate implementation that is thread-safe.
>>
>> Additionally, I think that this feature needs more tooling to make
>> parallel projects understandable.  The logger should be enhanced to identify
>> which projects are logging which messages, and the final output of Maven
>> should report more clearly which projects had warning/error messages, and
>> how many such messages were reported.
>>
>> What do you say?
>>
>> -Dan
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@...
>> For additional commands, e-mail: dev-help@...
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@...
> For additional commands, e-mail: dev-help@...
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...