Postponing test compile to after all modules have compiled

[Henrik Thostrup Jensen]

Hi again

Got basic jar dependencies working (thanks Alex), but ran into a
problem with circular dependencies in between projects:

Some of the tests in a module (say A) depends on a module (B), which
requires A to build. So I need to make buildr understand that it
should do something like:

A.compile, B.compile (with A), A.test.compile (with B).

I've grokked at http://incubator.apache.org/buildr/building.html, and
tried setting test.compile.with, but buildr coughs up a circular
dependency detected. Is there any way to make buildr do this?


Also, I'm running into some segfaults sometimes. They are
deterministic, but can often disappear with changing a bit in the
buildfile. Is this a known problem?, I am using version 1.3.3 (and
should I try out the trunk?).

Thanks.

--
   - Henrik

[Daniel Spiewak]

>
> Also, I'm running into some segfaults sometimes. They are
> deterministic, but can often disappear with changing a bit in the
> buildfile. Is this a known problem?, I am using version 1.3.3 (and
> should I try out the trunk?).
>

A segfault is almost certainly a Ruby bug.  Technically, it's possible that
one of Buildr's dependencies is using buggy native code, but that's somewhat
less likely.  Have you tried updating Ruby to the latest 1.8.x version?

Daniel

[Henrik Thostrup Jensen]

2009/1/14 Daniel Spiewak <djspiewak@...>:
>>
>> Also, I'm running into some segfaults sometimes. They are
>> deterministic, but can often disappear with changing a bit in the
>> buildfile. Is this a known problem?, I am using version 1.3.3 (and
>> should I try out the trunk?).
>
> A segfault is almost certainly a Ruby bug.  Technically, it's possible that
> one of Buildr's dependencies is using buggy native code, but that's somewhat
> less likely.  Have you tried updating Ruby to the latest 1.8.x version?

I'm on:
ruby 1.8.7 (2008-08-11 patchlevel 72) [i486-linux] (Ubuntu 8.10)

Is 1.9 likely to make a difference?

Btw. It is not the issue mentioned here:
http://incubator.apache.org/buildr/troubleshooting.html

--
   - Henrik

[Alex Boisvert-3]

The only segfaults I've seen on Ubuntu were cases using JDK 1.6 with RJB.
You can use either JDK 1.5 or switch to JRuby if you want to use JDK 1.6.

(The incompatibility with JDK 1.6 is due to RJB, not Buildr)

alex

On Wed, Jan 14, 2009 at 3:05 PM, Henrik Thostrup Jensen
<thostrup@...>wrote:

> 2009/1/14 Daniel Spiewak <djspiewak@...>:
> >>
> >> Also, I'm running into some segfaults sometimes. They are
> >> deterministic, but can often disappear with changing a bit in the
> >> buildfile. Is this a known problem?, I am using version 1.3.3 (and
> >> should I try out the trunk?).
> >
> > A segfault is almost certainly a Ruby bug.  Technically, it's possible
> that
> > one of Buildr's dependencies is using buggy native code, but that's
> somewhat
> > less likely.  Have you tried updating Ruby to the latest 1.8.x version?
>
> I'm on:
> ruby 1.8.7 (2008-08-11 patchlevel 72) [i486-linux] (Ubuntu 8.10)
>
> Is 1.9 likely to make a difference?
>
> Btw. It is not the issue mentioned here:
> http://incubator.apache.org/buildr/troubleshooting.html
>
> --
>    - Henrik
>

[Ittay Dror-2]

Alex Boisvert wrote:
> The only segfaults I've seen on Ubuntu were cases using JDK 1.6 with RJB.
> You can use either JDK 1.5 or switch to JRuby if you want to use JDK 1.6.
>  
I have run into many segfaults. The reason is usually a deep recursion
which exhausts the execution stack (a ruby stack of 20 methods is
something like 400 c functions stack). Since Rake uses recursion to
invoke prerequisites of tasks and since Buildr uses a lot of task
dependencies (e.g. compile.with project('B')), this happens when the
project tree gets large. As far as I remember this happens on *nix
systems, not Windows.

The way I debug this is by using trace to find where approximately the
crash happens and then turning on method tracing in ruby.

If running on *nix, try increasing the stack limit ('ulimit -s unlimited')

HTH,
Ittay

> (The incompatibility with JDK 1.6 is due to RJB, not Buildr)
>
> alex
>
> On Wed, Jan 14, 2009 at 3:05 PM, Henrik Thostrup Jensen
> <thostrup@...>wrote:
>
>  
>> 2009/1/14 Daniel Spiewak <djspiewak@...>:
>>    
>>>> Also, I'm running into some segfaults sometimes. They are
>>>> deterministic, but can often disappear with changing a bit in the
>>>> buildfile. Is this a known problem?, I am using version 1.3.3 (and
>>>> should I try out the trunk?).
>>>>        
>>> A segfault is almost certainly a Ruby bug.  Technically, it's possible
>>>      
>> that
>>    
>>> one of Buildr's dependencies is using buggy native code, but that's
>>>      
>> somewhat
>>    
>>> less likely.  Have you tried updating Ruby to the latest 1.8.x version?
>>>      
>> I'm on:
>> ruby 1.8.7 (2008-08-11 patchlevel 72) [i486-linux] (Ubuntu 8.10)
>>
>> Is 1.9 likely to make a difference?
>>
>> Btw. It is not the issue mentioned here:
>> http://incubator.apache.org/buildr/troubleshooting.html
>>
>> --
>>    - Henrik
>>
>>    
>
>  

--
Ittay Dror <ittayd@...>
Tikal <http://www.tikalk.com>
Tikal Project <http://tikal.sourceforge.net>

[Henrik Thostrup Jensen]

2009/1/15 Ittay Dror <ittayd@...>:

>
> Alex Boisvert wrote:
>>
>> The only segfaults I've seen on Ubuntu were cases using JDK 1.6 with RJB.
>> You can use either JDK 1.5 or switch to JRuby if you want to use JDK 1.6.
>
> I have run into many segfaults. The reason is usually a deep recursion which
> exhausts the execution stack (a ruby stack of 20 methods is something like
> 400 c functions stack). Since Rake uses recursion to invoke prerequisites of
> tasks and since Buildr uses a lot of task dependencies (e.g. compile.with
> project('B')), this happens when the project tree gets large. As far as I
> remember this happens on *nix systems, not Windows.
>
> The way I debug this is by using trace to find where approximately the crash
> happens and then turning on method tracing in ruby.
>
> If running on *nix, try increasing the stack limit ('ulimit -s unlimited')

I tried this and compared buildr --trace with regular ulimit, it
doesn't get any further :-/.
The dependencies in the project are quite complex.

I am also using JDK 1.6, and tried to downgrade to 1.5. However many
(most) of the libraries I need to use are compiled with 1.6, making a
downgrade very inconvenient.

The segfaults does not appear to come a particular place though:

** Invoke sgas:iterator:build (first_time)
** Invoke sgas:common:test:compile (first_time, not_needed)
zsh: segmentation fault  buildr --trace

** Invoke /home/htj/src/sgas/sgas-git-buildr/buildfile (not_needed)
** Invoke /home/htj/src/sgas/sgas-git-buildr/iterator/target/main/classes
(first_time)
zsh: segmentation fault  buildr --trace

Any ideas, before I bow my head, and go back to maven :-(.

--
   - Henrik

[Alex Boisvert-3]

On Wed, Jan 14, 2009 at 2:35 PM, Henrik Thostrup Jensen
<thostrup@...>wrote:

> Some of the tests in a module (say A) depends on a module (B), which
> requires A to build. So I need to make buildr understand that it
> should do something like:
>
> A.compile, B.compile (with A), A.test.compile (with B).


This cannot be made to work with Buildr at the moment.  You would need to
introduce a new project (C) to break the circular dependency, or merge
things in the same project (everything in A).

(If you're insistent on the file layout with test files in A, then you could
disable the tests in A and map C's test files to A's directory structure.
It's somewhat of a hack but works...)

alex

[Daniel Spiewak]

Could this be an incompatibility with Buildr and ZSH?  It looks to me like
that's where the problem is considering the fact that Buildr moves on after
the segfault.

I use Buildr with Bash.

Daniel

On Thu, Jan 15, 2009 at 3:38 AM, Henrik Thostrup Jensen
<thostrup@...>wrote:

> 2009/1/15 Ittay Dror <ittayd@...>:
> >
> > Alex Boisvert wrote:
> >>
> >> The only segfaults I've seen on Ubuntu were cases using JDK 1.6 with
> RJB.
> >> You can use either JDK 1.5 or switch to JRuby if you want to use JDK
> 1.6.
> >
> > I have run into many segfaults. The reason is usually a deep recursion
> which
> > exhausts the execution stack (a ruby stack of 20 methods is something
> like
> > 400 c functions stack). Since Rake uses recursion to invoke prerequisites
> of
> > tasks and since Buildr uses a lot of task dependencies (e.g. compile.with
> > project('B')), this happens when the project tree gets large. As far as I
> > remember this happens on *nix systems, not Windows.
> >
> > The way I debug this is by using trace to find where approximately the
> crash
> > happens and then turning on method tracing in ruby.
> >
> > If running on *nix, try increasing the stack limit ('ulimit -s
> unlimited')
>
> I tried this and compared buildr --trace with regular ulimit, it
> doesn't get any further :-/.
> The dependencies in the project are quite complex.
>
> I am also using JDK 1.6, and tried to downgrade to 1.5. However many
> (most) of the libraries I need to use are compiled with 1.6, making a
> downgrade very inconvenient.
>
> The segfaults does not appear to come a particular place though:
>
> ** Invoke sgas:iterator:build (first_time)
> ** Invoke sgas:common:test:compile (first_time, not_needed)
> zsh: segmentation fault  buildr --trace
>
> ** Invoke /home/htj/src/sgas/sgas-git-buildr/buildfile (not_needed)
> ** Invoke /home/htj/src/sgas/sgas-git-buildr/iterator/target/main/classes
> (first_time)
> zsh: segmentation fault  buildr --trace
>
> Any ideas, before I bow my head, and go back to maven :-(.
>
> --
>    - Henrik
>

[Henrik Thostrup Jensen]

2009/1/15 Daniel Spiewak <djspiewak@...>:
> Could this be an incompatibility with Buildr and ZSH?  It looks to me like
> that's where the problem is considering the fact that Buildr moves on after
> the segfault.
>
> I use Buildr with Bash.

Those are actually two different segfaults (in order to show that the
segfault doesn't occur in any specific phase).

I also tried with bash, but same result.

I did a quick core dump and back trace with gdb, but got the following
(rather useless) result:

(gdb) bt
#0  0xb7eaccb7 in ?? () from /usr/lib/libruby1.8.so.1.8
#1  0xb7eacd5a in ?? () from /usr/lib/libruby1.8.so.1.8
#2  0xb7eb2781 in ?? () from /usr/lib/libruby1.8.so.1.8
#3  0xb7eb2a1a in ?? () from /usr/lib/libruby1.8.so.1.8
#4  0xb7ead021 in ?? () from /usr/lib/libruby1.8.so.1.8
#5  0xb7eb2781 in ?? () from /usr/lib/libruby1.8.so.1.8
#6  0xb7eb2a1a in ?? () from /usr/lib/libruby1.8.so.1.8
#7  0xb7ead021 in ?? () from /usr/lib/libruby1.8.so.1.8
#8  0xb7eb2781 in ?? () from /usr/lib/libruby1.8.so.1.8
#9  0xb7eb2a1a in ?? () from /usr/lib/libruby1.8.so.1.8
#10 0xb7ead021 in ?? () from /usr/lib/libruby1.8.so.1.8
#11 0xb7eb2781 in ?? () from /usr/lib/libruby1.8.so.1.8
#12 0xb7eb2a1a in ?? () from /usr/lib/libruby1.8.so.1.8
#13 0xb7ead021 in ?? () from /usr/lib/libruby1.8.so.1.8
#14 0xb7eb2781 in ?? () from /usr/lib/libruby1.8.so.1.8
#15 0xb7eb2a1a in ?? () from /usr/lib/libruby1.8.so.1.8
#16 0xb7ead021 in ?? () from /usr/lib/libruby1.8.so.1.8
#17 0xb7eb2781 in ?? () from /usr/lib/libruby1.8.so.1.8
#18 0xb7eb2a1a in ?? () from /usr/lib/libruby1.8.so.1.8
#19 0xb7ead021 in ?? () from /usr/lib/libruby1.8.so.1.8
#20 0xb7eb2781 in ?? () from /usr/lib/libruby1.8.so.1.8
#21 0xb7eb2a1a in ?? () from /usr/lib/libruby1.8.so.1.8
#22 0xb7ead141 in ?? () from /usr/lib/libruby1.8.so.1.8
#23 0xb7eae0a2 in ?? () from /usr/lib/libruby1.8.so.1.8
#24 0xb7eb2781 in ?? () from /usr/lib/libruby1.8.so.1.8
#25 0xb7eb2a1a in ?? () from /usr/lib/libruby1.8.so.1.8
#26 0xb7ead021 in ?? () from /usr/lib/libruby1.8.so.1.8
#27 0xb7eb2781 in ?? () from /usr/lib/libruby1.8.so.1.8
#28 0xb7eb2a1a in ?? () from /usr/lib/libruby1.8.so.1.8
#29 0xb7ead021 in ?? () from /usr/lib/libruby1.8.so.1.8
#30 0xb7eae294 in ?? () from /usr/lib/libruby1.8.so.1.8
#31 0xb7eb2781 in ?? () from /usr/lib/libruby1.8.so.1.8
#32 0xb7eb2a1a in ?? () from /usr/lib/libruby1.8.so.1.8
#33 0xb7ead021 in ?? () from /usr/lib/libruby1.8.so.1.8
#34 0xb7eb0dbc in ?? () from /usr/lib/libruby1.8.so.1.8
#35 0xb7ebc847 in rb_yield () from /usr/lib/libruby1.8.so.1.8
#36 0xb7e8cf91 in rb_ary_each () from /usr/lib/libruby1.8.so.1.8
[---- much more ----]

Which doesn't reveal much. The stack level was 564 levels deep though.

--
   - Henrik

[Assaf Arkin]

On Thu, Jan 15, 2009 at 1:38 AM, Henrik Thostrup Jensen
<thostrup@...> wrote:

> 2009/1/15 Ittay Dror <ittayd@...>:
>>
>> Alex Boisvert wrote:
>>>
>>> The only segfaults I've seen on Ubuntu were cases using JDK 1.6 with RJB.
>>> You can use either JDK 1.5 or switch to JRuby if you want to use JDK 1.6.
>>
>> I have run into many segfaults. The reason is usually a deep recursion which
>> exhausts the execution stack (a ruby stack of 20 methods is something like
>> 400 c functions stack). Since Rake uses recursion to invoke prerequisites of
>> tasks and since Buildr uses a lot of task dependencies (e.g. compile.with
>> project('B')), this happens when the project tree gets large. As far as I
>> remember this happens on *nix systems, not Windows.
>>
>> The way I debug this is by using trace to find where approximately the crash
>> happens and then turning on method tracing in ruby.
>>
>> If running on *nix, try increasing the stack limit ('ulimit -s unlimited')
>
> I tried this and compared buildr --trace with regular ulimit, it
> doesn't get any further :-/.
> The dependencies in the project are quite complex.
>
> I am also using JDK 1.6, and tried to downgrade to 1.5. However many
> (most) of the libraries I need to use are compiled with 1.6, making a
> downgrade very inconvenient.
>
> The segfaults does not appear to come a particular place though:
>
> ** Invoke sgas:iterator:build (first_time)
> ** Invoke sgas:common:test:compile (first_time, not_needed)
> zsh: segmentation fault  buildr --trace
>
> ** Invoke /home/htj/src/sgas/sgas-git-buildr/buildfile (not_needed)
> ** Invoke /home/htj/src/sgas/sgas-git-buildr/iterator/target/main/classes
> (first_time)
> zsh: segmentation fault  buildr --trace
>
> Any ideas, before I bow my head, and go back to maven :-(.

Buildr for JRuby.

I had similar issue under Ubuntu, Ruby and JDK 1.6. If I ran a task
that took a while, say a minute, Buildr would seg fault at some point,
not always the same point. No such problem with Ruby/1.5 or JRuby/1.6,
so I would recommend switching to JRuby.

Assaf

>
> --
>   - Henrik
>

[Henrik Thostrup Jensen]

2009/1/15 Assaf Arkin <arkin@...>:

> On Thu, Jan 15, 2009 at 1:38 AM, Henrik Thostrup Jensen
>>
>> Any ideas, before I bow my head, and go back to maven :-(.
>
> Buildr for JRuby.
>
> I had similar issue under Ubuntu, Ruby and JDK 1.6. If I ran a task
> that took a while, say a minute, Buildr would seg fault at some point,
> not always the same point. No such problem with Ruby/1.5 or JRuby/1.6,
> so I would recommend switching to JRuby.

Switching to JRuby fixed the segfaults, and I now have a working build.

Is a bit sluggish tough (10 seconds for clean). Is there a way to improve this?

I will find a way to work around the semi-circular dependency for now.


--
   - Henrik

[Assaf Arkin]

On Thu, Jan 15, 2009 at 9:49 AM, Henrik Thostrup Jensen
<thostrup@...> wrote:

> 2009/1/15 Assaf Arkin <arkin@...>:
>> On Thu, Jan 15, 2009 at 1:38 AM, Henrik Thostrup Jensen
>>>
>>> Any ideas, before I bow my head, and go back to maven :-(.
>>
>> Buildr for JRuby.
>>
>> I had similar issue under Ubuntu, Ruby and JDK 1.6. If I ran a task
>> that took a while, say a minute, Buildr would seg fault at some point,
>> not always the same point. No such problem with Ruby/1.5 or JRuby/1.6,
>> so I would recommend switching to JRuby.
>
> Switching to JRuby fixed the segfaults, and I now have a working build.
>
> Is a bit sluggish tough (10 seconds for clean). Is there a way to improve this?

1.3.4 will cut down the start time, you can give it a try:
http://incubator.apache.org/buildr/contributing.html#living_on_the_edge

Assaf

>
> I will find a way to work around the semi-circular dependency for now.
>
>
> --
>   - Henrik
>