Emake on SMP

[Nicolas Charpentier-2]

Hi,
Here is a proposal to make parallel compilation with emake.
The patch is based on OTP R12 b5 source code.

I modified the make:process/3 function in order to let it spawn a
process per file to compile. The 'main' process wait for the result of
all children before to return.
I let an 'after' clause in the receive loop in order to not get stuck if
one compilation never notify its result.


Regards,
----
Nicolas Charpentier

--- /opt/src/otp_src_R12B-5/lib/tools/src/make.erl 2006-05-03 10:26:46.000000000 +0200
+++ ../src/make.erl 2009-01-29 16:30:57.000000000 +0100
@@ -189,17 +189,34 @@
     end
     end.
 
-
-process([{[],_Opts}|Rest], NoExec, Load) ->
-    process(Rest, NoExec, Load);
-process([{[H|T],Opts}|Rest], NoExec, Load) ->
-    case recompilep(coerce_2_list(H), NoExec, Load, Opts) of
- error ->
-    error;
- _ ->
-    process([{T,Opts}|Rest], NoExec, Load)
-    end;
-process([], _NoExec, _Load) ->
+process([], _, _) ->
+    up_to_date;
+process(Lines, NoExec, Load) ->
+    Parent = self(),
+    Pids = lists:map(fun({Files,Opts}) ->
+       [spawn(compile_fun(Parent,File, Opts, NoExec, Load)) || File <- Files]
+       end, Lines),
+    collect_results(lists:flatten(Pids),[]).
+
+compile_fun(Parent, File, Opts, NoExec, Load) ->
+    fun() ->
+      Result = recompilep(coerce_2_list(File), NoExec, Load, Opts),
+      Parent ! {self(),Result}
+    end.
+
+collect_results([], Acc) ->
+    compute_results(lists:member(error, Acc));
+collect_results([Pid|Pids], Acc) ->
+    receive
+   {Pid, Result} ->
+      collect_results(Pids,[Result|Acc])
+    after 60000 ->
+      [error|Acc]
+    end.
+
+compute_results(true) ->
+    error;
+compute_results(_) ->
     up_to_date.
 
 recompilep(File, NoExec, Load, Opts) ->

_______________________________________________
erlang-patches mailing list
erlang-patches@...
http://www.erlang.org/mailman/listinfo/erlang-patches

[Björn Gustavsson]

2009/1/29 Nicolas Charpentier <nc@...>:
> Hi,
> Here is a proposal to make parallel compilation with emake.
> The patch is based on OTP R12 b5 source code.
>
> I modified the make:process/3 function in order to let it spawn a process
> per file to compile. The 'main' process wait for the result of all children
> before to return.
> I let an 'after' clause in the receive loop in order to not get stuck if one
> compilation never notify its result.

Sorry, but we cannot accept a timeout. Sooner or later someone compiles
a huge module on a slow machine and the compilation time will exceed
one minute.

The correct way to handle failing worker processes is to use either monitors
or links.

/Bjorn

--
Björn Gustavsson, Erlang/OTP, Ericsson AB
_______________________________________________
erlang-patches mailing list
erlang-patches@...
http://www.erlang.org/mailman/listinfo/erlang-patches

[Björn Gustavsson]

I found another problem...

2009/1/29 Nicolas Charpentier <nc@...>:
> I modified the make:process/3 function in order to let it spawn a process
> per file to compile. The 'main' process wait for the result of all children
> before to return.

It is not acceptable to spawn as many processes as there modules to compile.
The Erlang virtual machine could easily run out of memory. For instance, our
test suit for STDLIB contains 66 modules and some them are HUGE.

The number of compilation processes should never be more than the number
of scheduler threads, erlang:system_info(schedulers), and it should probably
be limited to a value such as 4 or 8 anyway to avoid running of out memory.
(The actual number should be an option, with a suitable default.)

/Bjorn
--
Björn Gustavsson, Erlang/OTP, Ericsson AB
_______________________________________________
erlang-patches mailing list
erlang-patches@...
http://www.erlang.org/mailman/listinfo/erlang-patches

[Nicolas Charpentier-2]

> I found another problem...
>
> 2009/1/29 Nicolas Charpentier <nc@...>:
>> I modified the make:process/3 function in order to let it spawn a
>> process
>> per file to compile. The 'main' process wait for the result of all
>> children
>> before to return.
>
> It is not acceptable to spawn as many processes as there modules to
> compile.
> The Erlang virtual machine could easily run out of memory. For instance,
> our
> test suit for STDLIB contains 66 modules and some them are HUGE.
>
> The number of compilation processes should never be more than the number
> of scheduler threads, erlang:system_info(schedulers), and it should
> probably
> be limited to a value such as 4 or 8 anyway to avoid running of out
> memory.
> (The actual number should be an option, with a suitable default.)
>

Humm, it seems that I have to work on it this week-end.
I'll be back with another proposal without the "after" and a configuration
parameter to limit the number of concurrent compilation.

Regards,
----
Nicolas Charpentier
http://charpi.net
_______________________________________________
erlang-patches mailing list
erlang-patches@...
http://www.erlang.org/mailman/listinfo/erlang-patches