Per-thread FPE pointer not being initialized

View: New views
7 Messages — Rating Filter:   Alert me  

Per-thread FPE pointer not being initialized

by David Reiss-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

The per-thread last floating point exception pointer is not initialized to
NULL when it is allocated.  As a result, spurious floating point
exceptions are reported.

This is 100% reproducible for me on CentOS 5.1 for x86-64, with Erlang/OTP
R13B built from source with gcc 2.5 and gcc 4.1.2.  It doesn't happen with
R13B01, but, based on the nature of the bug, I think that is most likely
due to chance.

Steps to reproduce:

1/ Start up a node with
  dreiss@dev016:~$ erl -noshell -name bob@`hostname`

2/ Build and compile test program
  dreiss@dev016:~$ cat test.erl
  -module(test).
  -export([start/1]).
  start([Remote]) ->
    rpc:call(Remote, erlang, is_float, [750.0]).
  dreiss@dev016:~$ erlc test.erl

3/ Run test program
  dreiss@dev016:~$ erl -noshell -name alice@`hostname` -pz . -s test
start bob@`hostname` -s init stop

The "alice" node exits with no output.  After running the "alice" command,
the "bob" node prints the following

  ERTS_FP_CHECK_INIT at 0x4aaded: detected unhandled FPE at 0x2

Patch:

diff otp_src_R13B/erts/emulator/sys/unix/sys_float.c
otp_src_R13B/sys/unix/sys_float.c/sys/unix/sys_float.c
--- otp_src_R13B/erts/emulator/sys/unix/sys_float.c 2009-06-30
17:53:31.000000000 -0700
+++ otp_src_R13B/erts/emulator/sys/unix/sys_float.c 2009-06-30
17:53:31.000000000 -0700
@@ -57,7 +57,8 @@
 void erts_thread_init_fp_exception(void)
 {
     unsigned long *fpe = erts_alloc(ERTS_ALC_T_FP_EXCEPTION, sizeof(*fpe));
+    *fpe = 0;
     erts_tsd_set(fpe_key, fpe);
 }



--David

________________________________________________________________
erlang-patches mailing list. See http://www.erlang.org/faq.html
erlang-patches (at) erlang.org


Re: Per-thread FPE pointer not being initialized

by Raimo Niskanen-5 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

I can not reproduce it - that would need exact match
of floating point exception vs signals i guess.
I see no harm ininitializing an allocated buffer
(except for performance) but suspect the bug might be elswhere.
The patch is scheduled for R13B02 and I will check
with the guys that know better if the fix should
be somewhere else...

On Tue, Jun 30, 2009 at 07:34:39PM -0700, David Reiss wrote:

> The per-thread last floating point exception pointer is not initialized to
> NULL when it is allocated.  As a result, spurious floating point
> exceptions are reported.
>
> This is 100% reproducible for me on CentOS 5.1 for x86-64, with Erlang/OTP
> R13B built from source with gcc 2.5 and gcc 4.1.2.  It doesn't happen with
> R13B01, but, based on the nature of the bug, I think that is most likely
> due to chance.
>
> Steps to reproduce:
>
> 1/ Start up a node with
>   dreiss@dev016:~$ erl -noshell -name bob@`hostname`
>
> 2/ Build and compile test program
>   dreiss@dev016:~$ cat test.erl
>   -module(test).
>   -export([start/1]).
>   start([Remote]) ->
>     rpc:call(Remote, erlang, is_float, [750.0]).
>   dreiss@dev016:~$ erlc test.erl
>
> 3/ Run test program
>   dreiss@dev016:~$ erl -noshell -name alice@`hostname` -pz . -s test
> start bob@`hostname` -s init stop
>
> The "alice" node exits with no output.  After running the "alice" command,
> the "bob" node prints the following
>
>   ERTS_FP_CHECK_INIT at 0x4aaded: detected unhandled FPE at 0x2
>
> Patch:
>
> diff otp_src_R13B/erts/emulator/sys/unix/sys_float.c
> otp_src_R13B/sys/unix/sys_float.c/sys/unix/sys_float.c
> --- otp_src_R13B/erts/emulator/sys/unix/sys_float.c 2009-06-30
> 17:53:31.000000000 -0700
> +++ otp_src_R13B/erts/emulator/sys/unix/sys_float.c 2009-06-30
> 17:53:31.000000000 -0700
> @@ -57,7 +57,8 @@
>  void erts_thread_init_fp_exception(void)
>  {
>      unsigned long *fpe = erts_alloc(ERTS_ALC_T_FP_EXCEPTION, sizeof(*fpe));
> +    *fpe = 0;
>      erts_tsd_set(fpe_key, fpe);
>  }
>
>
>
> --David
>
> ________________________________________________________________
> erlang-patches mailing list. See http://www.erlang.org/faq.html
> erlang-patches (at) erlang.org

--

/ Raimo Niskanen, Erlang/OTP, Ericsson AB

________________________________________________________________
erlang-patches mailing list. See http://www.erlang.org/faq.html
erlang-patches (at) erlang.org


Re: Per-thread FPE pointer not being initialized

by Mikael Pettersson-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

David Reiss writes:
 > The per-thread last floating point exception pointer is not initialized to
 > NULL when it is allocated.  As a result, spurious floating point
 > exceptions are reported.
 >
 > This is 100% reproducible for me on CentOS 5.1 for x86-64, with Erlang/OTP
 > R13B built from source with gcc 2.5 and gcc 4.1.2.  It doesn't happen with
 > R13B01, but, based on the nature of the bug, I think that is most likely
 > due to chance.
 >
 > Steps to reproduce:
 >
 > 1/ Start up a node with
 >   dreiss@dev016:~$ erl -noshell -name bob@`hostname`
 >
 > 2/ Build and compile test program
 >   dreiss@dev016:~$ cat test.erl
 >   -module(test).
 >   -export([start/1]).
 >   start([Remote]) ->
 >     rpc:call(Remote, erlang, is_float, [750.0]).
 >   dreiss@dev016:~$ erlc test.erl
 >
 > 3/ Run test program
 >   dreiss@dev016:~$ erl -noshell -name alice@`hostname` -pz . -s test
 > start bob@`hostname` -s init stop
 >
 > The "alice" node exits with no output.  After running the "alice" command,
 > the "bob" node prints the following
 >
 >   ERTS_FP_CHECK_INIT at 0x4aaded: detected unhandled FPE at 0x2
 >
 > Patch:
 >
 > diff otp_src_R13B/erts/emulator/sys/unix/sys_float.c
 > otp_src_R13B/sys/unix/sys_float.c/sys/unix/sys_float.c
 > --- otp_src_R13B/erts/emulator/sys/unix/sys_float.c 2009-06-30
 > 17:53:31.000000000 -0700
 > +++ otp_src_R13B/erts/emulator/sys/unix/sys_float.c 2009-06-30
 > 17:53:31.000000000 -0700
 > @@ -57,7 +57,8 @@
 >  void erts_thread_init_fp_exception(void)
 >  {
 >      unsigned long *fpe = erts_alloc(ERTS_ALC_T_FP_EXCEPTION, sizeof(*fpe));
 > +    *fpe = 0;
 >      erts_tsd_set(fpe_key, fpe);
 >  }

Thanks, that bug was mine :-( Your fix is correct.

I believe the initialisation here wasn't needed originally,
but became necessary as a side effect of some FP exn machinery
redesign I did late last year.

Where did the spurious FP exception get detected, i.e.,
what runtime system procedure does 0x4aaded correspond to?

/Mikael

________________________________________________________________
erlang-patches mailing list. See http://www.erlang.org/faq.html
erlang-patches (at) erlang.org


Re: Per-thread FPE pointer not being initialized

by Raimo Niskanen-5 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Wed, Jul 01, 2009 at 12:03:04PM +0200, Mikael Pettersson wrote:

> David Reiss writes:
>  > The per-thread last floating point exception pointer is not initialized to
>  > NULL when it is allocated.  As a result, spurious floating point
>  > exceptions are reported.
>  >
>  > This is 100% reproducible for me on CentOS 5.1 for x86-64, with Erlang/OTP
>  > R13B built from source with gcc 2.5 and gcc 4.1.2.  It doesn't happen with
>  > R13B01, but, based on the nature of the bug, I think that is most likely
>  > due to chance.
>  >
>  > Steps to reproduce:
>  >
>  > 1/ Start up a node with
>  >   dreiss@dev016:~$ erl -noshell -name bob@`hostname`
>  >
>  > 2/ Build and compile test program
>  >   dreiss@dev016:~$ cat test.erl
>  >   -module(test).
>  >   -export([start/1]).
>  >   start([Remote]) ->
>  >     rpc:call(Remote, erlang, is_float, [750.0]).
>  >   dreiss@dev016:~$ erlc test.erl
>  >
>  > 3/ Run test program
>  >   dreiss@dev016:~$ erl -noshell -name alice@`hostname` -pz . -s test
>  > start bob@`hostname` -s init stop
>  >
>  > The "alice" node exits with no output.  After running the "alice" command,
>  > the "bob" node prints the following
>  >
>  >   ERTS_FP_CHECK_INIT at 0x4aaded: detected unhandled FPE at 0x2
>  >
>  > Patch:
>  >
>  > diff otp_src_R13B/erts/emulator/sys/unix/sys_float.c
>  > otp_src_R13B/sys/unix/sys_float.c/sys/unix/sys_float.c
>  > --- otp_src_R13B/erts/emulator/sys/unix/sys_float.c 2009-06-30
>  > 17:53:31.000000000 -0700
>  > +++ otp_src_R13B/erts/emulator/sys/unix/sys_float.c 2009-06-30
>  > 17:53:31.000000000 -0700
>  > @@ -57,7 +57,8 @@
>  >  void erts_thread_init_fp_exception(void)
>  >  {
>  >      unsigned long *fpe = erts_alloc(ERTS_ALC_T_FP_EXCEPTION, sizeof(*fpe));
>  > +    *fpe = 0;
>  >      erts_tsd_set(fpe_key, fpe);
>  >  }
>
> Thanks, that bug was mine :-( Your fix is correct.

Would your patch in http://www.erlang.org/cgi-bin/ezmlm-cgi/2/1380
also solve this problem? Or are both needed?

--- otp-0622/erts/emulator/sys/unix/sys_float.c.~1~ 2009-02-26 11:45:48.000000000 +0100
+++ otp-0622/erts/emulator/sys/unix/sys_float.c 2009-06-26 11:14:15.000000000 +0200
@@ -792,6 +792,8 @@ sys_chars_to_double(char* buf, double* f
 int
 matherr(struct exception *exc)
 {
+#if !defined(NO_FPE_SIGNALS)
     set_current_fp_exception((unsigned long)__builtin_return_address(0));
+#endif
     return 1;
 }



>
> I believe the initialisation here wasn't needed originally,
> but became necessary as a side effect of some FP exn machinery
> redesign I did late last year.
>
> Where did the spurious FP exception get detected, i.e.,
> what runtime system procedure does 0x4aaded correspond to?
>
> /Mikael
>
> ________________________________________________________________
> erlang-patches mailing list. See http://www.erlang.org/faq.html
> erlang-patches (at) erlang.org

--

/ Raimo Niskanen, Erlang/OTP, Ericsson AB

________________________________________________________________
erlang-patches mailing list. See http://www.erlang.org/faq.html
erlang-patches (at) erlang.org


Re: Per-thread FPE pointer not being initialized

by Mikael Pettersson-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Raimo Niskanen writes:
 > On Wed, Jul 01, 2009 at 12:03:04PM +0200, Mikael Pettersson wrote:
 > > David Reiss writes:
 > >  > The per-thread last floating point exception pointer is not initialized to
 > >  > NULL when it is allocated.  As a result, spurious floating point
 > >  > exceptions are reported.
 > >  >
 > >  > This is 100% reproducible for me on CentOS 5.1 for x86-64, with Erlang/OTP
 > >  > R13B built from source with gcc 2.5 and gcc 4.1.2.  It doesn't happen with
 > >  > R13B01, but, based on the nature of the bug, I think that is most likely
 > >  > due to chance.
 > >  >
 > >  > Steps to reproduce:
 > >  >
 > >  > 1/ Start up a node with
 > >  >   dreiss@dev016:~$ erl -noshell -name bob@`hostname`
 > >  >
 > >  > 2/ Build and compile test program
 > >  >   dreiss@dev016:~$ cat test.erl
 > >  >   -module(test).
 > >  >   -export([start/1]).
 > >  >   start([Remote]) ->
 > >  >     rpc:call(Remote, erlang, is_float, [750.0]).
 > >  >   dreiss@dev016:~$ erlc test.erl
 > >  >
 > >  > 3/ Run test program
 > >  >   dreiss@dev016:~$ erl -noshell -name alice@`hostname` -pz . -s test
 > >  > start bob@`hostname` -s init stop
 > >  >
 > >  > The "alice" node exits with no output.  After running the "alice" command,
 > >  > the "bob" node prints the following
 > >  >
 > >  >   ERTS_FP_CHECK_INIT at 0x4aaded: detected unhandled FPE at 0x2
 > >  >
 > >  > Patch:
 > >  >
 > >  > diff otp_src_R13B/erts/emulator/sys/unix/sys_float.c
 > >  > otp_src_R13B/sys/unix/sys_float.c/sys/unix/sys_float.c
 > >  > --- otp_src_R13B/erts/emulator/sys/unix/sys_float.c 2009-06-30
 > >  > 17:53:31.000000000 -0700
 > >  > +++ otp_src_R13B/erts/emulator/sys/unix/sys_float.c 2009-06-30
 > >  > 17:53:31.000000000 -0700
 > >  > @@ -57,7 +57,8 @@
 > >  >  void erts_thread_init_fp_exception(void)
 > >  >  {
 > >  >      unsigned long *fpe = erts_alloc(ERTS_ALC_T_FP_EXCEPTION, sizeof(*fpe));
 > >  > +    *fpe = 0;
 > >  >      erts_tsd_set(fpe_key, fpe);
 > >  >  }
 > >
 > > Thanks, that bug was mine :-( Your fix is correct.
 >
 > Would your patch in http://www.erlang.org/cgi-bin/ezmlm-cgi/2/1380
 > also solve this problem? Or are both needed?
 >
 > --- otp-0622/erts/emulator/sys/unix/sys_float.c.~1~ 2009-02-26 11:45:48.000000000 +0100
 > +++ otp-0622/erts/emulator/sys/unix/sys_float.c 2009-06-26 11:14:15.000000000 +0200
 > @@ -792,6 +792,8 @@ sys_chars_to_double(char* buf, double* f
 >  int
 >  matherr(struct exception *exc)
 >  {
 > +#if !defined(NO_FPE_SIGNALS)
 >      set_current_fp_exception((unsigned long)__builtin_return_address(0));
 > +#endif
 >      return 1;
 >  }

The two patches fix independent issues. The #if !defined change in
sys_chars_to_double() fixes a compile/link error with !gcc.

/Mikael

________________________________________________________________
erlang-patches mailing list. See http://www.erlang.org/faq.html
erlang-patches (at) erlang.org


Re: Per-thread FPE pointer not being initialized

by Raimo Niskanen-5 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

On Wed, Jul 01, 2009 at 01:55:39PM +0200, Mikael Pettersson wrote:

> Raimo Niskanen writes:
>  > On Wed, Jul 01, 2009 at 12:03:04PM +0200, Mikael Pettersson wrote:
>  > > David Reiss writes:
>  > >  > The per-thread last floating point exception pointer is not initialized to
>  > >  > NULL when it is allocated.  As a result, spurious floating point
>  > >  > exceptions are reported.
>  > >  >
>  > >  > This is 100% reproducible for me on CentOS 5.1 for x86-64, with Erlang/OTP
>  > >  > R13B built from source with gcc 2.5 and gcc 4.1.2.  It doesn't happen with
>  > >  > R13B01, but, based on the nature of the bug, I think that is most likely
>  > >  > due to chance.
>  > >  >
>  > >  > Steps to reproduce:
>  > >  >
>  > >  > 1/ Start up a node with
>  > >  >   dreiss@dev016:~$ erl -noshell -name bob@`hostname`
>  > >  >
>  > >  > 2/ Build and compile test program
>  > >  >   dreiss@dev016:~$ cat test.erl
>  > >  >   -module(test).
>  > >  >   -export([start/1]).
>  > >  >   start([Remote]) ->
>  > >  >     rpc:call(Remote, erlang, is_float, [750.0]).
>  > >  >   dreiss@dev016:~$ erlc test.erl
>  > >  >
>  > >  > 3/ Run test program
>  > >  >   dreiss@dev016:~$ erl -noshell -name alice@`hostname` -pz . -s test
>  > >  > start bob@`hostname` -s init stop
>  > >  >
>  > >  > The "alice" node exits with no output.  After running the "alice" command,
>  > >  > the "bob" node prints the following
>  > >  >
>  > >  >   ERTS_FP_CHECK_INIT at 0x4aaded: detected unhandled FPE at 0x2
>  > >  >
>  > >  > Patch:
>  > >  >
>  > >  > diff otp_src_R13B/erts/emulator/sys/unix/sys_float.c
>  > >  > otp_src_R13B/sys/unix/sys_float.c/sys/unix/sys_float.c
>  > >  > --- otp_src_R13B/erts/emulator/sys/unix/sys_float.c 2009-06-30
>  > >  > 17:53:31.000000000 -0700
>  > >  > +++ otp_src_R13B/erts/emulator/sys/unix/sys_float.c 2009-06-30
>  > >  > 17:53:31.000000000 -0700
>  > >  > @@ -57,7 +57,8 @@
>  > >  >  void erts_thread_init_fp_exception(void)
>  > >  >  {
>  > >  >      unsigned long *fpe = erts_alloc(ERTS_ALC_T_FP_EXCEPTION, sizeof(*fpe));
>  > >  > +    *fpe = 0;
>  > >  >      erts_tsd_set(fpe_key, fpe);
>  > >  >  }
>  > >
>  > > Thanks, that bug was mine :-( Your fix is correct.
>  >
>  > Would your patch in http://www.erlang.org/cgi-bin/ezmlm-cgi/2/1380
>  > also solve this problem? Or are both needed?
>  >
>  > --- otp-0622/erts/emulator/sys/unix/sys_float.c.~1~ 2009-02-26 11:45:48.000000000 +0100
>  > +++ otp-0622/erts/emulator/sys/unix/sys_float.c 2009-06-26 11:14:15.000000000 +0200
>  > @@ -792,6 +792,8 @@ sys_chars_to_double(char* buf, double* f
>  >  int
>  >  matherr(struct exception *exc)
>  >  {
>  > +#if !defined(NO_FPE_SIGNALS)
>  >      set_current_fp_exception((unsigned long)__builtin_return_address(0));
>  > +#endif
>  >      return 1;
>  >  }
>
> The two patches fix independent issues. The #if !defined change in
> sys_chars_to_double() fixes a compile/link error with !gcc.

Sorry I misread the patch. I accidentally saw it add
set_current_fp_exception(0) to matherr(exc) but it
only added preprocessor directives...

>
> /Mikael
>
> ________________________________________________________________
> erlang-patches mailing list. See http://www.erlang.org/faq.html
> erlang-patches (at) erlang.org

--

/ Raimo Niskanen, Erlang/OTP, Ericsson AB

________________________________________________________________
erlang-patches mailing list. See http://www.erlang.org/faq.html
erlang-patches (at) erlang.org


Re: Per-thread FPE pointer not being initialized

by David Reiss-3 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

> Where did the spurious FP exception get detected, i.e.,
> what runtime system procedure does 0x4aaded correspond to?
It looks like it is dec_term (and gdb reports it as line 1206 of
beam/external.c, which is a call to sys_chars_to_double).  When I
first detected the problem, it was in the context of a larger
application.  The reported address was 0x51d507, which is in
sys_chars_to_double.  The boundaries get a bit blurred in optimized
builds. :)

--David



On Wed, Jul 1, 2009 at 8:16 AM, Raimo
Niskanen<raimo+erlang-patches@...> wrote:

> On Wed, Jul 01, 2009 at 01:55:39PM +0200, Mikael Pettersson wrote:
>> Raimo Niskanen writes:
>>  > On Wed, Jul 01, 2009 at 12:03:04PM +0200, Mikael Pettersson wrote:
>>  > > David Reiss writes:
>>  > >  > The per-thread last floating point exception pointer is not initialized to
>>  > >  > NULL when it is allocated.  As a result, spurious floating point
>>  > >  > exceptions are reported.
>>  > >  >
>>  > >  > This is 100% reproducible for me on CentOS 5.1 for x86-64, with Erlang/OTP
>>  > >  > R13B built from source with gcc 2.5 and gcc 4.1.2.  It doesn't happen with
>>  > >  > R13B01, but, based on the nature of the bug, I think that is most likely
>>  > >  > due to chance.
>>  > >  >
>>  > >  > Steps to reproduce:
>>  > >  >
>>  > >  > 1/ Start up a node with
>>  > >  >   dreiss@dev016:~$ erl -noshell -name bob@`hostname`
>>  > >  >
>>  > >  > 2/ Build and compile test program
>>  > >  >   dreiss@dev016:~$ cat test.erl
>>  > >  >   -module(test).
>>  > >  >   -export([start/1]).
>>  > >  >   start([Remote]) ->
>>  > >  >     rpc:call(Remote, erlang, is_float, [750.0]).
>>  > >  >   dreiss@dev016:~$ erlc test.erl
>>  > >  >
>>  > >  > 3/ Run test program
>>  > >  >   dreiss@dev016:~$ erl -noshell -name alice@`hostname` -pz . -s test
>>  > >  > start bob@`hostname` -s init stop
>>  > >  >
>>  > >  > The "alice" node exits with no output.  After running the "alice" command,
>>  > >  > the "bob" node prints the following
>>  > >  >
>>  > >  >   ERTS_FP_CHECK_INIT at 0x4aaded: detected unhandled FPE at 0x2
>>  > >  >
>>  > >  > Patch:
>>  > >  >
>>  > >  > diff otp_src_R13B/erts/emulator/sys/unix/sys_float.c
>>  > >  > otp_src_R13B/sys/unix/sys_float.c/sys/unix/sys_float.c
>>  > >  > --- otp_src_R13B/erts/emulator/sys/unix/sys_float.c   2009-06-30
>>  > >  > 17:53:31.000000000 -0700
>>  > >  > +++ otp_src_R13B/erts/emulator/sys/unix/sys_float.c   2009-06-30
>>  > >  > 17:53:31.000000000 -0700
>>  > >  > @@ -57,7 +57,8 @@
>>  > >  >  void erts_thread_init_fp_exception(void)
>>  > >  >  {
>>  > >  >      unsigned long *fpe = erts_alloc(ERTS_ALC_T_FP_EXCEPTION, sizeof(*fpe));
>>  > >  > +    *fpe = 0;
>>  > >  >      erts_tsd_set(fpe_key, fpe);
>>  > >  >  }
>>  > >
>>  > > Thanks, that bug was mine :-( Your fix is correct.
>>  >
>>  > Would your patch in http://www.erlang.org/cgi-bin/ezmlm-cgi/2/1380
>>  > also solve this problem? Or are both needed?
>>  >
>>  > --- otp-0622/erts/emulator/sys/unix/sys_float.c.~1~        2009-02-26 11:45:48.000000000 +0100
>>  > +++ otp-0622/erts/emulator/sys/unix/sys_float.c    2009-06-26 11:14:15.000000000 +0200
>>  > @@ -792,6 +792,8 @@ sys_chars_to_double(char* buf, double* f
>>  >  int
>>  >  matherr(struct exception *exc)
>>  >  {
>>  > +#if !defined(NO_FPE_SIGNALS)
>>  >      set_current_fp_exception((unsigned long)__builtin_return_address(0));
>>  > +#endif
>>  >      return 1;
>>  >  }
>>
>> The two patches fix independent issues. The #if !defined change in
>> sys_chars_to_double() fixes a compile/link error with !gcc.
>
> Sorry I misread the patch. I accidentally saw it add
> set_current_fp_exception(0) to matherr(exc) but it
> only added preprocessor directives...
>
>>
>> /Mikael
>>
>> ________________________________________________________________
>> erlang-patches mailing list. See http://www.erlang.org/faq.html
>> erlang-patches (at) erlang.org
>
> --
>
> / Raimo Niskanen, Erlang/OTP, Ericsson AB
>
> ________________________________________________________________
> erlang-patches mailing list. See http://www.erlang.org/faq.html
> erlang-patches (at) erlang.org
>
>

________________________________________________________________
erlang-patches mailing list. See http://www.erlang.org/faq.html
erlang-patches (at) erlang.org