Why doesn't erl_eval compile code?

[Tony Arcieri]

In its present implementation erl_eval accepts an Erlang parse tree and it
walks the nodes of the parse tree, evaluating them node by node.  This is
somewhat inelegant in many cases, such as when you run into constructs like
funs:

http://gist.github.com/79022

Why not transform the parse tree into something that provides erl_eval's
return value (i.e. real return value + final bindings), compile that to a
temporary module, load that into the code server, and execute it, removing
the module when it's done.  Wouldn't that solution provide more consistent
semantics while also being faster at things like tail calls and other
computationally complex problems?

I'll go ahead and concede there's a great deal of complexity to both
solutions, and the compiled solution presents some concurrency problems, as
I understand.  Please expound on that issue either way.

As a bit of background, I'm working on moving the eval strategy of my
language Reia (which runs on the Erlang VM) off of erl_eval completely,
having it always compile to BEAM/HiPE bytecode, to make the semantics more
consistent and also improve performance.  I'm curious why Erlang doesn't
take this approach.

--
Tony Arcieri

[Robert Virding]

Seeing no one has replied I will give some comments:

- What problem are you trying to solve? I admit that it would be cool to do
it, but why?
- While the fun code is not very elegant :-), it does work as it should.
What does not work as it should is interpreting receive.
- If you remove the module when you are done, which you must explicitly do,
then there could be some problem with funs if you have passed them on and
they still are in use as they will not work after their module has been
removed.
- The interpreter was never intended to run larger programs so in one
respect it wold be a waste of time.
- If you implemented this then PDQ you would get requests to be able to
define functions and modules in the shell. :-)

But it would be cool code,

Robert

2009/6/24 Tony Arcieri <tony@...>

> In its present implementation erl_eval accepts an Erlang parse tree and it
> walks the nodes of the parse tree, evaluating them node by node.  This is
> somewhat inelegant in many cases, such as when you run into constructs like
> funs:
>
> http://gist.github.com/79022
>
> Why not transform the parse tree into something that provides erl_eval's
> return value (i.e. real return value + final bindings), compile that to a
> temporary module, load that into the code server, and execute it, removing
> the module when it's done.  Wouldn't that solution provide more consistent
> semantics while also being faster at things like tail calls and other
> computationally complex problems?
>
> I'll go ahead and concede there's a great deal of complexity to both
> solutions, and the compiled solution presents some concurrency problems, as
> I understand.  Please expound on that issue either way.
>
> As a bit of background, I'm working on moving the eval strategy of my
> language Reia (which runs on the Erlang VM) off of erl_eval completely,
> having it always compile to BEAM/HiPE bytecode, to make the semantics more
> consistent and also improve performance.  I'm curious why Erlang doesn't
> take this approach.
>
> --
> Tony Arcieri
>

[Tony Arcieri]

On Sun, Jul 5, 2009 at 5:11 PM, Robert Virding <rvirding@...> wrote:

> Seeing no one has replied I will give some comments:
>
> - What problem are you trying to solve? I admit that it would be cool to do
> it, but why?


Well, specifically I'm curious about this approach for Reia.  Right now I've
had difficulty keeping the semantics consistent between the compiled and
evaluated versions (and this is using erl_eval to handle the evaluation).

I figure by using a compiled solution for eval there's a single path all
code goes through to get executed.


> - While the fun code is not very elegant :-), it does work as it should.


Unless you're crazy enough to pass more than 20 arguments :)


> - If you remove the module when you are done, which you must explicitly do,
> then there could be some problem with funs if you have passed them on and
> they still are in use as they will not work after their module has been
> removed.


Oh right, crap... that's a tricky problem.  I can't think of a good solution
offhand either.

--
Tony Arcieri
medioh.com

[Tony Arcieri]

On Sun, Jul 5, 2009 at 5:11 PM, Robert Virding <rvirding@...> wrote:

> - If you remove the module when you are done, which you must explicitly do,
> then there could be some problem with funs if you have passed them on and
> they still are in use as they will not work after their module has been
> removed.


So here's a question... if I code:soft_purge a module which was used to
create funs which are still in scope in some process somewhere, will that
call to soft_purge fail with false?

If so, as a hack I could have a "garbage collector" process which tries to
soft purge the temporary modules periodically until it succeeds.

This is a far uglier solution that I thought it would be when I started out,
though :(

Would it work though?

--
Tony Arcieri
medioh.com

[Robert Virding]

Sorry, don't know about soft_purge. In my time we were hard. :-)

The basic problem is that you are trying to use the code system in a way we
never intended. To do this properly as you intend would mean a rewrite of
the code system, which would probably result in changes all over the place.
But it would be more beautiful. :-(

Robert

2009/7/7 Tony Arcieri <tony@...>

> On Sun, Jul 5, 2009 at 5:11 PM, Robert Virding <rvirding@...> wrote:
>
>> - If you remove the module when you are done, which you must explicitly
>> do, then there could be some problem with funs if you have passed them on
>> and they still are in use as they will not work after their module has been
>> removed.
>
>
> So here's a question... if I code:soft_purge a module which was used to
> create funs which are still in scope in some process somewhere, will that
> call to soft_purge fail with false?
>
> If so, as a hack I could have a "garbage collector" process which tries to
> soft purge the temporary modules periodically until it succeeds.
>
> This is a far uglier solution that I thought it would be when I started
> out, though :(
>
> Would it work though?
>
> --
> Tony Arcieri
> medioh.com
>

[Tony Arcieri]

On Wed, Jul 8, 2009 at 4:46 PM, Robert Virding <rvirding@...> wrote:

> The basic problem is that you are trying to use the code system in a way we
> never intended.


Yes, at times I feel like I'm in square peg round hole territory.  Usually
when I reach this point I try to take a step back and do things in an
Erlangier way.  I now see the reasoning behind why Erlang implements eval
metacircularly.  However, if there's one thing that sets scripting languages
apart from other languages I believe it's extensive use of eval and runtime
code generation/execution.  In that regard you might call Lisp the world's
first scripting language (although I believe that honor is typically
attributed to awk)

However, it's a lot easier to write a metacircular evaluator for a Lisp than
for something like Reia...


> To do this properly as you intend would mean a rewrite of the code system,
> which would probably result in changes all over the place. But it would be
> more beautiful. :-(
>

I'm okay with hacks as long as they work :)

--
Tony Arcieri
medioh.com

[Robert Virding]

2009/7/9 Tony Arcieri <tony@...>

> On Wed, Jul 8, 2009 at 4:46 PM, Robert Virding <rvirding@...> wrote:
>
>> The basic problem is that you are trying to use the code system in a way
>> we never intended.
>
>
> Yes, at times I feel like I'm in square peg round hole territory.  Usually
> when I reach this point I try to take a step back and do things in an
> Erlangier way.  I now see the reasoning behind why Erlang implements eval
> metacircularly.  However, if there's one thing that sets scripting languages
> apart from other languages I believe it's extensive use of eval and runtime
> code generation/execution.  In that regard you might call Lisp the world's
> first scripting language (although I believe that honor is typically
> attributed to awk)
>

I think that many scripting languages probably do interpret the code
internally and don't compile it, though I haven't studied them so I am not
certain. What makes lisp (and prolog for that matter) so useful as a
scripting language is that most implementations allow you to freely intermix
compiled and interpreted functions. Which Erlang does not, and that when it
manages code it manages whole modules not functions, and that modules are
compiled.

I ran into exactly the same problem with LFE, plus that lisp also has macros
as separate entities.

However, it's a lot easier to write a metacircular evaluator for a Lisp than
> for something like Reia...
>
>
>> To do this properly as you intend would mean a rewrite of the code system,
>> which would probably result in changes all over the place. But it would be
>> more beautiful. :-(
>>
>
> I'm okay with hacks as long as they work :)


In this case using soft_purge and having a process which occasionally tries
to remove your temporary modules would be the hack. Looking at the
documentation for soft_purge seems to imply that it is fun safe, but try it.

Robert