How would shallow generators compose with lambda?

[Mark S. Miller-2]

Two features that have been proposed for Harmony are shallow
generators and lambda.

In the absence of lambda, I think I understand shallow generators.
Because of the way my brain is wired, for whatever reason, my path to
understanding them is as a source-to-source cps-style transform of the
generator function -- the function immediately containing the 'yield'.
Because generators are shallow, only this function need be translated.
I do understand that other people find this route to understanding
shallow generators unnatural.

In the absence of generators, I am confident I understand lambda. A
return in a lambda returns from the immediately containing function
activation, so long as that function activation is still active. If
the containing function has already returned, then a postponed attempt
to return from it again will fail[1], presumably with a thrown error.
This applies as well to break, continue, labeled break, labeled
continue, and labeled return if we introduce such a thing. What
happens to a postponed 'yield'? Also, does it make as much sense to
introduce a labeled yield as it does to introduce a labeled return?

Given both shallow generators and lambda, I don't understand how they
would interact.

[1] Schemers would say "dynamic extent continuations" vs Scheme's
"indefinite extent continuations". Indefinite extent continuations --
which can be returned from multiple times -- are IMO a horror.
Indefinite extent continuations would enable deep generators, which
only increases my sense of horror. Perhaps combining lambda with
shallow generators produces something like delimited continuations?
The literature on these makes my brain hurt.

--
    Cheers,
    --MarkM
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Jason Orendorff]

On Thu, May 14, 2009 at 12:25 PM, Mark S. Miller <erights@...> wrote:
> Given both shallow generators and lambda, I don't understand how they
> would interact.

This is a good question. I don't have the answer, but thinking aloud:

You can implement shallow generators either by doing the CPS
transformation you described or by storing (a reference to) the top
stack frame in the generator-iterator object. (The latter is how
SpiderMonkey and CPython currently do it.)

Now, suppose 'yield' and 'return' are extended in analogous ways in lambdas.

1. A lambda can escape and be called from other code.  So both
implementation approaches must be modified:

* The CPS transformation would have to be applied to almost all
functions, even functions not lexically inside the generator-function.
This is a major change.

* The stack-snapshot approach would have to store (a reference to) a
range of stack frames, which may include frames for functions not
lexically inside the generator-function. This might be a minor change.

So much for duality. :-|

2. With shallow generators, there is a check so that you can't call
.next() on a generator if it's already executing.  It just throws a
TypeError.  This enforces an invariant:

* In the continuations view, each captured continuation will be called
(resumed) at most once.

* In the stack-snapshot view, no stack frame may ever appear twice in the stack.

The implementation of the check is unaffected by lambdas--.next() can
trivially check a bit in the generator-iterator, regardless of
implementation.  But with lambdas, the invariants take on a new
meaning.  In particular, note that generators+lambdas are not fully as
powerful as delimited continuations. They are more like coroutines.
...In fact, you might be able to use them to implement coroutines.

3. When a lambda yields, the lines of code corresponding to the
topmost captured frame (or, the beginning of the delimited
continuation) and the bottommost (the tail end of the delimited
continuation) are both lexically inside the generator-function. But
there may be other functions on the stack, in between. You can't
always statically tell which ones.  This means that generator
semantics affect the integrity of code that isn't in a generator.

In ES5, when you call a function, you can expect it to return or throw
eventually.  (Unless you run out of memory, or time, and the whole
script gets terminated.)  With shallow generators, this is still true.
 A 'yield' might never return control, but function calls are ok.  But
with generators+lambdas, almost any function call *anywhere* in the
program might never return or throw.  This weakens 'finally', at
least.

-j
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

We still have open controversy over lambdas, if I recall correctly.  
Separate thread, but I wanted to confirm that there is disagreement on  
completion value being the implicit return value, at least. Some  
strawman syntax for explicit result value was mooted.

I also seem to recall some folks (Waldemar in particular) thinking  
that lambdas are redundant, and also potentially confusing to average  
users precisely *because* of Tennant Correspondence being two-edged in  
languages that have statements as well as expressions. The question  
for naive readers is: "return" from what? The answer proposed is  
always from the nearest enclosing function (ignoring return to label),  
but some may stop at the nearest enclosing lambda, and resulting bugs  
might hide in plain sight.

On May 14, 2009, at 10:25 AM, Mark S. Miller wrote:

> A return in a lambda returns from the immediately containing function
> activation, so long as that function activation is still active. If
> the containing function has already returned, then a postponed attempt
> to return from it again will fail[1], presumably with a thrown error.
> This applies as well to break, continue, labeled break, labeled
> continue, and labeled return if we introduce such a thing. What
> happens to a postponed 'yield'?

Same thing: yield is very much like return, except that the activation  
is not deallocated because the generator-iterator g, created by  
calling a generator function f that contains yield expresssions, which  
was invoked via g.next(), g.send(v), or g.throw(e), holds a strong  
reference to that activation for generator function f.

So a lambda that tries to yield from a deactivated function that  
encloses it should fail with a big fat exception, same as for return.


> Also, does it make as much sense to
> introduce a labeled yield as it does to introduce a labeled return?

Return to label, aka "escape continuations", are motivated largely by  
the expression-oriented lambda proposal with its final expression  
evaluating to the return value, for working around the consequent lack  
of "early returns".

Note that return to label as proposed works in any labeled statement,  
and does not require a lambda to surround the return. It works via TC  
in lambdas just as return does. The motivation via lambda having  
implicit return of completion value is what connects the return to  
label and lambda proposals.

In this light, yield to label is plausible, although the "early/
exceptional return from lambda" motivation is missing in the sense  
that you can write yield expressions in larger expressions -- you  
can't use return in the middle of an expression. But yield to label  
could be just as useful as return to label.

The syntax works when the yield expression forms a complete expression  
statement:

     yield : expr;

but it requires parenthesization in the middle operand of the ?:  
ternary operator.

Indeed the low-precedence (same as assignment) nature of the yield  
unary caused JS1.7 to follow Python in requiring parentheses around  
yield expressions in lists: in comma expressions and actual parameter  
lists (although we diverged from Python by allowing an unparenthesized  
yield expression as the last actual argument in a call or new  
expression's parameter list -- Python requires foo((yield bar))  
instead of foo(yield bar)).

/be
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Neil Mix]

On May 14, 2009, at 2:38 PM, Brendan Eich wrote:

> Indeed the low-precedence (same as assignment) nature of the yield  
> unary caused JS1.7 to follow Python in requiring parentheses around  
> yield expressions in lists: in comma expressions and actual  
> parameter lists (although we diverged from Python by allowing an  
> unparenthesized yield expression as the last actual argument in a  
> call or new expression's parameter list -- Python requires  
> foo((yield bar)) instead of foo(yield bar)).

I have this idea that it would be better for yield expressions to look  
like function calls: 'yield' '(' expression? ')'.  (Note that this is  
only a syntactical suggestion; clearly an implementation wouldn't  
actually treat it like a function.)  That would eliminate the  
precedence issues Brendan cites while also making the syntax backward  
compatible with earlier ES parsers.  Is there any technical reason why  
that wouldn't be possible?

Perhaps the "looks-like-a-duck, quacks-like-a-cow" objection would  
apply here -- a corner-case thingy that looks like a function but  
isn't should be avoided.  But I have to say, in practice the required  
parenthesizing around yield expressions in JS 1.7 is an eyesore I'd  
like to see go away.
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

On May 14, 2009, at 12:24 PM, Jason Orendorff wrote:

> 1. A lambda can escape and be called from other code.  So both
> implementation approaches must be modified:
>
> * The CPS transformation would have to be applied to almost all
> functions, even functions not lexically inside the generator-function.
> This is a major change.
>
> * The stack-snapshot approach would have to store (a reference to) a
> range of stack frames, which may include frames for functions not
> lexically inside the generator-function. This might be a minor change.
>
> So much for duality. :-|

It's overrated :-P.

But this may be a problem with lambdas as proposed, not generators per  
se.


> 3. When a lambda yields, the lines of code corresponding to the
> topmost captured frame (or, the beginning of the delimited
> continuation) and the bottommost (the tail end of the delimited
> continuation) are both lexically inside the generator-function. But
> there may be other functions on the stack, in between. You can't
> always statically tell which ones.  This means that generator
> semantics affect the integrity of code that isn't in a generator.

Right. We crossed the finally-may-not-run Rubicon along with Python  
(with a minor improvement, avoiding GeneratorExit), already, for  
finally clauses in the generator function itself. But this extends the  
finally integrity degradation outside of the lexical scope of the  
generator function. Good point.


> In ES5, when you call a function, you can expect it to return or throw
> eventually.  (Unless you run out of memory, or time, and the whole
> script gets terminated.)  With shallow generators, this is still true.
> A 'yield' might never return control, but function calls are ok.  But
> with generators+lambdas, almost any function call *anywhere* in the
> program might never return or throw.  This weakens 'finally', at
> least.


To make this clear with an example (thanks to Jason for some IRC  
interaction):

function gen(arg) {
     foo((lambda (x) yield x), arg);
}
function foo(callback, arg) {
     try {
         callback(arg);
     } finally {
         alert("I'm ok!");
     }
}
g = gen(42);
print(g.next()); // tell the user the meaning of life, etc.
g = null;
gc();

I think finally is the only issue, since how else can you tell that  
foo didn't see a return or exception from the callback?

It's true that this finally-may-not-run wrinkle is confined to  
generator functions in JS1.7, i.e., without lambda.

But I think any function that invokes an arbitrary other function in a  
try has to fear that its finally might not run. There could be a hard  
stop due to iloop prevention, or an uncatchable security fail-stop  
error.

FWIW I'm cool on lambda and return to label atm.

/be
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Waldemar Horwat]

This whole thing is another nail in the coffin of generators.  Generators are the root of the troublesome longjmp's which don't play well with others.

    Waldemar


Brendan Eich wrote:

> To make this clear with an example (thanks to Jason for some IRC
> interaction):
>
> function gen(arg) {
>     foo((lambda (x) yield x), arg);
> }
> function foo(callback, arg) {
>     try {
>         callback(arg);
>     } finally {
>         alert("I'm ok!");
>     }
> }
> g = gen(42);
> print(g.next()); // tell the user the meaning of life, etc.
> g = null;
> gc();
>
> I think finally is the only issue, since how else can you tell that foo
> didn't see a return or exception from the callback?
>
> It's true that this finally-may-not-run wrinkle is confined to generator
> functions in JS1.7, i.e., without lambda.
>
> But I think any function that invokes an arbitrary other function in a
> try has to fear that its finally might not run. There could be a hard
> stop due to iloop prevention, or an uncatchable security fail-stop error.
>
> FWIW I'm cool on lambda and return to label atm.
>
> /be

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

On May 14, 2009, at 1:14 PM, Neil Mix wrote:

> On May 14, 2009, at 2:38 PM, Brendan Eich wrote:
>
>> Indeed the low-precedence (same as assignment) nature of the yield  
>> unary caused JS1.7 to follow Python in requiring parentheses around  
>> yield expressions in lists: in comma expressions and actual  
>> parameter lists (although we diverged from Python by allowing an  
>> unparenthesized yield expression as the last actual argument in a  
>> call or new expression's parameter list -- Python requires  
>> foo((yield bar)) instead of foo(yield bar)).
>
> I have this idea that it would be better for yield expressions to  
> look like function calls: 'yield' '(' expression? ')'.  (Note that  
> this is only a syntactical suggestion; clearly an implementation  
> wouldn't actually treat it like a function.)  That would eliminate  
> the precedence issues Brendan cites while also making the syntax  
> backward compatible with earlier ES parsers.  Is there any technical  
> reason why that wouldn't be possible?

The syntax could work but we need to reserve yield contextually. It  
can't be a user-defined function name and a built-in function. The  
compiler must unambiguously know that yield (the built-in) is being  
called in order to make the enclosing function a generator.

This is reason enough in my view to keep yield a prefix operator and  
reserve it. I wish I had done that with eval in JS1.0!

Another reason is your duck/cow point, which I think is a separate  
point from compiler analyzability. Really, no one writes yield(...) in  
Python, and extra parens hurt (I know RSI sufferers who benefit from  
lack of shifting in Python and Ruby).

Now you could argue that we have eval already, and a cheap-ish way to  
detect direct calls to it (see https://mail.mozilla.org/pipermail/es-discuss/2009-January/008656.html)
, so why not add something like that again? One reason is that the  
cost is not zero -- cheap-ish, not cheap and not free. Adding more  
distinguished special-form instructions that fall back on call  
behavior if the callee is not the magic built-in will annoy  
implementors.

But you do make a good no-new-syntax case, ignoring the above  
considerations. I still don't buy it, given let not being doable the  
same way. We need new keywords and special forms (not many, but it's  
absurd to freeze the set from ES3 in 1999).

/be
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

On May 14, 2009, at 1:47 PM, Waldemar Horwat wrote:

> This whole thing is another nail in the coffin of generators.  
> Generators are the root of the troublesome longjmp's which don't  
> play well with others.

Are you talking about generators, or lambdas?

Generators have no longjmps at all. The only issue here is due to the  
lambda following Tennent's Correspondence Principle.

/be

>
>
>   Waldemar
>
>
> Brendan Eich wrote:
>> To make this clear with an example (thanks to Jason for some IRC  
>> interaction):
>> function gen(arg) {
>>    foo((lambda (x) yield x), arg);
>> }
>> function foo(callback, arg) {
>>    try {
>>        callback(arg);
>>    } finally {
>>        alert("I'm ok!");
>>    }
>> }
>> g = gen(42);
>> print(g.next()); // tell the user the meaning of life, etc.
>> g = null;
>> gc();
>> I think finally is the only issue, since how else can you tell that  
>> foo didn't see a return or exception from the callback?
>> It's true that this finally-may-not-run wrinkle is confined to  
>> generator functions in JS1.7, i.e., without lambda.
>> But I think any function that invokes an arbitrary other function  
>> in a try has to fear that its finally might not run. There could be  
>> a hard stop due to iloop prevention, or an uncatchable security  
>> fail-stop error.
>> FWIW I'm cool on lambda and return to label atm.
>> /be

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Mark S. Miller-2]

On Thu, May 14, 2009 at 1:22 PM, Brendan Eich <brendan@...> wrote:

> On May 14, 2009, at 12:24 PM, Jason Orendorff wrote:
>> 3. When a lambda yields, [...]
>> there may be other functions on the stack, in between. You can't
>> always statically tell which ones.  This means that generator
>> semantics affect the integrity of code that isn't in a generator.
>
> [...] this extends the finally integrity
> degradation outside of the lexical scope of the generator function. Good
> point.
>
>> [...] with generators+lambdas, almost any function call *anywhere* in the
>> program might never return or throw.  This weakens 'finally', at
>> least.
> [...]

> function gen(arg) {
>    foo((lambda (x) yield x), arg);
> }
> function foo(callback, arg) {
>    try {
>        callback(arg);
>    } finally {
>        alert("I'm ok!");
>    }
> }
> g = gen(42);
> print(g.next()); // tell the user the meaning of life, etc.
> g = null;
> gc();

Thanks all, this has been very clarifying. You both have put your
finger on what was nagging at me and explained it clearly.

It seems that either lambda or generators by themselves may be ok, but
together they make a fatal combination. Whichever we might eventually
decide to add to Harmony, we probably forever preclude the other. I do
not yet have an opinion on which.

--

    Cheers,
    --MarkM
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

On May 14, 2009, at 2:10 PM, Mark S. Miller wrote:

> It seems that either lambda or generators by themselves may be ok, but
> together they make a fatal combination. Whichever we might eventually
> decide to add to Harmony, we probably forever preclude the other. I do
> not yet have an opinion on which.

Just to be extra clear (for my benefit, at least ;-), is the problem  
the finally-may-not-run issue?

If so, did you have a different way of reasoning about the reasons  
today why finally might not run that I mentioned (iloop detection or  
other hard stop)?

/be
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Mark S. Miller-2]

On Thu, May 14, 2009 at 2:38 PM, Brendan Eich <brendan@...> wrote:
> On May 14, 2009, at 2:10 PM, Mark S. Miller wrote:
>
>> It seems that either lambda or generators by themselves may be ok, but
>> together they make a fatal combination. Whichever we might eventually
>> decide to add to Harmony, we probably forever preclude the other. I do
>> not yet have an opinion on which.
>
> Just to be extra clear (for my benefit, at least ;-), is the problem the
> finally-may-not-run issue?

Yes.


> If so, did you have a different way of reasoning about the reasons today why
> finally might not run that I mentioned (iloop detection or other hard stop)?

Those hard stops kill all further activity within that event loop.
Once a universe has been destroyed, no further bad things can happen
in that universe.

Infinite loops don't kill their universe, so this case is similar but
different. As the halting problem teaches us, an infinite loop is
generally indistinguishable from a loop that hasn't terminated yet.
Since control flow has not yet escaped the loop, it hasn't yet
bypassed the finally, and so no invariants have yet been violated.

--
    Cheers,
    --MarkM
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

On May 14, 2009, at 2:50 PM, Mark S. Miller wrote:

>> If so, did you have a different way of reasoning about the reasons  
>> today why
>> finally might not run that I mentioned (iloop detection or other  
>> hard stop)?
>
> Those hard stops kill all further activity within that event loop.
> Once a universe has been destroyed, no further bad things can happen
> in that universe.

There's always the next universe (new event starts another control  
flow). Life goes on, in the JS serial multiverse, and those finally  
clauses failed to run even though control abruptly left the lambda  
under the hypothesis.


> Infinite loops don't kill their universe, so this case is similar but
> different. As the halting problem teaches us, an infinite loop is
> generally indistinguishable from a loop that hasn't terminated yet.
> Since control flow has not yet escaped the loop, it hasn't yet
> bypassed the finally, and so no invariants have yet been violated.

True enough, but see above.

This isn't entirely academic, since information leaks include  
termination channels.

/be
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

On May 14, 2009, at 3:42 PM, Brendan Eich wrote:

>> Those hard stops kill all further activity within that event loop.
>> Once a universe has been destroyed, no further bad things can happen
>> in that universe.
>
> There's always the next universe (new event starts another control  
> flow). Life goes on, in the JS serial multiverse, and those finally  
> clauses failed to run even though control abruptly left the lambda  
> under the hypothesis.
>
>
>> Infinite loops don't kill their universe, so this case is similar but
>> different. As the halting problem teaches us, an infinite loop is
>> generally indistinguishable from a loop that hasn't terminated yet.
>> Since control flow has not yet escaped the loop, it hasn't yet
>> bypassed the finally, and so no invariants have yet been violated.
>
> True enough, but see above.
>
> This isn't entirely academic, since information leaks include  
> termination channels.

And that previous universe ended, detectably. It's not as if the loop  
is still running in the lambda when the next event-flow little bang  
blooms into a universe, since there is a big fat shared heap (no  
quantum uncertainty obscuring inter-verse sharing!) accessible by  
which to detect mutations. If the iloop were incrementing a counter,  
it is guaranteed that the counter stopped before the new event-flow  
started.

No racing scripts; run-to-completion is the observable execution model  
(ignoring web workers, which are shared nothing in JS heap terms but  
may share effects in DOM storage, etc).

/be
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Igor Bukanov-2]

2009/5/14 Brendan Eich <brendan@...>:

> function gen(arg) {
>    foo((lambda (x) yield x), arg);
> }
> function foo(callback, arg) {
>    try {
>        callback(arg);
>    } finally {
>        alert("I'm ok!");
>    }
> }
> g = gen(42);
> print(g.next()); // tell the user the meaning of life, etc.
> g = null;
> gc();
>
> I think finally is the only issue, since how else can you tell that foo
> didn't see a return or exception from the callback?
>
> It's true that this finally-may-not-run wrinkle is confined to generator
> functions in JS1.7, i.e., without lambda.

lambda can be combined with the generators if the yield semantic would
be altered to execute any finally block it resides in in the same way
the return does. Then in the above example the alert would be executed
when the lambda does yield.

This is how continuations in Rhino works with any continuation long
jump respecting lexical finally blocks.

Igor
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[David-Sarah Hopwood-2]

Brendan Eich wrote:

> On May 14, 2009, at 1:14 PM, Neil Mix wrote:
>
>> I have this idea that it would be better for yield expressions to look
>> like function calls: 'yield' '(' expression? ')'.  (Note that this is
>> only a syntactical suggestion; clearly an implementation wouldn't
>> actually treat it like a function.)  That would eliminate the
>> precedence issues Brendan cites while also making the syntax backward
>> compatible with earlier ES parsers.  Is there any technical reason why
>> that wouldn't be possible?
>
> The syntax could work but we need to reserve yield contextually.
> It can't be a user-defined function name and a built-in function. The
> compiler must unambiguously know that yield (the built-in) is being
> called in order to make the enclosing function a generator.
>
> This is reason enough in my view to keep yield a prefix operator and
> reserve it.

But that doesn't help: the argument to yield is an arbitrary expression,
so 'yield (foo)' could be either a function call or a yield-expression.
That means that this approach can at best be no simpler to implement or
specify than the function call syntax.

With the function call syntax, it would be sufficient to keep the
existing ES5 grammar for function calls, and then check after parsing
whether a MemberExpression or CallExpression followed by Arguments is
the string "yield". With the operator syntax, it's more complicated
than that because there are more syntactic contexts to consider.

> Another reason is your duck/cow point, which I think is a separate point
> from compiler analyzability. Really, no one writes yield(...) in Python,
> and extra parens hurt (I know RSI sufferers who benefit from lack of
> shifting in Python and Ruby).

Yes, those are separate points that I am not arguing against here.

--
David-Sarah Hopwood ⚥

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[David-Sarah Hopwood-2]

Brendan Eich wrote:
> On May 14, 2009, at 4:34 PM, David-Sarah Hopwood wrote:
>
>> This approach avoids any problems due to a generator being able
>> to interfere with the control flow of its callers.
>
> A generator can't interfere with the control flow of its callers.
>
> Can you give an example of what you meant by that?

I meant this:

Brendan Eich wrote:

> Jason Orendorff wrote:
>> In ES5, when you call a function, you can expect it to return or throw
>> eventually.  (Unless you run out of memory, or time, and the whole
>> script gets terminated.)  With shallow generators, this is still true.
>> A 'yield' might never return control, but function calls are ok.  But
>> with generators+lambdas, almost any function call *anywhere* in the
>> program might never return or throw.  This weakens 'finally', at
>> least.
>
> To make this clear with an example (thanks to Jason for some IRC interaction):
>
> function gen(arg) {
>     foo((lambda (x) yield x), arg);
> }
> function foo(callback, arg) {
>     try {
>         callback(arg);
>     } finally {
>         alert("I'm ok!");
>     }
> }
> g = gen(42);
> print(g.next()); // tell the user the meaning of life, etc.
> g = null;
> gc();
>
> I think finally is the only issue, since how else can you tell that foo
> didn't see a return or exception from the callback?

The consequences of this issue are not restricted to code using
'finally'; even without finally, yield+generators complicates the
conceptual model of call-return control flow, in ways that are not
possible with yield alone (since yield is restricted to only making
non-local jumps to lexically enclosing labels) or shallow generators
alone (since they can be modelled as a local transformation).

The recent huge thread on python-ideas about "yield-from" disabused
me once and for all of the idea that (non-shallow) generator semantics
are simple to understand, even for experts.

--
David-Sarah Hopwood ⚥

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[David-Sarah Hopwood-2]

[I sent this to es5-discuss, when I intended es-discuss. Sorry for the
noise for people subscribed to both.]

David-Sarah Hopwood wrote:

> Jason Orendorff wrote:
>> On Thu, May 14, 2009 at 12:25 PM, Mark S. Miller <erights@...> wrote:
>>> Given both shallow generators and lambda, I don't understand how they
>>> would interact.
>> This is a good question.
>
> So, why do we need generators anyway?
>
> I know generators are quite popular in languages that support them.
> However, there are other language features that can be used to
> provide similar (or greater) functionality, and that would not
> introduce the same problematic control flow interactions.
>
> For instance, suppose that you have dataflow concurrency, as supported
> by Oz and by recent dataflow extensions for Ruby, Python, and Scala:
>
> <http://www.mozart-oz.org/documentation/tutorial/node8.html>
> <http://github.com/larrytheliquid/dataflow/tree/master>
> <http://pypi.python.org/pypi/dataflow/0.1.0>
> <http://github.com/jboner/scala-dataflow/tree/master>
>
> Then the functionality of a generator can be implemented using a
> process/thread that extends a list or queue constructed from
> dataflow variables.
>
> This approach avoids any problems due to a generator being able
> to interfere with the control flow of its callers. It also allows
> the producer process to run truly in parallel with the consumer(s),
> possibly taking advantage of multiple CPU cores. The programming
> model is no more complicated for the cases that correspond to
> correct use of generators, because strict use of dataflow variables
> for communication between processes (with no other mutable data
> structures shared between processes) is declarative, i.e. it will
> give the same results as a sequential generator-based implementation
> would have done.
>
> Although it gives the same computational results, the dataflow-
> concurrent approach allows more flexibility in the flow control
> between producer and consumer: for example, the producer process
> can be allowed to run freely ahead of the consumer process, or
> constrained to generate only a bounded number of unconsumed
> elements. The special case where the producer process only
> generates the next unconsumed element and only starts to generate
> it when needed -- effectively sequentializing the producer and
> consumer -- corresponds to a sequential generator or coroutine.
> (This requires by-need dataflow variables, as supported by Oz and
> at least the Ruby library mentioned above.)
> However, I suspect that the bounded queue is likely to be more
> efficient and more often what is really wanted.
>
> This flow-control flexibility can be exercised by passing different
> kinds of dataflow list/queue implementation into the producer process,
> without changing the latter's code. It is possible to construct
> more general dataflow networks that can split or merge streams,
> if needed. Dataflow concurrency can also be extended to more
> expressive concurrency models that introduce nondeterminism,
> and the dataflow features only gain in usefulness in that case.
>
> If asked to pick two language features from
> {TC-respecting lambdas, generators, dataflow concurrency},
> I would always pick lambdas and dataflow concurrency, and drop
> generators as a primitive feature. It is still possible to mimic
> a generator-like API, for programmers who are used to that, in
> terms of the dataflow concurrency features. The semantics will
> be slightly different because a generator will not be able to
> directly access shared mutable data structures (it could still
> access structures containing immutable and dataflow variables),
> but this limitation is IMHO more than outweighed by the greater
> generality and potential parallelism.

--
David-Sarah Hopwood ⚥

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Waldemar Horwat]

Brendan Eich wrote:
> On May 14, 2009, at 1:47 PM, Waldemar Horwat wrote:
>
>> This whole thing is another nail in the coffin of generators.  
>> Generators are the root of the troublesome longjmp's which don't play
>> well with others.
>
> Are you talking about generators, or lambdas?

Generators.  They cause control flow to jump around between two independent functions.

THe same problem occurs without lambdas:

function gen(arg) {
  try {
    yield 1;
    yield 2;
  } finally {
    alert("Finally called up to three times?!");
  }
}

That finally will be called either 1, 2, or 3 times depending on what the caller does, which violates the principle that finally is called exactly once, no matter how you leave a scope.

    Waldemar
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

On May 14, 2009, at 6:04 PM, Waldemar Horwat wrote:

> Brendan Eich wrote:
>> On May 14, 2009, at 1:47 PM, Waldemar Horwat wrote:
>>> This whole thing is another nail in the coffin of generators.  
>>> Generators are the root of the troublesome longjmp's which don't  
>>> play well with others.
>> Are you talking about generators, or lambdas?
>
> Generators.  They cause control flow to jump around between two  
> independent functions.
>
> THe same problem occurs without lambdas:
>
> function gen(arg) {
> try {
>   yield 1;
>   yield 2;
> } finally {
>   alert("Finally called up to three times?!");
> }
> }
>
> That finally will be called either 1, 2, or 3 times depending on  
> what the caller does, which violates the principle that finally is  
> called exactly once, no matter how you leave a scope.

Did you actually test this? Firefox 2 and up work the same.

js> g = gen()
[object Generator]
js> g.next()
1
js> g.next()
2
js> g.next()
typein:6: ReferenceError: alert is not defined
js> alert=print
function print() {
     [native code]
}
js> g = gen(42)
[object Generator]
js> g.next()
1
js> g.next()
2
js> g.next()
Finally called up to three times?!
uncaught exception: [object StopIteration]

Exactly one finally.

/be

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

On May 14, 2009, at 5:29 PM, David-Sarah Hopwood wrote:

> Brendan Eich wrote:
>> On May 14, 2009, at 4:34 PM, David-Sarah Hopwood wrote:
>>
>>> This approach avoids any problems due to a generator being able
>>> to interfere with the control flow of its callers.
>>
>> A generator can't interfere with the control flow of its callers.
>>
>> Can you give an example of what you meant by that?
>
> I meant this:
>
> Brendan Eich wrote:
>> Jason Orendorff wrote:
>>> In ES5, when you call a function, you can expect it to return or  
>>> throw
>>> eventually.  (Unless you run out of memory, or time, and the whole
>>> script gets terminated.)  With shallow generators, this is still  
>>> true.
>>> A 'yield' might never return control, but function calls are ok.  
>>> But
>>> with generators+lambdas, almost any function call *anywhere* in the
>>> program might never return or throw.  This weakens 'finally', at
>>> least.
>>
>> To make this clear with an example (thanks to Jason for some IRC  
>> interaction):
>>
>> function gen(arg) {
>>    foo((lambda (x) yield x), arg);

Yeah, you meant generators *plus* lambdas -- not just generators.

Leave out lambda and there's no effect on control flow in callers of  
the thing (generator function, not lambda) that yields.

/be

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss