How would shallow generators compose with lambda?

[Brendan Eich-3]

On May 14, 2009, at 5:13 PM, David-Sarah Hopwood wrote:

> Brendan Eich wrote:
>> On May 14, 2009, at 1:14 PM, Neil Mix wrote:
>>
>>> I have this idea that it would be better for yield expressions to  
>>> look
>>> like function calls: 'yield' '(' expression? ')'.  (Note that this  
>>> is
>>> only a syntactical suggestion; clearly an implementation wouldn't
>>> actually treat it like a function.)  That would eliminate the
>>> precedence issues Brendan cites while also making the syntax  
>>> backward
>>> compatible with earlier ES parsers.  Is there any technical reason  
>>> why
>>> that wouldn't be possible?
>>
>> The syntax could work but we need to reserve yield contextually.
>> It can't be a user-defined function name and a built-in function. The
>> compiler must unambiguously know that yield (the built-in) is being
>> called in order to make the enclosing function a generator.
>>
>> This is reason enough in my view to keep yield a prefix operator and
>> reserve it.
>
> But that doesn't help: the argument to yield is an arbitrary  
> expression,
> so 'yield (foo)' could be either a function call or a yield-
> expression.
> That means that this approach can at best be no simpler to implement  
> or
> specify than the function call syntax.

Were you responding to Neil instead of me? I'm not advocating Neil's  
proposal, but it seems to me he's arguing for it to avoid the  
mandatory parentheses around the entire yield expression in almost any  
surrounding expression in which it could be embedded. He is right that  
requiring parentheses around yield's operand avoids mandatory parens  
around yield expressions in list contexts.

Simpler to implement is not the issue, but it's a wash as you say.


> With the function call syntax, it would be sufficient to keep the
> existing ES5 grammar for function calls, and then check after parsing
> whether a MemberExpression or CallExpression followed by Arguments is
> the string "yield". With the operator syntax, it's more complicated
> than that because there are more syntactic contexts to consider.

Yes, but it's not that complicated. SpiderMonkey and Rhino do it. Code  
size burden is in the noise.


>> Another reason is your duck/cow point, which I think is a separate  
>> point
>> from compiler analyzability. Really, no one writes yield(...) in  
>> Python,
>> and extra parens hurt (I know RSI sufferers who benefit from lack of
>> shifting in Python and Ruby).
>
> Yes, those are separate points that I am not arguing against here.

Ok.

I was replying to Neil, not to you (hadn't seen any message from you  
on this sub-thread).

/be

>
>
> --
> David-Sarah Hopwood ⚥
>
> _______________________________________________
> es-discuss mailing list
> es-discuss@...
> https://mail.mozilla.org/listinfo/es-discuss

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[kevin curtis]

If a function is used instead of a lambda, do the the same - syntactic
or semantic - problems arise?

 foo((lambda (x) yield x), arg);

to:

 foo((function (x) yield x), arg);  // js1.7 expression closure syntax
OR
 foo(function (x) { yield x}, arg);

On Fri, May 15, 2009 at 6:11 AM, Brendan Eich <brendan@...> wrote:

> On May 14, 2009, at 6:04 PM, Waldemar Horwat wrote:
>
>> Brendan Eich wrote:
>>>
>>> On May 14, 2009, at 1:47 PM, Waldemar Horwat wrote:
>>>>
>>>> This whole thing is another nail in the coffin of generators.
>>>>  Generators are the root of the troublesome longjmp's which don't play well
>>>> with others.
>>>
>>> Are you talking about generators, or lambdas?
>>
>> Generators.  They cause control flow to jump around between two
>> independent functions.
>>
>> THe same problem occurs without lambdas:
>>
>> function gen(arg) {
>> try {
>>  yield 1;
>>  yield 2;
>> } finally {
>>  alert("Finally called up to three times?!");
>> }
>> }
>>
>> That finally will be called either 1, 2, or 3 times depending on what the
>> caller does, which violates the principle that finally is called exactly
>> once, no matter how you leave a scope.
>
> Did you actually test this? Firefox 2 and up work the same.
>
> js> g = gen()
> [object Generator]
> js> g.next()
> 1
> js> g.next()
> 2
> js> g.next()
> typein:6: ReferenceError: alert is not defined
> js> alert=print
> function print() {
>    [native code]
> }
> js> g = gen(42)
> [object Generator]
> js> g.next()
> 1
> js> g.next()
> 2
> js> g.next()
> Finally called up to three times?!
> uncaught exception: [object StopIteration]
>
> Exactly one finally.
>
> /be
>
> _______________________________________________
> es-discuss mailing list
> es-discuss@...
> https://mail.mozilla.org/listinfo/es-discuss
>

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[David-Sarah Hopwood-2]

John Cowan wrote:
> David-Sarah Hopwood scripsit:
>
>> Then the functionality of a generator can be implemented using a
>> process/thread that extends a list or queue constructed from
>> dataflow variables.
>
> Quite so.  How, if at all, do these dataflow variables differ from
> Prolog variables?

Prolog itself is a sequential language (although there have been
many concurrent extensions of it). Prolog supports logic variables,
which are a generalisation of single-assignment variables that use
a unification algorithm for update. Dataflow variables are generalised
from single-assignment variables in a different direction, in order to
support sychronization between concurrent threads or lightweight
processes.

(view in fixed-width font)

  single-assignment   + unification       logic
     variables       --------------->   variables
         |                                  |
         | + concurrency                    | + concurrency
         |                                  |
         v            + unification         v
  dataflow variables ---------------> concurrent logic
                                        variables


Historically (and at the risk of oversimplifying), the development
was from logic programming languages such as Prolog, to concurrent
logic languages such as Flat Concurrent Prolog, and later to
concurrent constraint languages such as AKL and Oz. Dataflow
variables are a simplified version of concurrent logic variables
that do not support update by full unification, but do support
being bound more than once to identical values.

> As a less complex option, lambdas + Lua-style semicoroutines.

Why do you say that the option I've suggested is complex?
Anecdotal evidence from programmers with experience of Oz and
other languages supporting dataflow concurrency suggests that
this programming model is typically found to be quite simple to
use. Certainly it is not complicated to specify or implement.

> These are
> first-class and deep; you can yield at any point, not just lexically
> within the coroutine, but don't support resuming arbitrary coroutines,
> only the caller (but it's easy to write a general coroutine dispatcher).
> Lua also provides multiple value returns for both coroutines and
> functions, but currently has no support for native threads.

I don't agree that coroutines are less complex than dataflow
concurrency. In fact, coroutines present almost exactly the same
potential complexities in the programming model as threading with
shared memory (which is significantly more complex than dataflow
concurrency), but without the performance advantage of being able
to support actual parallel execution.

--
David-Sarah Hopwood ⚥

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Jason Orendorff]

On Thu, May 14, 2009 at 7:35 PM, David-Sarah Hopwood
<david-sarah@...> wrote:

>> For instance, suppose that you have dataflow concurrency, as supported
>> by Oz and by recent dataflow extensions for Ruby, Python, and Scala:
>>
>> <http://www.mozart-oz.org/documentation/tutorial/node8.html>
>> <http://github.com/larrytheliquid/dataflow/tree/master>
>> <http://pypi.python.org/pypi/dataflow/0.1.0>
>> <http://github.com/jboner/scala-dataflow/tree/master>
>>
>> Then the functionality of a generator can be implemented using a
>> process/thread that extends a list or queue constructed from
>> dataflow variables.

I think you are proposing a model in which threads cannot share
mutable state and cannot have side effects which other threads can
see.

Even the simplest generator use cases can't be implemented using
threads and dataflow variables in that kind of model.  To take a silly
example:

  function iter(arraylike) {
      for (var i = 0; i < arraylike.length; i++)
          yield arraylike[i];
  }

  for (elt in iter(document.getElementsByTagName("A")))
      ...

DOM NodeLists are mutable and so are the nodes being yielded.

I like the idea of going after the concurrency problem now.  I like
the idea of doing one language feature instead of two (concurrency and
generators).  Do you have a solution in mind for this kind of problem?

-j
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

On May 15, 2009, at 11:15 AM, Jason Orendorff wrote:

> I like the idea of going after the concurrency problem now.  I like
> the idea of doing one language feature instead of two (concurrency and
> generators).  Do you have a solution in mind for this kind of problem?

Not to rain on anyone's parade here, but Ecma TC39 is not the right  
group to do research and then standardize it.

We can spend unlimited es-discuss messages on this, of couse, but (a)  
at a cost to other discussions; (b) not if the undelivered, always-
over-the-next-hill promise of a unified solution is used to stop  
progress of extensions toward standardization in TC39. One can easily  
wait forever for the best -- Voltaire had something to say about that.

Implementors should experiment and take risks in the market. TC39  
should standardize after that all shakes out. If we can have  
agreements in committee on direction and even details, great. But the  
committee does not have the breadth or competence to invent much  
that's new while standardizing.

Meanwhile, until some two-birds-one-stone solution emerges (which  
could happen, I'm skeptical about it happening soon), we have  
generators shipping in SpiderMonkey and Rhino, and they help  
programmers avoid having to hand-code tedious and error-prone state  
management (control blocks for callbacks) when solving common  
iteration and pseudo-threading problems.

But perhaps generators could be informative for a while. The reason  
they came up was, as Mark said, because they were an extension already  
considered Harmonious in TC39.

/be
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Graydon Hoare-3]

On 15/05/09 11:15 AM, Jason Orendorff wrote:

> Even the simplest generator use cases can't be implemented using
> threads and dataflow variables in that kind of model.  To take a silly
> example:
>
>    function iter(arraylike) {
>        for (var i = 0; i<  arraylike.length; i++)
>            yield arraylike[i];
>    }
>
>    for (elt in iter(document.getElementsByTagName("A")))
>        ...
>
> DOM NodeLists are mutable and so are the nodes being yielded.

Oh, this will *work* in the supposed "dataflow" system. It works in the
libraries David-Sarah pointed to. It'll just be completely unsafe in the
bolt-on-the-language cases (i.e. "not Oz"). Because in those cases, the
"dataflow" system is being used as little more than a voluntary
half-duplex channel scheme. Key point being voluntary. Threads can still
interfere via the shared graph, bypassing the channels. Which everyone will.

Unfortunately ES, ruby, python, etc. are all cyclic-memory,
universal-shared-ownership OO languages, so you are transmitting nodes
in your shared graph between threads. Even if you lock each thread in a
box, you can't reasonably make a full copy of the world on each
transmission, and confinement isn't designed-in to the ownership
structure, so you're forced to do something cheesy like "attempt deep
copy and throw when someone tries to transmit any cyclic bits" as a
heuristic for preventing transmission of something that might be
shared-ownership.

This is a design problem in the language. You want to be writing in
newsqueak or erlang or something properly designed for channels (CoW
acyclic ownership, or immutability, take your pick). ES isn't. This is
why workers[1] live in isolated universes and communicate via JSON (or
possibly this has been changed to an acyclic "structured clone"[2]).
Anyway, both approaches do the "deep copy and throw on cyclic" thing.
It's a kludge, but the best you can do here. The language wasn't
designed to blur concurrency and iteration. If you try, you'll just get
a mess (or systematic lack of safety).

Generators are comparatively much lighter (eg. no OS thread) and can be
used to structure walks through existing object graphs, by a single
thread, without being locked in a box or marshalling yield values.
They're just an iteration-modularity device.

-Graydon

[1] http://www.whatwg.org/specs/web-workers/current-work/
[2] http://www.w3.org/TR/html5/infrastructure.html#structured-clone
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

On May 15, 2009, at 2:34 AM, kevin curtis wrote:

> If a function is used instead of a lambda, do the the same - syntactic
> or semantic - problems arise?

No, because yield in a function makes it a generator, whereas in the  
lambda proposal extended to treat yield as return/break/continue are  
treated (per Tennent's Correspondence Principle), yield does not make  
the lamda some kind of generator-lambda -- instead it yields the  
nearest enclosing function (if active; else throw).


> foo((lambda (x) yield x), arg);
>
> to:
>
> foo((function (x) yield x), arg);  // js1.7 expression closure syntax
> OR
> foo(function (x) { yield x}, arg);

That passes a generator function to foo. If foo calls it, foo gets a  
generator-iterator, which if used in a for-in construct, or explicitly  
iterated via .next/send/throw, yields x once then throws StopIteration.

/be

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Waldemar Horwat]

Brendan Eich wrote:

> On May 14, 2009, at 6:04 PM, Waldemar Horwat wrote:
>
>> Brendan Eich wrote:
>>> On May 14, 2009, at 1:47 PM, Waldemar Horwat wrote:
>>>> This whole thing is another nail in the coffin of generators.  
>>>> Generators are the root of the troublesome longjmp's which don't
>>>> play well with others.
>>> Are you talking about generators, or lambdas?
>>
>> Generators.  They cause control flow to jump around between two
>> independent functions.
>>
>> THe same problem occurs without lambdas:
>>
>> function gen(arg) {
>> try {
>>   yield 1;
>>   yield 2;
>> } finally {
>>   alert("Finally called up to three times?!");
>> }
>> }
>>
>> That finally will be called either 1, 2, or 3 times depending on what
>> the caller does, which violates the principle that finally is called
>> exactly once, no matter how you leave a scope.
>
> Did you actually test this? Firefox 2 and up work the same.
>
> js> g = gen()
> [object Generator]
> js> g.next()
> 1
> js> g.next()
> 2
> js> g.next()
> typein:6: ReferenceError: alert is not defined
> js> alert=print
> function print() {
>     [native code]
> }
> js> g = gen(42)
> [object Generator]
> js> g.next()
> 1
> js> g.next()
> 2
> js> g.next()
> Finally called up to three times?!
> uncaught exception: [object StopIteration]
>
> Exactly one finally.

That behavior is wrong in the case that g gets forgotten after calling next just once.  You're silently exiting through a finally clause without calling it.

    Waldemar
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

On May 15, 2009, at 1:47 PM, Waldemar Horwat wrote:

>> Did you actually test this? Firefox 2 and up work the same.
>> js> g = gen()
>> [object Generator]
>> js> g.next()
>> 1
>> js> g.next()
>> 2
>> js> g.next()
>> typein:6: ReferenceError: alert is not defined
>> js> alert=print
>> function print() {
>>    [native code]
>> }
>> js> g = gen(42)
>> [object Generator]
>> js> g.next()
>> 1
>> js> g.next()
>> 2
>> js> g.next()
>> Finally called up to three times?!
>> uncaught exception: [object StopIteration]
>> Exactly one finally.
>
> That behavior is wrong in the case that g gets forgotten after  
> calling next just once.

Progress! You're not alleging longjmps or multiple finallys ;-).


> You're silently exiting through a finally clause without calling it.


You must mean where I hit that ReferenceError due to alert not being  
defined, then reassigned to g without closing it or calling g.next()  
and hitting the StopIteration.

(I //totally// so meant to do that in order to provoke this  
discussion. :-P)

The alternative, which we prototyped and rejected, but which CPython  
implements, is to automate calling the generator's close method from  
the garbage collector (close "throws a return" at the generator,  
causing finallys to run). But this is a burden on either or both of  
content authors and implementators, since it makes finally execution  
unpredictably tardy, requires certain kinds of GC, exposes two-phase  
finalization, etc.

In browsers, automating close from the GC opens up trivial denial of  
service attacks that could prevent pages from being unloaded, which  
browsers can and must defeat in ways that again mean "finally does not  
always run."

The fact is, in browsers at least, finally does not always run --  
ignoring generators. There are good DOS-prevention reasons for  
skipping it.

If you want automated close calling in the absence of DOS suppression,  
use for-in. This was suggested by Chris Hansen here:

https://mail.mozilla.org/pipermail/es-discuss/2007-January/003743.html

That whole "Immediate closing of iterators" thread is worth a read if  
you missed it.

/be
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[David-Sarah Hopwood-2]

Brendan Eich wrote:

> On May 14, 2009, at 5:13 PM, David-Sarah Hopwood wrote:
>> Brendan Eich wrote:
>>> On May 14, 2009, at 1:14 PM, Neil Mix wrote:
>>>
>>>> I have this idea that it would be better for yield expressions to look
>>>> like function calls: 'yield' '(' expression? ')'. [...]
>>>
>>> The syntax could work but we need to reserve yield contextually.
>>> It can't be a user-defined function name and a built-in function. The
>>> compiler must unambiguously know that yield (the built-in) is being
>>> called in order to make the enclosing function a generator.
>>>
>>> This is reason enough in my view to keep yield a prefix operator and
>>> reserve it.
>>
>> But that doesn't help: the argument to yield is an arbitrary expression,
>> so 'yield (foo)' could be either a function call or a yield-expression.
>> That means that this approach can at best be no simpler to implement or
>> specify than the function call syntax.
>
> Were you responding to Neil instead of me?

I was responding to "This is reason enough in my view to keep yield a
prefix operator and reserve it".

My point was that the example of 'yield (foo)' (that is, yield as a
prefix operator applied to the expression '(foo)') shows that the prefix
operator syntax cannot possibly be easier to specify than the function
call syntax -- contrary to what you appeared to be arguing above.
In fact I think it is much harder to do correctly.

>> With the function call syntax, it would be sufficient to keep the
>> existing ES5 grammar for function calls, and then check after parsing
>> whether a MemberExpression or CallExpression followed by Arguments is
>> the string "yield". With the operator syntax, it's more complicated
>> than that because there are more syntactic contexts to consider.
>
> Yes, but it's not that complicated. SpiderMonkey and Rhino do it. Code
> size burden is in the noise.

Hmm. SpiderMonkey and Rhino use ad-hoc parsers. Show me an unambiguous
grammar for the prefix yield operator, and then I'll concede the point :-)

--
David-Sarah Hopwood ⚥

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[David-Sarah Hopwood-2]

Jason Orendorff wrote:

> On Thu, May 14, 2009 at 7:35 PM, David-Sarah Hopwood
> <david-sarah@...> wrote:
>>> For instance, suppose that you have dataflow concurrency, as supported
>>> by Oz and by recent dataflow extensions for Ruby, Python, and Scala:
>>>
>>> <http://www.mozart-oz.org/documentation/tutorial/node8.html>
>>> <http://github.com/larrytheliquid/dataflow/tree/master>
>>> <http://pypi.python.org/pypi/dataflow/0.1.0>
>>> <http://github.com/jboner/scala-dataflow/tree/master>
>>>
>>> Then the functionality of a generator can be implemented using a
>>> process/thread that extends a list or queue constructed from
>>> dataflow variables.
>
> I think you are proposing a model in which threads cannot share
> mutable state and cannot have side effects which other threads can
> see.

No. Even in a pure dataflow model, binding a dataflow variable
causes a side effect that other threads can see, if they have access
to that variable.

A pure dataflow model is sufficient to simulate generators, but as
you point out, it doesn't suport some of the things you might want to
do from a generator (or from other code). What I would like to propose
for Harmony is a dataflow + asynchronous message passing model.

> Even the simplest generator use cases can't be implemented using
> threads and dataflow variables in that kind of model.

The use case you've identified requires message passing in addition
to dataflow variables, and it ideally requires the message passing
to have a particular semantics that maximises compatibility with
existing JavaScript. Actually this is needed to support the call to
a DOM object in your example, rather than to support generators.
But you are right that dataflow concurrency is probably not
sufficient on its own if we want to make Harmony a practical
concurrent language.

Bear in mind, though, that adding message passing significantly
increases the expressiveness of the language, so we should take that
into account when comparing the complexity of (lambdas + generators)
with that of (lambdas + dataflow concurrency + message passing).
The latter is much more powerful.

I'll describe the details of what I'm proposing, and how it applies
to your example (which does work in this model), in a separate
message, with subject "Dataflow concurrency and message passing
for Harmony".

--
David-Sarah Hopwood ⚥

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

On May 16, 2009, at 5:50 PM, David-Sarah Hopwood wrote:

> But you are right that dataflow concurrency is probably not
> sufficient on its own if we want to make Harmony a practical
> concurrent language.

That is not a goal at this point, and TC39 wouldn't hold the "H"-
release for it.


> Bear in mind, though, that adding message passing significantly
> increases the expressiveness of the language, so we should take that
> into account when comparing the complexity of (lambdas + generators)
> with that of (lambdas + dataflow concurrency + message passing).
> The latter is much more powerful.

Who says lambdas are "in"? Who says power is the good to maximize?  
We've talked about avoiding things like lambdas with completion result  
and TCP hazards, and full call/cc, to avoid inappropriate programmer  
and implementor burdens and honey traps.

Generators do not require concurrency in the implementation or the  
language, and they're easy to implement in current engines.

Harmony is not a research language. Anything like what you describe  
would have to be prototyped by one or more of the major implementors.  
We'll see. Some grains of salt are prescribed along with enthusiasm in  
the mean time.


> I'll describe the details of what I'm proposing, and how it applies
> to your example (which does work in this model), in a separate
> message, with subject "Dataflow concurrency and message passing
> for Harmony".

Looking forward to it.

/be
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[David-Sarah Hopwood-2]

Brendan Eich wrote:
> On May 16, 2009, at 5:50 PM, David-Sarah Hopwood wrote:
>
>> Bear in mind, though, that adding message passing significantly
>> increases the expressiveness of the language, so we should take that
>> into account when comparing the complexity of (lambdas + generators)
>> with that of (lambdas + dataflow concurrency + message passing).
>> The latter is much more powerful.
>
> Who says lambdas are "in"?

The above argument applies equally to a comparison between just
(generators) and (dataflow concurrency + message passing).

(But I would be very disappointed if Harmony doesn't end up with
lambdas :-)

> Who says power is the good to maximize?

Expressive power isn't good to maximize if it is at the expense of
safety or the ability to reason about programs (preferably using
local reasoning whereever possible). However, if it is feasible to
increase expressive power *without* compromising those properties --
as I believe it is in this case -- then yes, I think that's definitely
a good thing.

(Implementation complexity and performance are also important, but
personally I consider them to be lower priorities than safety and
expressiveness -- especially since safety often has a lower performance
cost than many people believe.)

Remember, I started this thread by arguing against (lambdas + generators)
precisely because that combination caused a safety problem. So we
have no disagreement on the principle that unsafe combinations of
features should be avoided even if they would be highly expressive.
We disagree on whether the identified problem is best avoided by
ditching lambdas (and replacing the lost functionality with what?)
or by ditching generators (and exceeding their functionality using
dataflow concurrency and message passing, neither of which interact
badly with lambdas).

Note that the fact that a language supports highly expressive
features doesn't mean that any particular program must or should
use those features. A program that is restricted to only using
features that are necessary to solve the problem that it is supposed
to be solving, will probably be easier to understand, debug, and
maintain. This is essentially the "Principle of Least Expressiveness"
discussed in the book "Concepts, Techniques and Models of Computer
Programming". However, programming in a highly expressive language
that supports multiple programming paradigms (but designed with
considerable attention to maintaining safety properties) makes it
easier, not harder, to follow this principle. And maybe JavaScript
just isn't cut out to be that language, but I'd like to at least
have a stab at making it closer.

--
David-Sarah Hopwood ⚥

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

On May 16, 2009, at 9:29 PM, David-Sarah Hopwood wrote:

> (Implementation complexity and performance are also important, but
> personally I consider them to be lower priorities than safety and
> expressiveness -- especially since safety often has a lower  
> performance
> cost than many people believe.)

I'm still looking for evidence other than your certainty re: the last  
dispute about semantic cleanliness (by your lights) vs. performance:

https://mail.mozilla.org/pipermail/es-discuss/2009-February/008862.html

When we see SFX (Nitro, heh), V8, and TraceMonkey on board then I will  
be convinced. Until then hand-waves and generalizations don't really  
cut it.


> Remember, I started this thread by arguing against (lambdas +  
> generators)
> precisely because that combination caused a safety problem.

It's already the case that finallys may not run, whether or not you  
add generators and lambdas.


> So we
> have no disagreement on the principle that unsafe combinations of
> features should be avoided even if they would be highly expressive.
> We disagree on whether the identified problem is best avoided by
> ditching lambdas (and replacing the lost functionality with what?)

Waldemar argued for reforming function. If we add rest and default  
parameters then there isn't much to do, other than try to force  
Tennent into a language it doesn't fit.


> Note that the fact that a language supports highly expressive
> features doesn't mean that any particular program must or should
> use those features.

Yeah, yeah... There are many bodies, fingers and toes already chalked  
up to this kind of thinking applied elsewhere (C++).

Besides the honey traps, implementation barriers should not be raised  
too high. One good reason to avoid this is to achieve interoperation  
among many competing browser vendors. To pick on C++ again, it took  
too long to get semi-interoperable implementations due in part to the  
complexity inherent in all that expressiveness. There are other  
languages from which to learn the same cautionary tale.


> However, programming in a highly expressive language
> that supports multiple programming paradigms (but designed with
> considerable attention to maintaining safety properties) makes it
> easier, not harder, to follow this principle. And maybe JavaScript
> just isn't cut out to be that language, but I'd like to at least
> have a stab at making it closer.

On this very general point, we agree.

/be
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

This may be breaking a butterfly on a wheel, but I am game if it  
improves the state of the strawman proposals.

On May 15, 2009, at 7:16 PM, David-Sarah Hopwood wrote:

> My point was that the example of 'yield (foo)' (that is, yield as a
> prefix operator applied to the expression '(foo)') shows that the  
> prefix
> operator syntax cannot possibly be easier to specify than the function
> call syntax -- contrary to what you appeared to be arguing above.

Analogous to direct vs. indirect eval in ES5 (15.1.2.1.1), there is no  
purely syntactic specification for what Neil proposes. A runtime check  
is required. So I don't see why you are focusing only on syntax here.

All the costs count, but considering "who pays" can trump "how much";  
see the postscript below. I'm more concerned with usability than  
implementability, but the latter is not hard and there's no soundness  
issue.


> In fact I think it is much harder to do correctly.

See below, it's quite easy.


>> Yes, but it's not that complicated. SpiderMonkey and Rhino do it.  
>> Code
>> size burden is in the noise.
>
> Hmm. SpiderMonkey and Rhino use ad-hoc parsers. Show me an unambiguous
> grammar for the prefix yield operator, and then I'll concede the  
> point :-)

Python 2.5 has essentially the same issues, and it has a formal  
grammar (several, the one CPython uses and at least one other at  
antlr.org). From CPython's Grammar/Grammar EBNF:

$ grep yield Grammar
expr_stmt: testlist (augassign (yield_expr|testlist) |
                      ('=' (yield_expr|testlist))*)
flow_stmt: break_stmt | continue_stmt | return_stmt | raise_stmt |  
yield_stmt
yield_stmt: yield_expr
atom: ('(' [yield_expr|testlist_gexp] ')' |
yield_expr: 'yield' [testlist]

What we are discussing is almost trivial in terms of usability, but  
the difference is big enough to care about, IMHO. Either you have to  
write, in Neil's approach:

yield (E)      (or just yield() to yield undefined)

or in the Pythonic approach:

(yield E)      (or just (yield))

with the parentheses in the Pythonic approach not required when the  
yield expression is either (a) the entire expression in an expression  
statement; (b) the right-hand side of an assignment.

Python differs from JS by having only assignment statements, not  
assignment expressions, which simplifies the problem a bit, as the  
grep output cited above shows.

In any case, yield E; as a statement is the most common kind of yield  
in real-world Python code, both because it came first and because  
sending values to generators is less commonly done than iterating  
generators.

So with either approach you have some "insignificant, silly  
parentheses" -- but with Neil's approach you have "lots" more because  
the expression-statement yield must have at least () after the keyord.

This tips the usability contest in favor of the Pythonic approach.

In no case are there ambiguities for a bottom up grammar. The right  
parenthesis avoids nasty automatic semicolon insertion problems that  
plagued the unbracketed low-precedence expression on the right of a  
high-precedence prefix in the case of expression-bodied functions, AKA  
"expression closures" (see https://mail.mozilla.org/pipermail/es-discuss/2008-October/007888.html)
.

To confirm this, I patched WebKit's Bison grammar for JS to support  
yield //a la// JS1.7. I didn't hook up the semantic actions, just  
verified zero reduce-reduce conflicts. Patch attached below --  
comments welcome (including from you WebKit lurkers :-).

/be

P.S. Bean-counting productions in a bottom-up grammar is counting the  
wrong cost, given the few implementors and their greater skills,  
compared to the many users who need to be spared those painful and  
fruitless parentheses.

Bean-counting productions in a traditional bottom-up grammar is also  
silly given how many are required for ES1-3. From the patch, notice  
all the existing specialized NoIn/NoBF/NoNode variants in parser/
Grammar.y (NoBF = No Brace at Front, as far as I can tell).

Much of that production-forking could be avoided by using either  
parameterized LR parsing (www.cs.lth.se/Research/LDTA2004/d09_TheimannNeubauer.pdf)
, or else top-down ("ad-hoc" or not :-P) parsing.






_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

On May 17, 2009, at 11:00 AM, Brendan Eich wrote:

> Python differs from JS by having only assignment statements, not  
> assignment expressions, which simplifies the problem a bit, as the  
> grep output cited above shows.

Of course this makes trouble with automatic semicolon insertion,  
unless assignment statements are split from assignment expressions.  
D'oh!

js> function g() {z = yield a ? f : x++()}
typein:2: SyntaxError: missing ; before statement:
typein:2: function g() {z = yield a ? f : x++()}
typein:2: .................................^
js> function g() {(z = yield a ? f : x++)()}


> To confirm this, I patched WebKit's Bison grammar for JS to support  
> yield //a la// JS1.7. I didn't hook up the semantic actions, just  
> verified zero reduce-reduce conflicts. Patch attached below --  
> comments welcome (including from you WebKit lurkers :-).

Patching this is harder than I thought it would be. After hacking at  
the Bison grammar a bit, I concede your point about implementation  
difficulty!

Usability still favors the Python approach, at least for yield  
statements.

/be



_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

On May 17, 2009, at 11:00 AM, Brendan Eich wrote:

> Bean-counting productions in a traditional bottom-up grammar is also  
> silly given how many are required for ES1-3. From the patch, notice  
> all the existing specialized NoIn/NoBF/NoNode variants in parser/
> Grammar.y (NoBF = No Brace at Front, as far as I can tell).

Correction:

%type <expressionNode>  MemberExpr MemberExprNoBF /* BF => brace or  
function */

/be
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Mark S. Miller-2]

On Sun, May 17, 2009 at 11:00 AM, Brendan Eich <brendan@...> wrote:
> Analogous to direct vs. indirect eval in ES5 (15.1.2.1.1), there is no
> purely syntactic specification for what Neil proposes. A runtime check is
> required. So I don't see why you are focusing only on syntax here.

I don't follow. What runtime check? For the eval operator, the runtime
check is whether the value of the eval variable is the original global
eval function. It makes no sense to have a corresponding global yield
function value.

--
    Cheers,
    --MarkM
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

On May 17, 2009, at 12:43 PM, Mark S. Miller wrote:

> On Sun, May 17, 2009 at 11:00 AM, Brendan Eich <brendan@...>  
> wrote:
>> Analogous to direct vs. indirect eval in ES5 (15.1.2.1.1), there is  
>> no
>> purely syntactic specification for what Neil proposes. A runtime  
>> check is
>> required. So I don't see why you are focusing only on syntax here.
>
> I don't follow. What runtime check? For the eval operator, the runtime
> check is whether the value of the eval variable is the original global
> eval function. It makes no sense to have a corresponding global yield
> function value.

If we reserve yield then you're right. One of the appealing (at least  
to me) aspects of Neil's suggestion was that it would avoid opt-in  
versioning required by reserving yield (which is used in extant web  
content, or was when we tried reserving it without opt-in versioning  
-- the particular use was as a formal parameter name, used as a flag  
not a function).

/be

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[David-Sarah Hopwood-2]

Brendan Eich wrote:

> This may be breaking a butterfly on a wheel, but I am game if it
> improves the state of the strawman proposals.
>
> On May 15, 2009, at 7:16 PM, David-Sarah Hopwood wrote:
>
>> My point was that the example of 'yield (foo)' (that is, yield as a
>> prefix operator applied to the expression '(foo)') shows that the prefix
>> operator syntax cannot possibly be easier to specify than the function
>> call syntax -- contrary to what you appeared to be arguing above.
>
> Analogous to direct vs. indirect eval in ES5 (15.1.2.1.1), there is no
> purely syntactic specification for what Neil proposes. A runtime check
> is required. So I don't see why you are focusing only on syntax here.

Maybe I misunderstood what Neil was proposing, but I took him to mean
something equivalent to:

 - parse using essentially the current grammar.
 - if the text of the CallExpression in 'CallExpression Arguments',
   or of the MemberExpression in 'MemberExpression Arguments',
   is "yield" (excluding comments and whitespace), then the call
   is a yield-expression.

This is a purely syntactic specification.

It does have some quirks: if you define a function called 'yield' and
try to call it, then the call will still be treated as a yield-expression
rather than a function call. For that reason I'm going cold on the idea.
I would prefer something entirely unambiguous, like

  @yield expr

or an opt-in that makes 'yield' (and 'let') reserved everywhere.
Either would make parentheses around a yield expression unnecessary,
or at least not necessary in as many cases. ('yield' would behave
syntactically like 'typeof', perhaps with lower precedence.)

[...]
>> Show me an unambiguous grammar for the prefix yield operator, and
>> then I'll concede the point :-)
>
> Python 2.5 has essentially the same issues,

Python does not attempt to contextually reserve 'yield'.
It uses an opt-in that makes 'yield' a keyword that is reserved
everywhere. From PEP 255:

#  A new statement is introduced:
#
#      yield_stmt:    "yield" expression_list
#
#  "yield" is a new keyword, so a future statement[8] is needed to phase
#  this in:  in the initial release, a module desiring to use generators
#  must include the line
#
#      from __future__ import generators
#
#  near the top (see PEP 236[8]) for details).  Modules using the
#  identifier "yield" without a future statement will trigger warnings.
#  In the following release, yield will be a language keyword and the
#  future statement will no longer be needed.

> and it has a formal grammar
> (several, the one CPython uses and at least one other at antlr.org).
> From CPython's Grammar/Grammar EBNF:
>
> $ grep yield Grammar
> expr_stmt: testlist (augassign (yield_expr|testlist) |
>                      ('=' (yield_expr|testlist))*)
> flow_stmt: break_stmt | continue_stmt | return_stmt | raise_stmt |
> yield_stmt
> yield_stmt: yield_expr
> atom: ('(' [yield_expr|testlist_gexp] ')' |
> yield_expr: 'yield' [testlist]

That would be ambiguous, if (hypothetically) 'yield' were not a
keyword in Python. For example

  (yield (foo))

would match both a parenthesised <yield_expr>, and a parenthesised
<NAME> 'yield' followed by a trailer giving function arguments.
(This involves the <power>, <trailer>, <arglist> and <argument>
productions that weren't found by your grep.)

Similarly for (yield - 1) or (yield + 1) or (yield /re/g) in Harmony.
If these are treated as being yield-expressions, then that is
not compatible with any ES3/5 code using 'yield' as a variable
name, but if they are treated as having their ES5 meaning, then
there are a bunch of ugly special cases for what token can
validly begin the expression after 'yield'.

> What we are discussing is almost trivial in terms of usability, but the
> difference is big enough to care about, IMHO.

Unambiguous grammar first, please.

--
David-Sarah Hopwood ⚥

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss