JSON parser grammar

[Oliver Hunt-2]

On Jun 3, 2009, at 2:15 PM, Mark S. Miller wrote:

> On Wed, Jun 3, 2009 at 2:10 PM, Robert Sayre <sayrer@...> wrote:
>> OK, so, all such deviations will be considered bugs by  
>> implementations
>> that purport to conform. Right?
>
> Yes.

Awesome.

--Oliver

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Waldemar Horwat]

Here are my views on this.

> 2) Do we want to permit conforming implementations to extend the JSON grammar that they recognize?  This probably could be done by extending the syntax error extension allowance in section 16 to include the JSON grammar.  If we allow this then most of the observed variation for the current emerging implementation that we have been talking about would probably be acceptable extensions.

Yes, unless we want to have a confusing proliferation of different JSON method names as we add new data types to the language in the future.

> 3) If we disallow JSON grammar extensions (for JSON.parse) should we extend the existing grammar with some Postel's Law flexibility?

No, except for things we explicitly discuss and approve here.

> Here are the individual cases that I know of to consider:
>
> a) Allow strings, numbers, Booleans, and null in addition to objects and arrays as top level JSON text.

Yes.

> b) Permit leading zeros on numbers either with or without octal implications.

No.

> c) Trailing commas in objects and arrays

No.

> d)  Holes in arrays, eg [1,,3]

No.

> e) Allow some/all control characters to appear unescaped in JSON string literals. Which ones?

Don't care, as long as all of them (except line terminators) are treated alike.

> f) Allow single quotes within JSON text as string delimiters

No.

    Waldemar
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Douglas Crockford]

Waldemar Horwat wrote:
> Here are my views on this.
>
>> 2) Do we want to permit conforming implementations to extend the JSON
>> grammar that they recognize?  This probably could be done by extending
>> the syntax error extension allowance in section 16 to include the JSON
>> grammar.  If we allow this then most of the observed variation for the
>> current emerging implementation that we have been talking about would
>> probably be acceptable extensions.

JSON is done. JSON will not be revised. Someday it might be replaced and
that replacement will have a different name and likely a different model.
Chapter 16 should not give a license to fiddle with the JSON grammar.
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Waldemar Horwat]

Douglas Crockford wrote:

> Waldemar Horwat wrote:
>>> 2) Do we want to permit conforming implementations to extend the JSON
>>> grammar that they recognize?  This probably could be done by
>>> extending the syntax error extension allowance in section 16 to
>>> include the JSON grammar.  If we allow this then most of the observed
>>> variation for the current emerging implementation that we have been
>>> talking about would probably be acceptable extensions.
>
> JSON is done. JSON will not be revised. Someday it might be replaced and
> that replacement will have a different name and likely a different model.
> Chapter 16 should not give a license to fiddle with the JSON grammar.

OK, so we need not discuss any new numeric types any further in committee because it would be impossible to round-trip them through JSON.  Do we have agreement on that?

    Waldemar
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Allen Wirfs-Brock-2]

>-----Original Message-----
>From: Waldemar Horwat [mailto:waldemar@...]
>
>OK, so we need not discuss any new numeric types any further in
>committee because it would be impossible to round-trip them through
>JSON.  Do we have agreement on that?
>

I think that's reality.  Languages with multiple numeric types already have to deal with encoding to/from standard JSON format. My understanding is the goal of JSON is simple data interchange rather than supporting the union of all data types available in all languages now or in the future.


_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Douglas Crockford]

Waldemar Horwat wrote:

> Douglas Crockford wrote:
>> Waldemar Horwat wrote:
>>>> 2) Do we want to permit conforming implementations to extend the
>>>> JSON grammar that they recognize?  This probably could be done by
>>>> extending the syntax error extension allowance in section 16 to
>>>> include the JSON grammar.  If we allow this then most of the
>>>> observed variation for the current emerging implementation that we
>>>> have been talking about would probably be acceptable extensions.
>>
>> JSON is done. JSON will not be revised. Someday it might be replaced and
>> that replacement will have a different name and likely a different model.
>> Chapter 16 should not give a license to fiddle with the JSON grammar.
>
> OK, so we need not discuss any new numeric types any further in
> committee because it would be impossible to round-trip them through
> JSON.  Do we have agreement on that?

Not necessarily. What we can agree on is that new numeric types cannot impose
changes on the JSON syntax.
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Waldemar Horwat]

Douglas Crockford wrote:
> Waldemar Horwat wrote:
>> OK, so we need not discuss any new numeric types any further in
>> committee because it would be impossible to round-trip them through
>> JSON.  Do we have agreement on that?
>
> Not necessarily. What we can agree on is that new numeric types cannot
> impose changes on the JSON syntax.

Which means that it would be impossible to round-trip them through JSON.  Anything that uses the existing syntax already has a set interpretation as an existing ECMAScript object.

    Waldemar
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Breton Slivka]

On Thu, Jun 4, 2009 at 11:25 AM, Waldemar Horwat <waldemar@...> wrote:
> Which means that it would be impossible to round-trip them through JSON.
>  Anything that uses the existing syntax already has a set interpretation as
> an existing ECMAScript object.
>

Sorry to bungle into this conversation, but just out of curiosity,
does there exist a specification that requires a JSON number to be
interpreted as an ecmascript Number* ? Would it be violating any spec
if a future javascript implementation interpreted the numbers in a
JSON object as some other Number type? The way I see it, it is already
a problem that some JSON supporting languages cannot round trip some
of their native number types. Was type accurate round tripping one of
the original goals of JSON? Has it become a goal?




*as in the IEEE 64-bit floating point type that javascript uses by default.
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Christian Plesner Hansen-3]

(Sorry for the repeat, my first attempt bounced because I wasn't
subscribed to es-discuss)

> The V8 implementation is a pretty early implementation and I would
> consider all of the issues raised here to be bugs in it.

It is indeed an early implementation but the decision to use the
source parser directly is deliberate.  Because of the way json2.js
works anyone currently using it would get our implementation in the
future and I didn't want to change the behavior in subtle ways under
people's feet.

It's unclear to me from reading this thread if any other browsers
actually implement JSON correctly according to ES5.  If none do, how
sure are we that these changes won't break people's programs?

As for octal literals I agree that they're not a good idea.  However,
adding new special cases to avoid them is a cure worse than the
disease.  It adds to the complexity of the language in an area that is
already dangerously complex.
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Mark S. Miller-2]

On Thu, Jun 4, 2009 at 1:19 AM, Christian Plesner Hansen
<christian.plesner.hansen@...> wrote:
> It's unclear to me from reading this thread if any other browsers
> actually implement JSON correctly according to ES5.

More, it seems clear from reading this thread that none do.

> If none do, how
> sure are we that these changes won't break people's programs?

"Sure" is never an option ;). Our relative confidence comes from our
knowledge of the behavior of various popular json emitters. In
particular, we know of none that emit octal.


> As for octal literals I agree that they're not a good idea.  However,
> adding new special cases to avoid them is a cure worse than the
> disease.  It adds to the complexity of the language in an area that is
> already dangerously complex.

Since octal syntax will currently be interpreted as decimal on some
browsers and octal on others, even by json.js (since it delegates to
eval), there is no safe way to allow this syntax. By allowing them,
one's JSON.parse is no longer validating. It will not complain about
data that will be interpreted differently by different browsers.

Recall that we felt we needed to add JSON support to the spec because
JS library code cannot provide parsers that are simultaneously fast,
safe, and validating. If all you want is safe and fast, json_sans_eval
is a fine solution that works on existing browsers.

--
    Cheers,
    --MarkM
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Mark S. Miller-2]

On Thu, Jun 4, 2009 at 7:42 AM, Mark S. Miller <erights@...> wrote:
> Since octal syntax will currently be interpreted as decimal on some
> browsers and octal on others, even by json.js (since it delegates to
> eval),

Oops. Meant "json2.js".



--
    Cheers,
    --MarkM
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Allen Wirfs-Brock-2]

>-----Original Message-----
>From: es-discuss-bounces@... [mailto:es-discuss-
>bounces@...] On Behalf Of Christian Plesner Hansen
>
>(Sorry for the repeat, my first attempt bounced because I wasn't
>subscribed to es-discuss)

Please join...

>
>> The V8 implementation is a pretty early implementation and I would
>> consider all of the issues raised here to be bugs in it.
>
>It is indeed an early implementation but the decision to use the
>source parser directly is deliberate.  Because of the way json2.js
>works anyone currently using it would get our implementation in the
>future and I didn't want to change the behavior in subtle ways under
>people's feet.
>
>It's unclear to me from reading this thread if any other browsers
>actually implement JSON correctly according to ES5.  If none do, how
>sure are we that these changes won't break people's programs?

The IE8 implementation also started with json2.js as the model and evolved in response to ES5 developments prior to IE8 production release.  There are a few variances between the currently shipping IE8 JSON implementation and the current ES5 spec.  A couple of these are bugs in our implementation and others are the result of late churn in the E5 specification (some of which we contributed to based upon our IE8 experiences). Almost all the variances deal with
API edge cases and unusual situations (for example, a Number wrapper object returned from a stringify replacer function might be rendered as an object rather than a numeric literal). Almost all of the known issues have been found by our internal testing and analysis of the algorithms and haven't shown up in user problem reports.  Regardless, we intend to evolve our implementation to strictly conform to the final ES5 spec. as quickly as we can.

In releasing the IE8 JSON support most of the problem reports we received weren't related to differences between the IE8 implementation and json2.js.  We actually discovered that jason2.js was not as widely used as we thought.  Most of the issues we received involved encoding/decoding differences between our implementation (and jason2.js)  and other encoders that applications were using. We also encountered issues where people were using the jason2 API names but applying them to differing, home grown APIs encoders/decoders.

So far, I think the implementers that have been actively engaged with ES5 development within TC39 have been doing a pretty good job of collaborating to make sure that all our JSON implementations are highly compatible with each other and with the emerging standard.  Please feel free to join in.  

>As for octal literals I agree that they're not a good idea.  However,
>adding new special cases to avoid them is a cure worse than the
>disease.  It adds to the complexity of the language in an area that is
>already dangerously complex.

Octal literals have never been part of the JSON format and support for them by some JSON parsers is likely a side-effect of those parsers use of eval.  This sort of implementation accident is exactly the sort of thing we want to make sure doesn't get unnecessarily enshrined in either standards or future implementations. Not supporting JSON octal literals adds no complexity to the ES5 JSON spec. because they are simply not part of the format and are never mentioned. It only adds complexity to implementations if they are trying to reuse their JavaScript lexer (assuming it supports octal constants) to lex JSON text and there are already enough other differences between the JSON and ECMAScript token set that it isn't clear that this is a good idea.  Regardless, ES5 lexers are already required to not recognize octal number when operating in strict mode.

Allen
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Christian Plesner Hansen-3]

> "Sure" is never an option ;). Our relative confidence comes from our
> knowledge of the behavior of various popular json emitters. In
> particular, we know of none that emit octal.

It takes just one home-made emitter on a popular site somewhere for
this to break down.  On the other hand if there are already browsers
that never interpret numbers as octal and haven't experienced any
problems with it then that's pretty close to "sure".

> Since octal syntax will currently be interpreted as decimal on some
> browsers and octal on others, even by json.js (since it delegates to
> eval), there is no safe way to allow this syntax. By allowing them,
> one's JSON.parse is no longer validating. It will not complain about
> data that will be interpreted differently by different browsers.

That sounds reasonable, though I'm unsure if the value of validation
justifies making the language more complex.  If we're already in a
situation where nobody uses octal numbers in their json then
validation would seem unnecessary.

> Recall that we felt we needed to add JSON support to the spec because
> JS library code cannot provide parsers that are simultaneously fast,
> safe, and validating. If all you want is safe and fast, json_sans_eval
> is a fine solution that works on existing browsers.

I wasn't aware of the validation aspect.  Also, I don't know of any
truly safe and fast json implementations that work on existing
browsers (no offense to json2.js).
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Mark S. Miller-2]

On Thu, Jun 4, 2009 at 9:02 AM, Christian Plesner Hansen
<christian.plesner.hansen@...> wrote:
> I wasn't aware of the validation aspect.  Also, I don't know of any
> truly safe and fast json implementations that work on existing
> browsers (no offense to json2.js).

http://code.google.com/p/json-sans-eval/
We're happily using it in Caja.

We do not consider json2.js to be safe enough.

--
    Cheers,
    --MarkM
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Mark S. Miller-2]

On Thu, Jun 4, 2009 at 8:07 AM, Allen Wirfs-Brock
<Allen.Wirfs-Brock@...> wrote:

> [...] Not supporting JSON octal literals adds no complexity to
> the ES5 JSON spec. because they are simply not part of the
> format and are never mentioned. It only adds complexity to
> implementations if they are trying to reuse their JavaScript
> lexer (assuming it supports octal constants) to lex JSON text
> and there are already enough other differences between the JSON
> and ECMAScript token set that it isn't clear that this is a
> good idea.  Regardless, ES5 lexers are already required to not
> recognize octal number when operating in strict mode.

No, it does add some complexity to the spec differently from the sense
in which octal is prohibited in ES5/strict. In ES5/strict code, the
literal 010 must be interpreted at a decimal 10. In JSON.parse, 010
must be rejected. In order to mandate this, the standard JSON grammar
(as documented both in the RFC and on json.org) does not allow any
digits after a leading zero. I support this added complexity for all
the reasons discussed. But it is undeniably added complexity.

--
    Cheers,
    --MarkM
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Christian Plesner Hansen-3]

> [snip]
>
> In releasing the IE8 JSON support most of the problem reports we received weren't related to differences between the IE8 implementation and json2.js.  We actually discovered that jason2.js was not as widely used as we thought.  Most of the issues we received involved encoding/decoding differences between our implementation (and jason2.js)  and other encoders that applications were using. We also encountered issues where people were using the jason2 API names but applying them to differing, home grown APIs encoders/decoders.

Great, it sounds like there is no reason to expect any problems from
conforming to the spec.  I have filed the incompatibilities as a bug
against our implementation.

> Octal literals have never been part of the JSON format and support for them by some JSON parsers is likely a side-effect of those parsers use of eval.  This sort of implementation accident is exactly the sort of thing we want to make sure doesn't get unnecessarily enshrined in either standards or future implementations. Not supporting JSON octal literals adds no complexity to the ES5 JSON spec. because they are simply not part of the format and are never mentioned. It only adds complexity to implementations if they are trying to reuse their JavaScript lexer (assuming it supports octal constants) to lex JSON text and there are already enough other differences between the JSON and ECMAScript token set that it isn't clear that this is a good idea.  Regardless, ES5 lexers are already required to not recognize octal number when operating in strict mode.

What I mean when I say that it increases complexity is that having
different interpretations of something that otherwise appears to be
the same thing adds to the cognitive overhead of using the language.
With 'eval' and 'JSON.parse' you have two functions that behave
deceivingly similarly, and that take similar input.  Big obvious
differences between the two are easy to deal with but making subtle
differences between them invites confusion, in particular in this case
because there is already several different interpretations of number
literals in different contexts.

The original post and this current discussion illustrates my point:
subtle differences invite confusion.
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Allen Wirfs-Brock-2]

Conforming consensus...

I want to make sure that I understand the consensus of last week's discussion on this thread so I can update the spec. accordingly.  Below is the decision points that I sent out last week.  I've annotated it with what I believe was the consensus of the discussion.  Let me know if anybody disagrees that this is actually the consensus conclusions.

There are two new items at the end of the list that came up in the thread.  It's not clear to me whether there was consensus on the final item.

Are there any other points that need to be captured?

Allen

>-----Original Message-----
>From: es-discuss-bounces@... [mailto:es-discuss-
>bounces@...] On Behalf Of Allen Wirfs-Brock
>Sent: Wednesday, June 03, 2009 12:43 PM
>To: Rob Sayre; Oliver Hunt
>Cc: Mark S.Miller; es-discuss@...; Douglas Crockford; Robert
>Sayre
>Subject: RE: JSON parser grammar
>
>I want to bring this discussion around to focus on concrete points that
>we need to make decisions on.
>
>1) There is a bug in the ES5 candidate spec. in that it says that:
>JSONSourceCharacter ::
>    SourceCharacter but not U+0000 thru U+001F
>
>This is pretty clearly bogus as it means that tabs and new line
>characters cannot occur anywhere in JSON source text (not just string
>literals).  I'll probably fix it by simply equating JSONSourceCharter to
>SourceCharacter.
>

Do apparent disagreement.  This is just a simple bug fix.

>2) Do we want to permit conforming implementations to extend the JSON
>grammar that they recognize?  This probably could be done by extending
>the syntax error extension allowance in section 16 to include the JSON
>grammar.  If we allow this then most of the observed variation for the
>current emerging implementation that we have been talking about would
>probably be acceptable extensions.
>
>My inclination is to say we should disallow such open-ended extensions.
>As I suggest earlier, an implementation can always provide a non-
>standard extended parse function if it wants to support an extended
>grammar.

We will require strict conformance with no extensions from the ES5 specificaiton.

>3) If we disallow JSON grammar extensions (for JSON.parse) should we
>extend the existing grammar with some Postel's Law flexibility?
>
>I could accept this for cases where we have some evidence that there are
>actual JSON encoders in the wild that violate/extend the JSON grammar in
>the identified manner.
>
>Here are the individual cases that I know of to consider:
>
>a) Allow strings, numbers, Booleans, and null in addition to objects and
>arrays as top level JSON text.

Yes we keep this extension in the specification

>
>The ES5 spec. already has this although it isn't in the RFC.  I haven't
>heard any suggestions that we remove it.
>
>b) Permit leading zeros on numbers either with or without octal
>implications.
>
>I'm with Brendan on this, I don't think we should let octal constants
>into JSON. I don't have deep problem with leading zeroes for decimal
>constants but given the historic octal interpretation within JavaScript
>it is probably safer to syntax error than to simply ignore leading
>zeros.
>
>Does anyone know of any encoders or uses that actually insert leading
>0's?

No, no octal and no leading 0 interpretation as decimal

>c) Trailing commas in objects and arrays
>
>Are there encoders that do this or are we just anticipating that there
>might be manually generated files where this is convenient?
>
>I could go either way on this one but would prefer some supporting
>evidence

no

>
>d)  Holes in arrays, eg [1,,3]
>
>I don't think we should allow it unless we know there are encoders that
>generate it was acceptable to legacy eval based parsers.
>

no
 


>e) Allow some/all control characters to appear unescaped in JSON string
>literals. Which ones?
>
>Might be plausible.  Crock, why did you originally forbid them?  Are
>there known encoders that pass through such characters without escaping
>them?

no

>
>f) Allow single quotes within JSON text as string delimiters
>
>I'm not really suggesting we allow this, but I'm told that at least one
>major web site has done this.

No

>
>Any other possible Postelisms?  I have to say, that going through this
>list I don't find many of them very compelling.
>

Allow JSON.parse to recognize unquote property name.

No

In addition of code units in the range 0x0000-0x001f JSON.stringify inserts escape sequences into string literals for some or all of the following code units:
0x007f-0x009f, 0x00ad, 0x0600-0x0604,0x070f,0x17bf,0x17b5,0x200c-0x200f,0x2028-0x202f,0x2060-0x206f,0xfeff,0xfff0-0xffff

Was there any consensus that at least some of these code points should be escapeded?  If show which if not all?
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Mark Miller-2]

On Mon, Jun 8, 2009 at 11:19 AM, Allen Wirfs-Brock
<Allen.Wirfs-Brock@...> wrote:
> Conforming consensus...
>
> I want to make sure that I understand the consensus of last week's discussion on this thread so I can update the spec. accordingly.  Below is the decision points that I sent out last week.  I've annotated it with what I believe was the consensus of the discussion.  Let me know if anybody disagrees that this is actually the consensus conclusions.
> [...]

I agree with all the decisions above.


> In addition of code units in the range 0x0000-0x001f JSON.stringify inserts escape sequences into string literals for some or all of the following code units:
> 0x007f-0x009f, 0x00ad, 0x0600-0x0604,0x070f,0x17bf,0x17b5,0x200c-0x200f,0x2028-0x202f,0x2060-0x206f,0xfeff,0xfff0-0xffff
>
> Was there any consensus that at least some of these code points should be escapeded?  If show which if not all?

I think it is important that 0x2028 and 0x2029 be escaped. I have no
opinion about the others.


--
Text by me above is hereby placed in the public domain

    Cheers,
    --MarkM
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Douglas Crockford]

> In addition of code units in the range 0x0000-0x001f JSON.stringify inserts escape sequences into string literals for some or all of the following code units:
> 0x007f-0x009f, 0x00ad, 0x0600-0x0604,0x070f,0x17bf,0x17b5,0x200c-0x200f,0x2028-0x202f,0x2060-0x206f,0xfeff,0xfff0-0xffff

There is no harm in doing this, and it will improve interoperability with ES3.
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Hallvord R. M. Steen]

On Wed, 03 Jun 2009 21:42:46 +0200, Allen Wirfs-Brock <Allen.Wirfs-Brock@...> wrote:

> a) Allow strings, numbers, Booleans, and null in addition to objects and  
> arrays as top level JSON text.
> The ES5 spec. already has this although it isn't in the RFC.  I haven't  
> heard any suggestions that we remove it.


This may be a stupid question, but..

How can you allow "strings" as top level JSON text and keep the 15.12.2 step 2 requirement that JSON.parse() must throw if the input isn't valid JSON source text? In other words, how is an implementation supposed to know if I'm passing in a string of random content that should be "parsed" into a string or a malformed piece of JSON that should cause an exception?

--
Hallvord R. M. Steen, Core Tester, Opera Software
http://www.opera.com http://my.opera.com/hallvors/
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss