more JSON spec questions

[Hallvord R. M. Steen]

Hi,
the feedback below refers to the "Initial ECMA formated ES5 draft" and  
consists of a few things I've noticed while working on a JSON test suite.

First, I think considerable work has gone into the JSON part of the spec  
since I first looked at it - it's beginning to look really good. My test  
suite is a work in progress, and of course both the tests and my questions  
might just show that there's something I haven't entirely understood  
rather than problems with the specification. Feedback is welcome.

Now, the questions..

1) Apparently the grammar still disallows literal tab inside JSONString.
Implementations that pass this test allow it:
http://testsuites.opera.com/JSON/correctness/035.html
- from earlier discussions I had the impression that the spec would  
change..?

2) The grammar and an existing test in the json.org test suite disallows  
final commas in array input (JSON.parse('[1,]') should apparently throw) -  
IE8, Safari 4 and Firefox 3.5 all seem to happily accept it though, as  
does the yet-to-go-public Opera implementation. I'd like some reassurance  
 from other browser vendors that you consider this a bug and intend to fix  
it before I push for a fix here because this is the kind of thing that  
might cause compat problems :-o (Guess this is more a question for the  
implementers on the list than for the spec authors.)

3) Should JSON.parse() without arguments throw? It's not obvious from the  
spec as far as I can tell, but it's what most browser implementations do.

4) (Editorial) - spec says "The abstract operation Str(key, holder) has  
access  to
PropertyList and ReplacerFunction" - the algorithm does not actually use
PropertyList (very trivial issue, of course)

5) (Editorial) It's not apparent from the prose description of  
JSON.stringify() that if the replacer argument is an array, it will only  
be applied to objects. I expected to be able to do something like  
JSON.stringify( [0,1,2], [1] ) => '[1]'. Suggest a small clarification,  
perhaps just saying "selecting object keys that will be stringified"  
instead of "selecting the keys that will be stringified"

6) (Editorial) Per the algorithm, passing in a "replacer" array to  
JSON.stringify() will cause it to also stringify dontEnum properties. An  
example would be JSON.stringify( Number, ['MAX_VALUE'] ) including the  
MAX_VALUE property where JSON.stringify(Number) would not. Perhaps clarify  
this in the prose too, not just in the algorithm? Calling it a "whitelist"  
seems it will only narrow down the number of elements that are included,  
not expand it..

7) I also didn't understand why the "reviver" function was called once  
with "key" being the empty string and "value" being the entire output  
object until I read the algorithm carefully.

Quick word about the test suite: it's current home is  
http://testsuites.opera.com/JSON/ where you can have a look at the README,  
load runner.htm to run all tests (Warning: if you do that in IE8 something  
makes it eat all your memory and make the PC entirely unusable!) or browse  
through folders to individual tests. Hope it's useful to all you  
implementers out there. Feedback welcome :)

--
Hallvord R. M. Steen, Core Tester, Opera Software
http://www.opera.com http://my.opera.com/hallvors/
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Oliver Hunt-2]

2) The grammar and an existing test in the json.org test suite disallows final commas in array input (JSON.parse('[1,]') should apparently throw) - IE8, Safari 4 and Firefox 3.5 all seem to happily accept it though, as does the yet-to-go-public Opera implementation. I'd like some reassurance from other browser vendors that you consider this a bug and intend to fix it before I push for a fix here because this is the kind of thing that might cause compat problems :-o (Guess this is more a question for the implementers on the list than for the spec authors.)

This is definitely a bug in the WebKit implementation (we disallow the trailing comma in object literals, i'm saddened that we allow the trailing comma in [] :-/ )

4) (Editorial) - spec says "The abstract operation Str(key, holder) has access  to
PropertyList and ReplacerFunction" - the algorithm does not actually use
PropertyList (very trivial issue, of course)

PropertyList is used in the abstract function JO

Quick word about the test suite: it's current home is http://testsuites.opera.com/JSON/ where you can have a look at the README, load runner.htm to run all tests (Warning: if you do that in IE8 something makes it eat all your memory and make the PC entirely unusable!) or browse through folders to individual tests. Hope it's useful to all you implementers out there. Feedback welcome :)

There are also reasonably extensive tests in the WebKit repository -- unfortunately they are designed primarily as regression tests so their output isn't astonishingly nice -- Browsers without a native JSON implementation just fail everything somewhat catastrophically with no obvious indication as to why.  That said the tests do cover most edge cases (although not the [] with terminating comma thing apparently :-( ) such as order of execution, number of times properties are accessed, etc.

http://trac.webkit.org/export/HEAD/trunk/LayoutTests/fast/js/JSON-stringify-replacer.html
http://trac.webkit.org/export/HEAD/trunk/LayoutTests/fast/js/JSON-parse.html

http://trac.webkit.org/export/HEAD/trunk/LayoutTests/fast/js/JSON-stringify.html (this one takes quite a while)

--Oliver

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Lucas Smith-2]

I shared this on the JSON Yahoo! group list, but this seems relevant here, now.  I extracted tests from the YUI JSON test suite for Native JSON support.

http://github.com/lsmith/JSON-test-suite/tree/master

Great to see the sharing of tests!

Luke

On Aug 26, 2009, at 3:37 PM, Oliver Hunt wrote:

2) The grammar and an existing test in the json.org test suite disallows final commas in array input (JSON.parse('[1,]') should apparently throw) - IE8, Safari 4 and Firefox 3.5 all seem to happily accept it though, as does the yet-to-go-public Opera implementation. I'd like some reassurance from other browser vendors that you consider this a bug and intend to fix it before I push for a fix here because this is the kind of thing that might cause compat problems :-o (Guess this is more a question for the implementers on the list than for the spec authors.)

This is definitely a bug in the WebKit implementation (we disallow the trailing comma in object literals, i'm saddened that we allow the trailing comma in [] :-/ )

4) (Editorial) - spec says "The abstract operation Str(key, holder) has access  to
PropertyList and ReplacerFunction" - the algorithm does not actually use
PropertyList (very trivial issue, of course)

PropertyList is used in the abstract function JO

Quick word about the test suite: it's current home is http://testsuites.opera.com/JSON/ where you can have a look at the README, load runner.htm to run all tests (Warning: if you do that in IE8 something makes it eat all your memory and make the PC entirely unusable!) or browse through folders to individual tests. Hope it's useful to all you implementers out there. Feedback welcome :)

There are also reasonably extensive tests in the WebKit repository -- unfortunately they are designed primarily as regression tests so their output isn't astonishingly nice -- Browsers without a native JSON implementation just fail everything somewhat catastrophically with no obvious indication as to why.  That said the tests do cover most edge cases (although not the [] with terminating comma thing apparently :-( ) such as order of execution, number of times properties are accessed, etc.

http://trac.webkit.org/export/HEAD/trunk/LayoutTests/fast/js/JSON-stringify-replacer.html
http://trac.webkit.org/export/HEAD/trunk/LayoutTests/fast/js/JSON-parse.html

http://trac.webkit.org/export/HEAD/trunk/LayoutTests/fast/js/JSON-stringify.html (this one takes quite a while)

--Oliver

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Oliver Hunt-2]

> Quick word about the test suite: it's current home is http://testsuites.opera.com/JSON/ 
>  where you can have a look at the README, load runner.htm to run all  
> tests (Warning: if you do that in IE8 something makes it eat all  
> your memory and make the PC entirely unusable!) or browse through  
> folders to individual tests. Hope it's useful to all you  
> implementers out there. Feedback welcome :)

The spec only defines Date.prototype.toJSON now -- so the tests for  
toJSON on String, Boolean and Number are wrong (technically they  
should not be present, and stringification should work if they aren't).

Octal values are completely specified -- the lexer defined in ES5 and json.org
  disallows a leading 0 on any value not between -1 and 1.

You should also ensure comments are not allowed in the content.

--Oliver

>
> --
> Hallvord R. M. Steen, Core Tester, Opera Software
> http://www.opera.com http://my.opera.com/hallvors/
> _______________________________________________
> es-discuss mailing list
> es-discuss@...
> https://mail.mozilla.org/listinfo/es-discuss

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Hallvord R. M. Steen]

On Thu, 27 Aug 2009 00:52:20 +0200, Oliver Hunt <oliver@...> wrote:

>> Quick word about the test suite: it's current home is  
>> http://testsuites.opera.com/JSON/ where you can have a look at the  
>> README, load runner.htm to run all tests (Warning: if you do that in  
>> IE8 something makes it eat all your memory and make the PC entirely  
>> unusable!) or browse through folders to individual tests. Hope it's  
>> useful to all you implementers out there. Feedback welcome :)
>
> The spec only defines Date.prototype.toJSON now -- so the tests for  
> toJSON on String, Boolean and Number are wrong (technically they should  
> not be present, and stringification should work if they aren't).

Aha, I was wondering why Safari failed all those tests - completely missed  
that these were removed from the spec :-p
Thanks!

> Octal values are completely specified -- the lexer defined in ES5 and  
> json.org  disallows a leading 0 on any value not between -1 and 1.

Right - I think the test itself is valid, so I just need to fix the  
comment saying it's an unspecified assertation, right?
(test is http://testsuites.opera.com/JSON/correctness/scripts/003.js )

> You should also ensure comments are not allowed in the content.

--
Hallvord R. M. Steen, Core Tester, Opera Software
http://www.opera.com http://my.opera.com/hallvors/
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Oliver Hunt-2]

Octal values are completely specified -- the lexer defined in ES5 and json.org  disallows a leading 0 on any value not between -1 and 1.

Right - I think the test itself is valid, so I just need to fix the comment saying it's an unspecified assertation, right?
(test is http://testsuites.opera.com/JSON/correctness/scripts/003.js )

Yup. You may also want tests for use of replacer arrays and functions, etc

--Oliver

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Hallvord R. M. Steen]

On Thu, 27 Aug 2009 01:22:46 +0200, Oliver Hunt <oliver@...> wrote:

>>> Octal values are completely specified -- the lexer defined in ES5
>>> and json.org  disallows a leading 0 on any value not between -1 and
>>> 1.

Good to hear but - sorry! - I can't quite see where the spec supports this  
conclusion. I tried looking at the lexical and syntactical grammars and  
the JSON.parse() algorithm.

Just reported the bug on us parsing '[1,]' and reversed tests for toJSON()  
on String, Boolean and Number (Safari surges to 90.5% pass - good for you  
and well done ;) )

--
Hallvord R. M. Steen, Core Tester, Opera Software
http://www.opera.com http://my.opera.com/hallvors/
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Oliver Hunt-2]

On Aug 26, 2009, at 4:50 PM, Hallvord R. M. Steen wrote:

On Thu, 27 Aug 2009 01:22:46 +0200, Oliver Hunt <oliver@...> wrote:

Octal values are completely specified -- the lexer defined in ES5

and json.org  disallows a leading 0 on any value not between -1 and

1.

Good to hear but - sorry! - I can't quite see where the spec supports this conclusion. I tried looking at the lexical and syntactical grammars and the JSON.parse() algorithm.

From sections 7.8.3 and 15.12.1.1

JSONNumber ::

 -<opt> DecimalIntegerLiteral JSONFraction <opt> ExponentPart <opt>

DecimalIntegerLiteral ::

0

NonZeroDigit DecimalDigits <opt>

eg. the only valid character left of the decimal point the begins with 0 is 0

--Oliver

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Mark S. Miller-2]

On Wed, Aug 26, 2009 at 5:02 PM, Oliver Hunt<oliver@...> wrote:
>
> From sections 7.8.3 and 15.12.1.1
> JSONNumber ::
>  -<opt> DecimalIntegerLiteral JSONFraction <opt> ExponentPart <opt>
> DecimalIntegerLiteral ::
> 0
> NonZeroDigit DecimalDigits <opt>
> eg. the only valid character left of the decimal point the begins with 0 is
> 0

As also shown by the railroad diagram: http://json.org/number.gif


--
    Cheers,
    --MarkM
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Oliver Hunt-2]

On Aug 26, 2009, at 4:22 PM, Oliver Hunt wrote:

Yup. You may also want tests for use of replacer arrays and functions, etc

On this topic, i'm fixing an issue in our handling of reviver functions in JSON.parse and noticed that json2.js and the JSON.parse specification differ in the handling of properties being deleted from the holder object, namely if a reviver function deletes a property from holder before it is visited then the current spec behaviour would imply that that property is still visited, only with the value undefined.

While json2.js disagrees with this implementation the firefox implementation does match spec here, so i'd just like to verify that the currently specified behaviour is intended (this is sufficiently edge-case like for me to doubt it has any significant compatibility impact, i just want clarification)

--Oliver

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Allen Wirfs-Brock-2]

Some parts of this message have been removed. Learn more about Nabble's security policy.

At this point it’s hard to say whether or not the exact handling of this specific edge case was intentional or not.  However, it isn’t obvious that this behavior is any worse or better than the possible alternatives so I don’t think we have a strong case for changing it.  Particularly, since it has been specified this way for in drafts for quite a while.

 

We’re about a year past json2.js serving as the reference implementation for the ES5 JSON spec. (although I believe Crock has done some json2.js updates to track the ES5 spec.)  There are some things in the current algorithms that just can’t be easily accomplished in pure ES3 code. 

 

Allen

 

From: es-discuss-bounces@... [mailto:es-discuss-bounces@...] On Behalf Of Oliver Hunt
Sent: Wednesday, August 26, 2009 8:23 PM
To: Oliver Hunt
Cc: es-discuss@...
Subject: Re: more JSON spec questions

 

 

On Aug 26, 2009, at 4:22 PM, Oliver Hunt wrote:

Yup. You may also want tests for use of replacer arrays and functions, etc

 

On this topic, i'm fixing an issue in our handling of reviver functions in JSON.parse and noticed that json2.js and the JSON.parse specification differ in the handling of properties being deleted from the holder object, namely if a reviver function deletes a property from holder before it is visited then the current spec behaviour would imply that that property is still visited, only with the value undefined.

 

While json2.js disagrees with this implementation the firefox implementation does match spec here, so i'd just like to verify that the currently specified behaviour is intended (this is sufficiently edge-case like for me to doubt it has any significant compatibility impact, i just want clarification)

 

--Oliver

 

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Robert Sayre-2]

On Wed, Aug 26, 2009 at 11:03 PM, Allen
Wirfs-Brock<Allen.Wirfs-Brock@...> wrote:
> At this point it’s hard to say whether or not the exact handling of this
> specific edge case was intentional or not.

I think it is obvious. The list of keys is built before iteration
takes place for a couple of reasons.

- Rob
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Oliver Hunt-2]

Rob, the specific issue is whether you call the reviver for properties  
that have been deleted during filtering -- eg. the behaviour of for(in)

--Oliver

On Aug 26, 2009, at 11:48 PM, Robert Sayre wrote:

> On Wed, Aug 26, 2009 at 11:03 PM, Allen
> Wirfs-Brock<Allen.Wirfs-Brock@...> wrote:
>> At this point it’s hard to say whether or not the exact handling of  
>> this
>> specific edge case was intentional or not.
>
> I think it is obvious. The list of keys is built before iteration
> takes place for a couple of reasons.
>
> - Rob

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Allen Wirfs-Brock-2]

>-----Original Message-----
>From: Robert Sayre [mailto:sayrer@...]
>
>I think it is obvious. The list of keys is built before iteration
>takes place for a couple of reasons.
>

Yes, but there could be subsequent explicit tests to see whether the property still exists.  However, I really don't think it would make any much differences as I don't think there are any reasonable use cases for a reviver that would be sensitive to this.

Allen
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Allen Wirfs-Brock-2]

>-----Original Message-----
>From: es-discuss-bounces@... [mailto:es-discuss-
>bounces@...] On Behalf Of Hallvord R. M. Steen
...
>1) Apparently the grammar still disallows literal tab inside JSONString.
>Implementations that pass this test allow it:
>http://testsuites.opera.com/JSON/correctness/035.html
>- from earlier discussions I had the impression that the spec would
>change..?
>

The JSON grammar and the RFC has "always" disallowed all control characters (including tabs) from appearing unescaped inside JSONStrings.  This has come up for discussion several times but I believe that we have always gone back to the RFC to guide us.  Can you point to what made you think this was going to change.

>2) The grammar and an existing test in the json.org test suite disallows
>final commas in array input (JSON.parse('[1,]') should apparently throw)
>-
>IE8, Safari 4 and Firefox 3.5 all seem to happily accept it though, as
>does the yet-to-go-public Opera implementation. I'd like some
>reassurance
> from other browser vendors that you consider this a bug and intend to
>fix
>it before I push for a fix here because this is the kind of thing that
>might cause compat problems :-o (Guess this is more a question for the
>implementers on the list than for the spec authors.)

The spec. means what it says in this regard and additionally forbids conforming implementations from extending specified JSON grammar for use in JSON.parse (you can, of course, provide a different implementation specific parsing function that uses an extended grammar).

I think I already stated on another thread that Microsoft plans on ultimately bringing our JSON implementation into full conformance with the spec.

>
>3) Should JSON.parse() without arguments throw? It's not obvious from
>the
>spec as far as I can tell, but it's what most browser implementations
>do.

Parse Step 1: "Let JText be ToString(text)".
If parse had no arguments, text will treated as the value undefined and do ToString(undefined) so JText is the string value 'undefined'
step 2. "throw a SyntaxError exception if the JText did not conform to the JSON grammar for the goal symbol JSONText."
The identifier undefined does not reduce to JSONText according to the grammar so a SyntaxError should be thrown.

>
>4) (Editorial) - spec says "The abstract operation Str(key, holder) has
>access  to
>PropertyList and ReplacerFunction" - the algorithm does not actually use
>PropertyList (very trivial issue, of course)
>

Thanks

>5) (Editorial) It's not apparent from the prose description of
>JSON.stringify() that if the replacer argument is an array, it will only
>be applied to objects. I expected to be able to do something like
>JSON.stringify( [0,1,2], [1] ) => '[1]'. Suggest a small clarification,
>perhaps just saying "selecting object keys that will be stringified"
>instead of "selecting the keys that will be stringified"

I replaced "keys" with "object properties" (even though arrays are also objects).  What's a "key" anyway.

In general, the algorithms are normative, not the prose descriptions.

>
>6) (Editorial) Per the algorithm, passing in a "replacer" array to
>JSON.stringify() will cause it to also stringify dontEnum properties. An
>example would be JSON.stringify( Number, ['MAX_VALUE'] ) including the
>MAX_VALUE property where JSON.stringify(Number) would not. Perhaps
>clarify
>this in the prose too, not just in the algorithm? Calling it a
>"whitelist"
>seems it will only narrow down the number of elements that are included,
>not expand it..

An interesting point that I don't recall having been mentioned before.  These isn't any prose that says that only enumerable properties get stringified and steps 5 and 6 of JO are pretty clear about which properties are examined in both the list and no-list cases. Given that (and the late date) I'm disinclined to add a note.
...

Allen
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Hallvord R. M. Steen]

On Thu, 27 Aug 2009 05:23:04 +0200, Oliver Hunt <oliver@...> wrote:

>> Yup. You may also want tests for use of replacer arrays and
>> functions, etc

You meant octal/noctal syntax in replacer arrays and functions? Wasn't  
quite sure what you were suggesting.

> On this topic, i'm fixing an issue in our handling of reviver
> functions in JSON.parse and noticed that json2.js and the JSON.parse
> specification differ in the handling of properties being deleted from
> the holder object, namely if a reviver function deletes a property
> from holder before it is visited then the current spec behaviour would
> imply that that property is still visited, only with the value
> undefined.

Looks like that's the way the spec works, and I've added another edge case  
test for it here:
http://testsuites.opera.com/JSON/correctness/050.html

--
Hallvord R. M. Steen, Core Tester, Opera Software
http://www.opera.com http://my.opera.com/hallvors/
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss