Should we move our test suites to ecmascript.org?

[Mark S. Miller-2]

Currently, there are two open source EcmaScript test suites and one open source JavaScript test suite.

* http://es5conform.codeplex.com/ for testing the differences between ES5 and ES3
* http://code.google.com/p/sputniktests/ for testing ES3
* http://www.mozilla.org/js/tests/library.html for testing Mozilla's JavaScript.

In committee, we all agreed that we like the structure of the first two, as they follow the structure of the spec. We also agreed that we'd like the results of developing and possibly merging these test suites to be hosted at ecmascript.org, and to eventually earn some kind of status as the official conformance test suites. The open question is where to do the development. Having the development be split among three, or even two, code development sites that some participants might regard as partisan, and that operate in somewhat different ways, is doable but not optimal.

Current resources at ecmascript.org that one might start from for joint development are http://hg.ecmascript.org/ and http://bugs.ecmascript.org/ . Are these adequate starting points? Are there other suitable resources hosted by Ecma? Could there be? Is this realistic? What about other alternatives like sourceforge or github? Or are we better off just sticking with the current split development that, frankly, is working rather well?

--
   Cheers,
   --MarkM

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Patrick Mueller]

There didn't seem to be any follow-up discussion on this, that I could see.  Looking at the (top-level) change logs for es5conform and sputniktests, you can see there has been some working going on since Mark's post.  

Any status on the individual suites, a combination of the suites, or re-hosting them?

On Sep 12, 2009, at 2:38 PM, Mark S. Miller wrote:

> Currently, there are two open source EcmaScript test suites and one open source JavaScript test suite.
>
> * http://es5conform.codeplex.com/ for testing the differences between ES5 and ES3
> * http://code.google.com/p/sputniktests/ for testing ES3
> * http://www.mozilla.org/js/tests/library.html for testing Mozilla's JavaScript.
>
> In committee, we all agreed that we like the structure of the first two, as they follow the structure of the spec. We also agreed that we'd like the results of developing and possibly merging these test suites to be hosted at ecmascript.org, and to eventually earn some kind of status as the official conformance test suites. The open question is where to do the development. Having the development be split among three, or even two, code development sites that some participants might regard as partisan, and that operate in somewhat different ways, is doable but not optimal.
>
> Current resources at ecmascript.org that one might start from for joint development are http://hg.ecmascript.org/ and http://bugs.ecmascript.org/ . Are these adequate starting points? Are there other suitable resources hosted by Ecma? Could there be? Is this realistic? What about other alternatives like sourceforge or github? Or are we better off just sticking with the current split development that, frankly, is working rather well?

Patrick Mueller - http://muellerware.org/




_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Christian Plesner Hansen-3]

Sorry, I missed Mark's original message.

Who maintains *.ecmascript.org?  If we can expect it to be reliable I
would have no problem moving development of the sputnik test suite
there.  Actually I would be prepared to move as soon as we can agree
on what structure we want.  One small issue is how to review code
changes.  Sputnik already uses a "third-party" tool (as in it's
developed by google but not an integrated part of code.google.com) and
for now we could continue to use that, just from a different base
repository.

As for merging sputnik with es5conform and the subset of the mozilla
tests that reflect the spec I see no reason why we couldn't do that,
other than the work it would take to merge the heterogeneous
frameworks.


-- Christian

On Wed, Dec 9, 2009 at 8:26 AM, Patrick Mueller <pmuellr@...> wrote:

> There didn't seem to be any follow-up discussion on this, that I could see.  Looking at the (top-level) change logs for es5conform and sputniktests, you can see there has been some working going on since Mark's post.
>
> Any status on the individual suites, a combination of the suites, or re-hosting them?
>
> On Sep 12, 2009, at 2:38 PM, Mark S. Miller wrote:
>
>> Currently, there are two open source EcmaScript test suites and one open source JavaScript test suite.
>>
>> * http://es5conform.codeplex.com/ for testing the differences between ES5 and ES3
>> * http://code.google.com/p/sputniktests/ for testing ES3
>> * http://www.mozilla.org/js/tests/library.html for testing Mozilla's JavaScript.
>>
>> In committee, we all agreed that we like the structure of the first two, as they follow the structure of the spec. We also agreed that we'd like the results of developing and possibly merging these test suites to be hosted at ecmascript.org, and to eventually earn some kind of status as the official conformance test suites. The open question is where to do the development. Having the development be split among three, or even two, code development sites that some participants might regard as partisan, and that operate in somewhat different ways, is doable but not optimal.
>>
>> Current resources at ecmascript.org that one might start from for joint development are http://hg.ecmascript.org/ and http://bugs.ecmascript.org/ . Are these adequate starting points? Are there other suitable resources hosted by Ecma? Could there be? Is this realistic? What about other alternatives like sourceforge or github? Or are we better off just sticking with the current split development that, frankly, is working rather well?
>
> Patrick Mueller - http://muellerware.org/
>
>
>
>
> _______________________________________________
> es-discuss mailing list
> es-discuss@...
> https://mail.mozilla.org/listinfo/es-discuss
>

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Brendan Eich-3]

On Dec 8, 2009, at 2:03 PM, Christian Plesner Hansen wrote:

> Sorry, I missed Mark's original message.
>
> Who maintains *.ecmascript.org?

I do with help from others (David-Sarah has helped fix up the trac,  
for example).


>  If we can expect it to be reliable I
> would have no problem moving development of the sputnik test suite
> there.  Actually I would be prepared to move as soon as we can agree
> on what structure we want.  One small issue is how to review code
> changes.  Sputnik already uses a "third-party" tool (as in it's
> developed by google but not an integrated part of code.google.com) and
> for now we could continue to use that, just from a different base
> repository.

Can you mail me about what is required to host Sputnik and this tool,  
in detail? Thanks.


> As for merging sputnik with es5conform and the subset of the mozilla
> tests that reflect the spec I see no reason why we couldn't do that,
> other than the work it would take to merge the heterogeneous
> frameworks.

This is a lot of work, so the best way to do it is according to a plan  
we all agree with, but almost ceratinly not with any "big bang"  
integration. Just lots of patches. The separate tests will have their  
own integrity and greater self-consistency for a while. If we get a  
single unified suite, we'll know when it is "ready".

/be

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Allen Wirfs-Brock-2]

I believe that there are still IPR policy issues that need to be worked through before any test suite development that is affiliated with ECMA/T39 could accept contributions from organizations or individuals who do not have an ECMA membership affiliation.  An advantage of the current google/codeplex/mozilla projects, is that they don't have this restriction. While ecmascript.org is not exactly officially associated with ECMA/TC-39 it is close enough that I don't think we should try to host a test suite project there until we resolve the IPR issues.

I don't particularly see why TC39 would want to go to the trouble of building and supporting a "foundry". It's enough trouble to do what is necessary to setup and manage individual project without having to be responsible for all the underlying infrastructure. I personally think that both the google code and codeplex hosting environments are fine and future work could be done using either foundry (or another that isn't tainted by a vender affiliation, if this continues to be a concern).

My sense is that both Google and Microsoft has taken the necessary steps to ensure that the projects we initiated can be very open. However, I also think that they both may suffer from a perception that they aren't. Note that I'm a "committer" on the sputnik project and Christian Plesner Hansen of Google and Rob Sayre of Mozilla are "coordinators" on the ES5conform codeplex project. However, I don't think any of us have exercised any of the associated privileges other than for our "own" projects.

I think the biggest issues that we need to resolve to move forward are about project planning, coordination, and process. What needs to be done.  How should it be done.  Who is going to commit to doing to the individual tasks?  I've informally suggested a couple time that maybe we need to have an open planning "summit" do work these things out.  I still think it would be a good idea.

Allen

-----Original Message-----
From: es-discuss-bounces@... [mailto:es-discuss-bounces@...] On Behalf Of Brendan Eich
Sent: Tuesday, December 08, 2009 2:10 PM
To: Christian Plesner Hansen
Cc: es-discuss
Subject: Re: Should we move our test suites to ecmascript.org?

On Dec 8, 2009, at 2:03 PM, Christian Plesner Hansen wrote:

> Sorry, I missed Mark's original message.
>
> Who maintains *.ecmascript.org?

I do with help from others (David-Sarah has helped fix up the trac,  
for example).


>  If we can expect it to be reliable I
> would have no problem moving development of the sputnik test suite
> there.  Actually I would be prepared to move as soon as we can agree
> on what structure we want.  One small issue is how to review code
> changes.  Sputnik already uses a "third-party" tool (as in it's
> developed by google but not an integrated part of code.google.com) and
> for now we could continue to use that, just from a different base
> repository.

Can you mail me about what is required to host Sputnik and this tool,  
in detail? Thanks.


> As for merging sputnik with es5conform and the subset of the mozilla
> tests that reflect the spec I see no reason why we couldn't do that,
> other than the work it would take to merge the heterogeneous
> frameworks.

This is a lot of work, so the best way to do it is according to a plan  
we all agree with, but almost ceratinly not with any "big bang"  
integration. Just lots of patches. The separate tests will have their  
own integrity and greater self-consistency for a while. If we get a  
single unified suite, we'll know when it is "ready".

/be

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Maciej Stachowiak]

On Dec 8, 2009, at 3:43 PM, Allen Wirfs-Brock wrote:

> I believe that there are still IPR policy issues that need to be  
> worked through before any test suite development that is affiliated  
> with ECMA/T39 could accept contributions from organizations or  
> individuals who do not have an ECMA membership affiliation.  An  
> advantage of the current google/codeplex/mozilla projects, is that  
> they don't have this restriction. While ecmascript.org is not  
> exactly officially associated with ECMA/TC-39 it is close enough  
> that I don't think we should try to host a test suite project there  
> until we resolve the IPR issues.

My main concern about these projects is not hosting or control, but  
the fact that there are several of them. I think it would be more  
useful to the community to have a single ECMAScript conformance test  
suite, covering all of ES5, including the bits inherited from ES3. If  
all parties are willing to use one of the existing sites for hosting,  
that is not a problem as far as I'm concerned.

Note: the WebKit project has a number of ECMAScript tests at <http://trac.webkit.org/browser/trunk/LayoutTests/fast/js 
 >, though we have not sorted over them to determine which are for  
ECMAScript proper rather than extensions, and which would be suitable  
as conforamance tests. We would consider donating tests to a shared  
ECMAScript conformance test suite if there were a single canonical  
test suite. Many of our tests cover edge cases that may not otherwise  
be covered by a conformance suite.

Regards,
Maciej

_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss

[Christian Plesner Hansen-3]

>> I believe that there are still IPR policy issues that need to be worked
>> through before any test suite development that is affiliated with ECMA/T39
>> could accept contributions from organizations or individuals who do not have
>> an ECMA membership affiliation.  An advantage of the current
>> google/codeplex/mozilla projects, is that they don't have this restriction.
>> While ecmascript.org is not exactly officially associated with ECMA/TC-39 it
>> is close enough that I don't think we should try to host a test suite
>> project there until we resolve the IPR issues.

I agree that for ECMA to give its stamp of approval, whatever form
that might take, causes extra complications.  But I don't know why
ecmascript.org should be so closely linked with ECMA that we can't use
it for this.  Isn't it up to us, Brendan really, to decide how to use
it?

> My main concern about these projects is not hosting or control, but the fact
> that there are several of them. I think it would be more useful to the
> community to have a single ECMAScript conformance test suite, covering all
> of ES5, including the bits inherited from ES3. If all parties are willing to
> use one of the existing sites for hosting, that is not a problem as far as
> I'm concerned.

I think it's important that this is and is perceived as, now and in
the long run, a vendor neutral project.  If we host it with vendor A,
for any choice of A including google, it's much too easy for people to
consider it A's project, and in particularly for other vendors to not
consider it theirs.

> Note: the WebKit project has a number of ECMAScript tests at
> <http://trac.webkit.org/browser/trunk/LayoutTests/fast/js>, though we have
> not sorted over them to determine which are for ECMAScript proper rather
> than extensions, and which would be suitable as conforamance tests. We would
> consider donating tests to a shared ECMAScript conformance test suite if
> there were a single canonical test suite. Many of our tests cover edge cases
> that may not otherwise be covered by a conformance suite.

I've been using the ES5 layout tests myself and they are indeed very
thorough.  Having them be part of the common test suite would be
excellent.


-- Christian
_______________________________________________
es-discuss mailing list
es-discuss@...
https://mail.mozilla.org/listinfo/es-discuss