VOTE: Approach for sharing code

[David Van Couvering]

Thanks for writing up a proposed solution, I was getting a little tired
of putting up targets for others to shoot down.

I agree with this wholeheartedly this was what I have wanted to do
(although perhaps not as well communicated and there are some details
here that are new and which seem really good/interesting).

You say Java does this -- is this documented somewhere? I'd love to have
that as a reference.

First of all, Kathey was concerned we wouldn't be able to follow this
kind of discipline.

http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c43207ABC.8080000@...%3e

But I'll also show where I saw this get tripped up:

 > * a consumer expecting X.Y will work with X.Y' when Y' < Y by version
 >   detection and degraded functionality (Y' level). If Y level function
 >   is required then it will gracefully die (able to detect rather
 >   than AbstractMethodError)


In my original proposal:

    * compatibility will be strongly encouraged but not guaranteed
      against previous minor versions (e.g. a 10.2 consumer works
      with 10.1 common classes, but a 10.3 consumer has a hard
      dependency on new methods, it can not work with 10.2
      common classes).

Perhaps I remember incorrectly, but I remember us (or enough of us)
generally agreeing that gracefully dying when Y level function was
required was not acceptable, as this was a regression of existing
behavior.  This was the "nail in the coffin" for my original proposal.

http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c43289F58.6000707@...%3e
http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c4328AC1F.6000504@...%3e

Unless you can address these issues I think we're back at square one...

David

Further asides below:

> As a quick braindump:
> * there are rules that restrict changes to the interface going forward
> * Sun (et al) stick to those rules even when it is painful
> * no class/method is removed without being deprecated first
> * the JVM supports version skew a little by not requiring an
>   implementation to implement all methods in a interface (at runtime)

That's nice to know, I didn't know that

> * the interfaces often provide a mechanism to determine what features of
>   an API the implementation implements
> * frameworks can utilize multiple classloaders to load multiple versions
>   of a class into the VM - tools have developed to simplify this
>

I don't know much about such frameworks and tools, but I don't think
this solves Dan's problem of "accidentally" loading multiple versions.

> I think we can support a scenario where:
> * a comsumer expecting version X.Y will work with any implementation
>   X.Y' when Y' >= Y

> * X.Y defines the interface version, i.e. X.Y.Z works with X.Y.Z' for
>   all Z'
> * X defines the compatibility level i.e. X and X' are not guaranteed
>   to work together (although we will try to ensure they do)
>
> Basic rules:
> * at the Z level, no API change is allowed, just implementation changes
> * at the Y level, additive API changes are allowed but must be
>   accompanied by support in the versioning mechanism so that they can be
>   detected. Things can be deprecated
> * at the Z level, incompatible changes are allowed. Items deprecated in
>   version X can be removed in version X+2 (implicitly (X+2).0.0).
>
> This all applies to operation in a single classloader. Where multiple
> classloaders are involved (inside one VM or in multiple VMs) a different
> set of versioning behaviours applies to the wire protocol.
>
> --
> Jeremy

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Jeremy Boynes]

David W. Van Couvering wrote:
>
> You say Java does this -- is this documented somewhere? I'd love to have
> that as a reference.
>

Do you mean the JVM impl or the general philosophy? I think the
versioning model is defined in the language spec but would have to go
digging.

> First of all, Kathey was concerned we wouldn't be able to follow this
> kind of discipline.
>
> http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c43207ABC.8080000@...%3e 
>

It is on us and our users to police ourselves.

>
> But I'll also show where I saw this get tripped up:
>
>  > * a consumer expecting X.Y will work with X.Y' when Y' < Y by version
>  >   detection and degraded functionality (Y' level). If Y level function
>  >   is required then it will gracefully die (able to detect rather
>  >   than AbstractMethodError)
>
>
> In my original proposal:
>
>    * compatibility will be strongly encouraged but not guaranteed
>      against previous minor versions (e.g. a 10.2 consumer works
>      with 10.1 common classes, but a 10.3 consumer has a hard
>      dependency on new methods, it can not work with 10.2
>      common classes).
>
> Perhaps I remember incorrectly, but I remember us (or enough of us)
> generally agreeing that gracefully dying when Y level function was
> required was not acceptable, as this was a regression of existing
> behavior.  This was the "nail in the coffin" for my original proposal.
>
> http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c43289F58.6000707@...%3e 
>
> http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c4328AC1F.6000504@...%3e 
>
>
> Unless you can address these issues I think we're back at square one...
>

I'd put this as part of the discipline. The Y consumer, as the newer
version, knows about earlier Y' versions and hence can be coded to
accept the downrev dependency - it can downgrade itself, it can offer
different behaviour, it can implement the functionality itself, it can
die gracefully, or we can just make it X+1.

Bear in mind there are no guarantees. For example, someone else could
have modified Y' themselves without any regard to compatibility and just
stuck it on the classpath.

Also, let's not forget that this is only in the case where we have
multiple applications with poor classpath discipline installed
concurrently. (And I haven't forgotten the request for a writeup on
stuff like ClassWorlds).

--
Jeremy

[M. Mortazavi]

Daniel John Debrunner wrote:

>Jeremy Boynes wrote:
>
>  
>
>>Think of the carnage if it was java_14208_b13.sql_300.Connection
>>    
>>
>
>It's actually instructive to look how Java solves this:
>
>- The interface is kept upwards compatible, by following rules such as
>only new methods/fields etc.
>
>Then to look at how a consumer, Derby, deals with the fact of multiple
>versions of the interface:
>
>- Derby only provides and uses the functionality to match the version of
>the interface (java.sql.Connection) loaded, determined at runtime.
>
>I'd thought this was a workable direction being proposed about six
>thousand message ago, upwards compatible apis and the ability for a
>consumer to handle a lower version. Not sure what derailed it.
>
>Dan.
>  
>

I'm not sure if this will help or hinder with your considerations but . . .

Compatible type evolution is described in the Java Serialization Spec.
The reason it is described there is because the problem arises when
Class A is compiled against a particular version of Class B but then at
run-time an instance of A encounters (through class loading upon
de-serialization, for example) an instance of some other version of
Class B.

These type compatibility rules can be found here:
http://java.sun.com/j2se/1.4.2/docs/guide/serialization/spec/version.html#wp9419

M.

--
M. Mortazavi
[phone] (office)
[phone] (mobile)

[Daniel John Debrunner]

David W. Van Couvering wrote:

> In my original proposal:
>
>    * compatibility will be strongly encouraged but not guaranteed
>      against previous minor versions (e.g. a 10.2 consumer works
>      with 10.1 common classes, but a 10.3 consumer has a hard
>      dependency on new methods, it can not work with 10.2
>      common classes).
>
> Perhaps I remember incorrectly, but I remember us (or enough of us)
> generally agreeing that gracefully dying when Y level function was
> required was not acceptable, as this was a regression of existing
> behavior.  This was the "nail in the coffin" for my original proposal.

Maybe I'm an optimist, but I think that a consumer of common code can
always be coded to keep running (in a reduced mode) when faced with an
older version of the common code. Thus I think this approach can be made
to work, just start with the mindset that dying is unacceptable, rather
than inevitable.

Dan.

[David Van Couvering]

Well, that works for me!

David

Daniel John Debrunner wrote:

> David W. Van Couvering wrote:
>
>
>
>>In my original proposal:
>>
>>   * compatibility will be strongly encouraged but not guaranteed
>>     against previous minor versions (e.g. a 10.2 consumer works
>>     with 10.1 common classes, but a 10.3 consumer has a hard
>>     dependency on new methods, it can not work with 10.2
>>     common classes).
>>
>>Perhaps I remember incorrectly, but I remember us (or enough of us)
>>generally agreeing that gracefully dying when Y level function was
>>required was not acceptable, as this was a regression of existing
>>behavior.  This was the "nail in the coffin" for my original proposal.
>
>
> Maybe I'm an optimist, but I think that a consumer of common code can
> always be coded to keep running (in a reduced mode) when faced with an
> older version of the common code. Thus I think this approach can be made
> to work, just start with the mindset that dying is unacceptable, rather
> than inevitable.
>
> Dan.
>

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[David Van Couvering]

So, I'll wait for another overnight (over-day) period, but it sounds
like we may have a level of consensus here (fingers crossed).

What I propose to do is take Jeremy's quick brain dump (thanks again for
doing this Jeremy) and turn it into a more complete policy around this.
  Hopefully I'll capture the essence of this correctly, but I'm sure
you'll correct me if I miss something :)

I'll put this on the Derby Wiki, to make it more readable/reviewable.

Thanks all,

David

Jeremy Boynes wrote:

> Daniel John Debrunner wrote:
>
>> Jeremy Boynes wrote:
>>
>>
>>> Think of the carnage if it was java_14208_b13.sql_300.Connection
>>
>>
>>
>> It's actually instructive to look how Java solves this:
>>
>> - The interface is kept upwards compatible, by following rules such as
>> only new methods/fields etc.
>>
>> Then to look at how a consumer, Derby, deals with the fact of multiple
>> versions of the interface:
>>
>> - Derby only provides and uses the functionality to match the version of
>> the interface (java.sql.Connection) loaded, determined at runtime.
>>
>> I'd thought this was a workable direction being proposed about six
>> thousand message ago, upwards compatible apis and the ability for a
>> consumer to handle a lower version. Not sure what derailed it.
>>
>
> Kathey asked if I would be willing to write something up and I started
> thinking about it but haven't had the bandwidth to write anything.
>
> As a quick braindump:
> * there are rules that restrict changes to the interface going forward
> * Sun (et al) stick to those rules even when it is painful
> * no class/method is removed without being deprecated first
> * the JVM supports version skew a little by not requiring an
>   implementation to implement all methods in a interface (at runtime)
> * the interfaces often provide a mechanism to determine what features of
>   an API the implementation implements
> * frameworks can utilize multiple classloaders to load multiple versions
>   of a class into the VM - tools have developed to simplify this
>
> I think we can support a scenario where:
> * a comsumer expecting version X.Y will work with any implementation
>   X.Y' when Y' >= Y
> * a consumer expecting X.Y will work with X.Y' when Y' < Y by version
>   detection and degraded functionality (Y' level). If Y level function
>   is required then it will gracefully die (able to detect rather
>   than AbstractMethodError)
> * X.Y defines the interface version, i.e. X.Y.Z works with X.Y.Z' for
>   all Z'
> * X defines the compatibility level i.e. X and X' are not guaranteed
>   to work together (although we will try to ensure they do)
>
> Basic rules:
> * at the Z level, no API change is allowed, just implementation changes
> * at the Y level, additive API changes are allowed but must be
>   accompanied by support in the versioning mechanism so that they can be
>   detected. Things can be deprecated
> * at the Z level, incompatible changes are allowed. Items deprecated in
>   version X can be removed in version X+2 (implicitly (X+2).0.0).
>
> This all applies to operation in a single classloader. Where multiple
> classloaders are involved (inside one VM or in multiple VMs) a different
> set of versioning behaviours applies to the wire protocol.
>
> --
> Jeremy

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[David Van Couvering]

Hi, all.  I haven't gotten any more comments on the proposal at
http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm going
to update this page based on the comments I have received so far and
then we'll try for another vote.

Thanks,

David

David W. Van Couvering wrote:

> So, I'll wait for another overnight (over-day) period, but it sounds
> like we may have a level of consensus here (fingers crossed).
>
> What I propose to do is take Jeremy's quick brain dump (thanks again for
> doing this Jeremy) and turn it into a more complete policy around this.
>  Hopefully I'll capture the essence of this correctly, but I'm sure
> you'll correct me if I miss something :)
>
> I'll put this on the Derby Wiki, to make it more readable/reviewable.
>
> Thanks all,
>
> David
>
> Jeremy Boynes wrote:
>
>> Daniel John Debrunner wrote:
>>
>>> Jeremy Boynes wrote:
>>>
>>>
>>>> Think of the carnage if it was java_14208_b13.sql_300.Connection
>>>
>>>
>>>
>>>
>>> It's actually instructive to look how Java solves this:
>>>
>>> - The interface is kept upwards compatible, by following rules such as
>>> only new methods/fields etc.
>>>
>>> Then to look at how a consumer, Derby, deals with the fact of multiple
>>> versions of the interface:
>>>
>>> - Derby only provides and uses the functionality to match the version of
>>> the interface (java.sql.Connection) loaded, determined at runtime.
>>>
>>> I'd thought this was a workable direction being proposed about six
>>> thousand message ago, upwards compatible apis and the ability for a
>>> consumer to handle a lower version. Not sure what derailed it.
>>>
>>
>> Kathey asked if I would be willing to write something up and I started
>> thinking about it but haven't had the bandwidth to write anything.
>>
>> As a quick braindump:
>> * there are rules that restrict changes to the interface going forward
>> * Sun (et al) stick to those rules even when it is painful
>> * no class/method is removed without being deprecated first
>> * the JVM supports version skew a little by not requiring an
>>   implementation to implement all methods in a interface (at runtime)
>> * the interfaces often provide a mechanism to determine what features of
>>   an API the implementation implements
>> * frameworks can utilize multiple classloaders to load multiple versions
>>   of a class into the VM - tools have developed to simplify this
>>
>> I think we can support a scenario where:
>> * a comsumer expecting version X.Y will work with any implementation
>>   X.Y' when Y' >= Y
>> * a consumer expecting X.Y will work with X.Y' when Y' < Y by version
>>   detection and degraded functionality (Y' level). If Y level function
>>   is required then it will gracefully die (able to detect rather
>>   than AbstractMethodError)
>> * X.Y defines the interface version, i.e. X.Y.Z works with X.Y.Z' for
>>   all Z'
>> * X defines the compatibility level i.e. X and X' are not guaranteed
>>   to work together (although we will try to ensure they do)
>>
>> Basic rules:
>> * at the Z level, no API change is allowed, just implementation changes
>> * at the Y level, additive API changes are allowed but must be
>>   accompanied by support in the versioning mechanism so that they can be
>>   detected. Things can be deprecated
>> * at the Z level, incompatible changes are allowed. Items deprecated in
>>   version X can be removed in version X+2 (implicitly (X+2).0.0).
>>
>> This all applies to operation in a single classloader. Where multiple
>> classloaders are involved (inside one VM or in multiple VMs) a
>> different set of versioning behaviours applies to the wire protocol.
>>
>> --
>> Jeremy

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Kathey Marsden]

David W. Van Couvering wrote:

> Hi, all.  I haven't gotten any more comments on the proposal at
> http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm
> going to update this page based on the comments I have received so far
> and then we'll try for another vote.
>

David will there be another document that describes the packaging and
any external impact of this change or will it be included in this one?

Thanks

Kathey

[David Van Couvering]

Thanks, Kathey, I missed that, I'll add that to this page.

David

Kathey Marsden wrote:

> David W. Van Couvering wrote:
>
>
>>Hi, all.  I haven't gotten any more comments on the proposal at
>>http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm
>>going to update this page based on the comments I have received so far
>>and then we'll try for another vote.
>>
>
>
> David will there be another document that describes the packaging and
> any external impact of this change or will it be included in this one?
>
> Thanks
>
> Kathey
>
>

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Daniel John Debrunner]

David W. Van Couvering wrote:

> Hi, all.  I haven't gotten any more comments on the proposal at
> http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm going
> to update this page based on the comments I have received so far and
> then we'll try for another vote.

Is there any reason this (and other wiki pages added by folks) are not
linked into the front page of the wiki? Is this just standard wiki
practice or are we just missing making the pages more visible?

It seems like unless you know of the page then it's hard to join in any
discussion about it, the automated wiki diffs sent to the committer list
are basically unreadable.



Dan.

[David Van Couvering]

I can gladly add this to the front page, and perhaps we can make this a
general policy.  I would like to suggest that we have a sub-page off the
front page for proposals that are being discussed/reviewed on the Wiki site.

David

Daniel John Debrunner wrote:

> David W. Van Couvering wrote:
>
>
>>Hi, all.  I haven't gotten any more comments on the proposal at
>>http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm going
>>to update this page based on the comments I have received so far and
>>then we'll try for another vote.
>
>
> Is there any reason this (and other wiki pages added by folks) are not
> linked into the front page of the wiki? Is this just standard wiki
> practice or are we just missing making the pages more visible?
>
> It seems like unless you know of the page then it's hard to join in any
> discussion about it, the automated wiki diffs sent to the committer list
> are basically unreadable.
>
>
>
> Dan.
>

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Kathey Marsden]

David W. Van Couvering wrote:

> I can gladly add this to the front page, and perhaps we can make this
> a general policy.  I would like to suggest that we have a sub-page off
> the front page for proposals that are being discussed/reviewed on the
> Wiki site.
>
That sounds like a good idea.

[David Van Couvering]

Hm, the front page appears to be an "Immutable Page."  Is this perhaps
because someone started editing it and didn't commit/back out their changes?

David

Daniel John Debrunner wrote:

> David W. Van Couvering wrote:
>
>
>>Hi, all.  I haven't gotten any more comments on the proposal at
>>http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm going
>>to update this page based on the comments I have received so far and
>>then we'll try for another vote.
>
>
> Is there any reason this (and other wiki pages added by folks) are not
> linked into the front page of the wiki? Is this just standard wiki
> practice or are we just missing making the pages more visible?
>
> It seems like unless you know of the page then it's hard to join in any
> discussion about it, the automated wiki diffs sent to the committer list
> are basically unreadable.
>
>
>
> Dan.
>

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[David Van Couvering]

Sorry, pilot error, I needed to log in.

David W. Van Couvering wrote:

> Hm, the front page appears to be an "Immutable Page."  Is this perhaps
> because someone started editing it and didn't commit/back out their
> changes?
>
> David
>
> Daniel John Debrunner wrote:
>
>> David W. Van Couvering wrote:
>>
>>
>>> Hi, all.  I haven't gotten any more comments on the proposal at
>>> http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm going
>>> to update this page based on the comments I have received so far and
>>> then we'll try for another vote.
>>
>>
>>
>> Is there any reason this (and other wiki pages added by folks) are not
>> linked into the front page of the wiki? Is this just standard wiki
>> practice or are we just missing making the pages more visible?
>>
>> It seems like unless you know of the page then it's hard to join in any
>> discussion about it, the automated wiki diffs sent to the committer list
>> are basically unreadable.
>>
>>
>>
>> Dan.
>>

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---