VOTE: Approach for sharing code

[Kathey Marsden]

Jeremy Boynes wrote:

> Rather than address this ourselves (and just for Derby) by convoluted
> code-rewrites and avoidance of third party libraries we should
> encourage the application providers avoid this entirely by exhibiting
> discipline when using shared resources


You have an extremely important point  Jeremy.   I think as with any
discipline issue, education is key.  We need to make mixed versions
truly be the edge case that we all want it to be.  Once that is the
case,  it is going to provide us with much more flexibility.  What I
need as a person who talks to users and support folks all the time is a
good document and hopefully an example of what what  developers should
do to make sure their application load  the copy of Derby that they want
even if they find themselves in a JVM with another instance of  Derby
loaded by the classpath or if they use a component that might use
another version of Derby.  

I think the discipline that we want to be able to  expect is not
mentioned at all in our documentation and we need to focus on educating
our users on exactly what to do.   Would you be willing to write
something up that we could put in the documentation or provide a white
paper?     Helping the Users do the right thing might untie the hands of
the Engineers so they can make the Product better sooner.

Kathey

[Daniel John Debrunner]

Jeremy Boynes wrote:

> Daniel John Debrunner wrote:
>
>>
>> But as an aside, I once downloaded an open source project, which then
>> instructed me to fetch a further five or six common Apache libraries,
>> either at a specific version numbers or higher than a given version. I
>> followed all this correctly, set up the classpath and immediately got an
>> abstract method error.
>>
>> That's an example of badly managed common versions, and I do not wish
>> Derby to fall into the same trap.

>
> And how many open source projects have you downloaded or used without
> such issues? Please don't taint this from one bad experience.

Nothing tainted, I'm just pointing out there are bad ways to handle
common code.

> All of these lead to a simple end-user experience

and that's what we need, any solution needs to continue the simple
end-user experience.

Dan.

[Daniel John Debrunner]

David W. Van Couvering wrote:

> - It seems to me that the environments in which mixed versions in the
> same VM occur are:
>
>    * app servers, which already deal with this through multiple
> classloaders

I'm not sure that's a completely valid assumption. Some app servers have
different classloaders for different ear/war files, but not for
different JDBC providers. (which would be a Derby client and a Derby
engine for example).

Dan.

[Andrew McIntyre-2]

On 9/15/05, Kathey Marsden <kmarsdenderby@...> wrote:

> Jeremy Boynes wrote:
>
> >
> > This dooms us forever to reinvent any functionality that could
> > be provided by other projects.
> >
> We are not "doomed forever". Requiring a new jar file for new
> functionality seems an entirely reasonable thing to me and at that time
> we can impose whatever restrictions the community sees fit.  Requiring a
> new jar file to have the product continue work, is another matter all
> together.

+1!

It looks like we've got a really intractable situation here. There are
those against source/binary modification because of the inherent
problems with maintenance and debugging, but you can't package the
same classes in multiple jars because of signing/sealing and
classloader issues, and if you split the classes out into a new jar
then you've got usability and deployment issues. So what do you do?

I'm wondering if this is really the functionality to be considering
all of this for. Two or three small classes related to localization.
Is it really worth the trouble? Are we really going to make a common
jar file with two classes, without which you don't get error messages?
Really?

And remember, this isn't about reusing someone else's code or
libraries. I don't think anyone is going to be against reusing other
project's code or libraries in order to promote cross-project goodwill
and general open source happiness. We're talking about copying two of
our own classes from one part of the tree to the other.

I think we can all tackle the code-sharing problem with some new
functionality later. For which, by the way, I think a good candidate
would be integrating Lucene for full text indexing as a good trial for
both code reuse from other projects and code sharing among the
different parts of Derby. And personally, I'd like to see David be
able to finish localizing the error messages sometime this decade. :-)

andrew

[David Van Couvering]

I appreciate your pragmatic approach, Andrew.  The thing is I have seen
a number of other pieces of functionality queued up for code sharing
between client and server.  These include DRDA network code, potentially
some higher-level JDBC functionality, logging and tracing, and
management via JMX.  I also discovered that the versioning mechanism in
org.apache.derby.iapi.services.info is actually reused across tools,
client and engine.

Given this, I thought that although it would be a tough debate it was
worthwhile having and one which, if we solved, would enable a lot of
other common services to be built across client and engine.  So I took
the challenge and decided to try and solve this.

We may still decide this is intractable and throw up our hands, but I
hope that's not the case.  It's been a healthy debate, and has brought
to bear some good discussions about "who" we want to be, what our key
principles are, and I think that's been good too.

You talked about signing/sealing problems about packaging the classes in
multiple jars.  I saw this mentioned before, can you or someone else
elaborate?  If it's really true that we can't package the same classes
in multiple jars, that's a very important data point...

Thanks,

David

Andrew McIntyre wrote:

> On 9/15/05, Kathey Marsden <kmarsdenderby@...> wrote:
>
>>Jeremy Boynes wrote:
>>
>>
>>>This dooms us forever to reinvent any functionality that could
>>>be provided by other projects.
>>>
>>
>>We are not "doomed forever". Requiring a new jar file for new
>>functionality seems an entirely reasonable thing to me and at that time
>>we can impose whatever restrictions the community sees fit.  Requiring a
>>new jar file to have the product continue work, is another matter all
>>together.
>
>
> +1!
>
> It looks like we've got a really intractable situation here. There are
> those against source/binary modification because of the inherent
> problems with maintenance and debugging, but you can't package the
> same classes in multiple jars because of signing/sealing and
> classloader issues, and if you split the classes out into a new jar
> then you've got usability and deployment issues. So what do you do?
>
> I'm wondering if this is really the functionality to be considering
> all of this for. Two or three small classes related to localization.
> Is it really worth the trouble? Are we really going to make a common
> jar file with two classes, without which you don't get error messages?
> Really?
>
> And remember, this isn't about reusing someone else's code or
> libraries. I don't think anyone is going to be against reusing other
> project's code or libraries in order to promote cross-project goodwill
> and general open source happiness. We're talking about copying two of
> our own classes from one part of the tree to the other.
>
> I think we can all tackle the code-sharing problem with some new
> functionality later. For which, by the way, I think a good candidate
> would be integrating Lucene for full text indexing as a good trial for
> both code reuse from other projects and code sharing among the
> different parts of Derby. And personally, I'd like to see David be
> able to finish localizing the error messages sometime this decade. :-)
>
> andrew

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Satheesh Bandaram]

+1. I also vote to solve the common code issues when we have sufficient
critical mass of it. Currently
there doesn't seem to be enough to justify a new mechanism, especially a
solution that is directly visible
to end users and could cause some disruptions. While the discussion has
been very useful to evolve a
possible solution in the future, I don't see an urgent need to address
this yet. Not sure how end-users
would see a derbycommon.jar that is only a few kilobytes (tens?) in size
and wonder why it is even there.

We just released a 10.1 feature release... Not sure when our next
feature release is going to be, but seems
like there might be many months of time to address any packaging issues.
Let us continue to enhance
Derby in the current framework and as we get closer to another release,
we could evaluate the packaging
issues.

Evolution, not revolution... Someone said this before, don't remember
who ... :-)

Satheesh

Andrew McIntyre wrote:

>+1!
>
>It looks like we've got a really intractable situation here. There are
>those against source/binary modification because of the inherent
>problems with maintenance and debugging, but you can't package the
>same classes in multiple jars because of signing/sealing and
>classloader issues, and if you split the classes out into a new jar
>then you've got usability and deployment issues. So what do you do?
>
>I'm wondering if this is really the functionality to be considering
>all of this for. Two or three small classes related to localization.
>Is it really worth the trouble? Are we really going to make a common
>jar file with two classes, without which you don't get error messages?
>Really?
>
>And remember, this isn't about reusing someone else's code or
>libraries. I don't think anyone is going to be against reusing other
>project's code or libraries in order to promote cross-project goodwill
>and general open source happiness. We're talking about copying two of
>our own classes from one part of the tree to the other.
>
>I think we can all tackle the code-sharing problem with some new
>functionality later. For which, by the way, I think a good candidate
>would be integrating Lucene for full text indexing as a good trial for
>both code reuse from other projects and code sharing among the
>different parts of Derby. And personally, I'd like to see David be
>able to finish localizing the error messages sometime this decade. :-)
>
>andrew
>
>
>
>  
>

[Andrew McIntyre-2]

On 9/16/05, David W. Van Couvering <David.Vancouvering@...> wrote:
> I appreciate your pragmatic approach, Andrew.  The thing is I have seen
> a number of other pieces of functionality queued up for code sharing
> between client and server.  These include DRDA network code, potentially
> some higher-level JDBC functionality, logging and tracing, and
> management via JMX.  I also discovered that the versioning mechanism in
> org.apache.derby.iapi.services.info is actually reused across tools,
> client and engine.

This is true, I realize that there are a lot of opportunities for code
sharing and reuse. I just don't know if *this* is the one that is
worth it. I think the way these possible code-sharing opportunities
should all be handled is for someone to implement some new
functionality as a module and package it into a new *optional* jar
file. The new code can be used if it is present, and the old code can
be deprecated, but remain packaged for a certain number of releases
before it is removed, if it is removed. This gives users time to
prepare for a change and developers to work through all the issues
without causing any unnecessary pain along the way.

Do we really want to go to this trouble for the client localization
functionality? I don't know. To me, logging seems like a better first
project for that.

> We may still decide this is intractable and throw up our hands, but I
> hope that's not the case. It's been a healthy debate, and has brought
> to bear some good discussions about "who" we want to be, what our key
> principles are, and I think that's been good too.

I agree, it's been a good healthy debate, but I thought it might be
good to reflect upon the task at hand. :-)
 
> You talked about signing/sealing problems about packaging the classes in
> multiple jars.  I saw this mentioned before, can you or someone else
> elaborate?

When a jar is sealed, classes cannot be loaded from multiple if the
package they are in is sealed. You can unseal specific packages, so
that's less of an issue than for signed jars. I believe attempting to
load the same class from different jars if the jars are signed will
cause a SecurityException, but I'm not well versed in the ins and outs
of signed jars.

andrew

[Daniel John Debrunner]

Andrew McIntyre wrote:


> When a jar is sealed, classes cannot be loaded from multiple if the
> package they are in is sealed. You can unseal specific packages, so
> that's less of an issue than for signed jars. I believe attempting to
> load the same class from different jars if the jars are signed will
> cause a SecurityException, but I'm not well versed in the ins and outs
> of signed jars.

I think signing and sealing are independent concepts.

If a class has been loaded from a signed jar, then it won't be loaded
again from any jar (within the same class loader), so I don't think
there can be any exception there.

A sealed package means that all classes within that package must be
loaded from the same jar. I assume this is true for a single
classloader, not across the vm.


Dan.

[Jeremy Boynes]

Evolution would start with a small change that was then built on and
expanded. From that perspective, starting with a small change like the
localization would be the beginning of an evolutionary change. Doing it
now, near the start of a feature release, and providing plenty of time
for feedback from users in the community is less disruptive than trying
to rush it in at the end.

This is one area where open source is different than closed source
development - all users get to see changes early and can provide
feedback into the process, not just a selected few during a beta cycle
when it is usually far to late.

As for the size of the common jar, once you go beyond one adding
additional libraries in becomes just more of the same. That leads to a
model where packaging is based on loose vs. close coupling
considerations. Some services may be small, some may be larger, but the
boundaries are determined by the use case and not an arbitrary packaging
scheme.

This was an issue originally with Jakarata Commons. In its early days,
although the libraries were separated out from the original monolith
they were still coupled together; it wasn't until later that the
decoupling of services we have now was attained (and to some extent that
is still a work in progress).

--
Jeremy

Satheesh Bandaram wrote:

> +1. I also vote to solve the common code issues when we have sufficient
> critical mass of it. Currently
> there doesn't seem to be enough to justify a new mechanism, especially a
> solution that is directly visible
> to end users and could cause some disruptions. While the discussion has
> been very useful to evolve a
> possible solution in the future, I don't see an urgent need to address
> this yet. Not sure how end-users
> would see a derbycommon.jar that is only a few kilobytes (tens?) in size
> and wonder why it is even there.
>
> We just released a 10.1 feature release... Not sure when our next
> feature release is going to be, but seems
> like there might be many months of time to address any packaging issues.
> Let us continue to enhance
> Derby in the current framework and as we get closer to another release,
> we could evaluate the packaging
> issues.
>
> Evolution, not revolution... Someone said this before, don't remember
> who ... :-)
>
> Satheesh
>
> Andrew McIntyre wrote:
>
>
>>+1!
>>
>>It looks like we've got a really intractable situation here. There are
>>those against source/binary modification because of the inherent
>>problems with maintenance and debugging, but you can't package the
>>same classes in multiple jars because of signing/sealing and
>>classloader issues, and if you split the classes out into a new jar
>>then you've got usability and deployment issues. So what do you do?
>>
>>I'm wondering if this is really the functionality to be considering
>>all of this for. Two or three small classes related to localization.
>>Is it really worth the trouble? Are we really going to make a common
>>jar file with two classes, without which you don't get error messages?
>>Really?
>>
>>And remember, this isn't about reusing someone else's code or
>>libraries. I don't think anyone is going to be against reusing other
>>project's code or libraries in order to promote cross-project goodwill
>>and general open source happiness. We're talking about copying two of
>>our own classes from one part of the tree to the other.
>>
>>I think we can all tackle the code-sharing problem with some new
>>functionality later. For which, by the way, I think a good candidate
>>would be integrating Lucene for full text indexing as a good trial for
>>both code reuse from other projects and code sharing among the
>>different parts of Derby. And personally, I'd like to see David be
>>able to finish localizing the error messages sometime this decade. :-)
>>
>>andrew
>>
>>
>>
>>
>>
>
>

[David Van Couvering]

It sounds to me then that it's OK to have the same packages in derby.jar
and derby-client.jar.  The classes in the common package would get
loaded from one and only of the jar files.  We just need them in both so
the client and engine jars can be independent.

David

Daniel John Debrunner wrote:

> Andrew McIntyre wrote:
>
>
>
>>When a jar is sealed, classes cannot be loaded from multiple if the
>>package they are in is sealed. You can unseal specific packages, so
>>that's less of an issue than for signed jars. I believe attempting to
>>load the same class from different jars if the jars are signed will
>>cause a SecurityException, but I'm not well versed in the ins and outs
>>of signed jars.
>
>
> I think signing and sealing are independent concepts.
>
> If a class has been loaded from a signed jar, then it won't be loaded
> again from any jar (within the same class loader), so I don't think
> there can be any exception there.
>
> A sealed package means that all classes within that package must be
> loaded from the same jar. I assume this is true for a single
> classloader, not across the vm.
>
>
> Dan.
>

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[David Van Couvering]

Hi, all.  I've been letting this brew in my subconscious this weekend,
with the strong belief that *somehow* there is a solution that can meet
all of our requirements (simple architecture, ease-of-use,
compatibility, no regressions, easy to maintain, reduce code duplication).

I think I may have something that will do this, and your comments are
much appreciated.  There is a bit of overhead for the release process,
but I think if we are going to have overhead, that's the place to do it.

The principle is that for each release, the common package is unique.
This is achieved by appending the release number to the package name.
So for the 10.1.2 release the package name is org.apache.common_1012.
In this way a consumer of the common classes for a given version is
guaranteed to load the right classes and no conflicts or confusion occur.

As part of the release process, when you make a branch for a release,
you run a script that renames the common package to match your release
name.  This script does the following:

- In the source, replace all instances of org.apache.common_<oldversion>
  with org.apache.common_<newversion>

- Do an svn move of java/common/org/apache/common_<oldversion> to
java/common/org/apache/common_<newversion>

Because you are using svn move, although it's a bit confusing, all
history is maintained and you can still use svn to do merges and ports
of changes.

The common package is put into both derby.jar and derby-client.jar, and
we do not create a new JAR file.

Thanks,

David

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[francois.orsini]

I think you meant 'org.apache.derby.common' in your examples.

Now, if the common package versioning is well automated as part of generating a new release, then it is probably ok to have a new version being generated for every release.

Are you thinking that we would always see 'org.apache.derby.common' in the source tree and that the common pkg version suffix would only be added when a (new) release gets generated?

--francois

On 9/19/05, David W. Van Couvering <David.Vancouvering@...> wrote:

Hi, all.  I've been letting this brew in my subconscious this weekend,
with the strong belief that *somehow* there is a solution that can meet
all of our requirements (simple architecture, ease-of-use,
compatibility, no regressions, easy to maintain, reduce code duplication).

I think I may have something that will do this, and your comments are
much appreciated.  There is a bit of overhead for the release process,
but I think if we are going to have overhead, that's the place to do it.

The principle is that for each release, the common package is unique.
This is achieved by appending the release number to the package name.
So for the 10.1.2 release the package name is org.apache.common_1012 .
In this way a consumer of the common classes for a given version is
guaranteed to load the right classes and no conflicts or confusion occur.

As part of the release process, when you make a branch for a release,
you run a script that renames the common package to match your release
name.  This script does the following:

- In the source, replace all instances of org.apache.common_<oldversion>
  with org.apache.common _<newversion>

- Do an svn move of java/common/org/apache/common_<oldversion> to
java/common/org/apache/common_<newversion>

Because you are using svn move, although it's a bit confusing, all
history is maintained and you can still use svn to do merges and ports
of changes.

The common package is put into both derby.jar and derby-client.jar, and
we do not create a new JAR file.

Thanks,

David

[David Van Couvering]

Francois Orsini wrote:
> I think you meant 'org.apache.derby.common' in your examples.

Yes, thanks.

>
> Now, if the common package versioning is well automated as part of
> generating a new release, then it is probably ok to have a new version
> being generated for every release.
>
> Are you thinking that we would always see 'org.apache.derby.common' in
> the source tree and that the common pkg version suffix would only be
> added when a (new) release gets generated?

Well, this is a good question, if you're asking about what should the
package name be in the trunk.  Yes, I suppose it could just be
'org.apache.derby.common' and the rename happens only when a branch is
made for a release.  I am trying to think if this would cause any issues
for people pulling the nightly builds -- I can't think of any, unless
people try to mix versions from old and new nightly builds, in which
case I think they're really on their own :)

An alternate approach is to do the rename on the trunk just prior to
making the branch, so the trunk package name always matches the most
recent release branch.  But this doesn't seem as clean...

David


[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Rick Hillegas-2]

There are now three cloning proposals for handling shared code:

1) Hand cloning by the developer as is done today.

2) Build-time cloning.

3) Release-time cloning (David's latest proposal).

All of these cloning approaches address Dan's problem case: a client and
server at diffferent rev levels in the same classpath. It's worth
pointing out that none of these proposals addresses the other mixed
usage scenario: wiring into the classpath two applications each of which
embeds a different version of the server. In that scenario we can still
expect Heisenbugs. We've never tested this scenario and so we have no
idea what our current exposure is. I agree with Jeremy that this
scenario is a systemic defect in Java and that we are not responsible
for solving it. The best we can do is point our users at best-of-breed
techniques for containing the problem.

Here's what I think about the cloning proposals:

(1)
+ Requires no changes.
- Brittle, bug-prone.
- Can't cut clone bloat by refactoring jar files.

(2)
+ Simple for bug porting.
- Can't cut clone bloat by refactoring jar files.
- Awkward to code and debug.

(3)
+ Easy to code and debug.
+ Clone bloat not necessary if client and server at same rev level.
- Awkward for bug porting.

In and of themselves, none of these approaches presents the naive
customer any additional complexity out-of-the-box. Additional complexity
would vex the customer if, for option (3) we refactored the jar files to
cut clone bloat. I dislike (3) the least.

Cheers,
-Rick

[Jeremy Boynes]

I find this arcane and confusing; worse, I don't think it really works.

Code that is using the common API becomes tied to a specific version. By
renaming the package we force all users of the common code to modify
their code. We can do that for things in our source tree but not in others.

Similarly this forces us to modify any external libraries we use to
exhibit the same behaviour - for example, renaming o.a.commons.logging
to o.a.commons.logging_1208. This means we have custom versions of all
external libraries.

Think of the carnage if it was java_14208_b13.sql_300.Connection

--
Jeremy

David W. Van Couvering wrote:

> Hi, all.  I've been letting this brew in my subconscious this weekend,
> with the strong belief that *somehow* there is a solution that can meet
> all of our requirements (simple architecture, ease-of-use,
> compatibility, no regressions, easy to maintain, reduce code duplication).
>
> I think I may have something that will do this, and your comments are
> much appreciated.  There is a bit of overhead for the release process,
> but I think if we are going to have overhead, that's the place to do it.
>
> The principle is that for each release, the common package is unique.
> This is achieved by appending the release number to the package name. So
> for the 10.1.2 release the package name is org.apache.common_1012. In
> this way a consumer of the common classes for a given version is
> guaranteed to load the right classes and no conflicts or confusion occur.
>
> As part of the release process, when you make a branch for a release,
> you run a script that renames the common package to match your release
> name.  This script does the following:
>
> - In the source, replace all instances of org.apache.common_<oldversion>
>  with org.apache.common_<newversion>
>
> - Do an svn move of java/common/org/apache/common_<oldversion> to
> java/common/org/apache/common_<newversion>
>
> Because you are using svn move, although it's a bit confusing, all
> history is maintained and you can still use svn to do merges and ports
> of changes.
>
> The common package is put into both derby.jar and derby-client.jar, and
> we do not create a new JAR file.
>
> Thanks,
>
> David

[Daniel John Debrunner]

Jeremy Boynes wrote:

> Think of the carnage if it was java_14208_b13.sql_300.Connection

It's actually instructive to look how Java solves this:

- The interface is kept upwards compatible, by following rules such as
only new methods/fields etc.

Then to look at how a consumer, Derby, deals with the fact of multiple
versions of the interface:

- Derby only provides and uses the functionality to match the version of
the interface (java.sql.Connection) loaded, determined at runtime.

I'd thought this was a workable direction being proposed about six
thousand message ago, upwards compatible apis and the ability for a
consumer to handle a lower version. Not sure what derailed it.

Dan.

[David Van Couvering]

Hm, that has continued to be the proposal I liked best.  I thought what
derailed it was the inability to guarantee forward compatibility, that
is when a new consumer depends upon a new API.

For example, let's say you have an interface in release 1:

public MyInterface
{
   int method1(int val);
}

Now in release 2 the consumer wants to add a new method:

public MyInterface
{
   int method1(int val);
   String method2(String val);
}

This is a backward-compatible change but not a forward-compatible
change, because if the new consumer calls method2 on the original
MyInterface, he'll get an error.

You could argue this is solvable by saying any modifications to an
interface require a new interface:

public MyInterface2 extends MyInterface
{
   int method2(String val);
}

This is both backward- and forward-compatible.

The problem with this is the jar sealing problem.  If the old jar is
first in the classpath, then MyInterface will be loaded from the old jar
and MyInterface2 will be loaded from the new jar.  If these interfaces
are in the same package (org.apache.derby.common) then you'll get a
SecurityException.

I don't think that Java solves this particular problem, as it generally
assumes you have only one version in your classpath at a given time.
That's what I think has continued to derail any solution we propose.

Thus my proposal to rename the package, to solve the jar sealing problem.

David

Daniel John Debrunner wrote:

> Jeremy Boynes wrote:
>
>
>>Think of the carnage if it was java_14208_b13.sql_300.Connection
>
>
> It's actually instructive to look how Java solves this:
>
> - The interface is kept upwards compatible, by following rules such as
> only new methods/fields etc.
>
> Then to look at how a consumer, Derby, deals with the fact of multiple
> versions of the interface:
>
> - Derby only provides and uses the functionality to match the version of
> the interface (java.sql.Connection) loaded, determined at runtime.
>
> I'd thought this was a workable direction being proposed about six
> thousand message ago, upwards compatible apis and the ability for a
> consumer to handle a lower version. Not sure what derailed it.
>
> Dan.
>
>
>
>

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[David Van Couvering]

This proposal was based on the assumption that our common code would
*never* be exposed to the outside world.

I also never claimed that we should apply this solution to third-party
components.  That's a fish I don't think we need to fry right now.

I think you are probably unhappy with this solution because it goes
against your vision of a modular architecture for Derby.

If you don't assume that any "module" of Derby can go with any other
version of any other module, but instead assume that all of Derby goes
together as a single unit, then it's fine if the client and engine code
for a given version are tied to the common code for the same version.
It actually makes sense -- you have well-defined and predictable
behavior in an unpredictable environment where multiple versions live in
the same classpath.

If you aren't thinking modular, I personally don't think it's that
arcane and confusing, particularly compared to making multiple copies of
an original "common" class.

I also can't think of any other solution besides class cloning (which I
really dislike) that can solve this problem and still give us common code.

David

Jeremy Boynes wrote:

> I find this arcane and confusing; worse, I don't think it really works.
>
> Code that is using the common API becomes tied to a specific version. By
> renaming the package we force all users of the common code to modify
> their code. We can do that for things in our source tree but not in others.
>
> Similarly this forces us to modify any external libraries we use to
> exhibit the same behaviour - for example, renaming o.a.commons.logging
> to o.a.commons.logging_1208. This means we have custom versions of all
> external libraries.
>
> Think of the carnage if it was java_14208_b13.sql_300.Connection
>
> --
> Jeremy
>
> David W. Van Couvering wrote:
>
>> Hi, all.  I've been letting this brew in my subconscious this weekend,
>> with the strong belief that *somehow* there is a solution that can meet
>> all of our requirements (simple architecture, ease-of-use,
>> compatibility, no regressions, easy to maintain, reduce code
>> duplication).
>>
>> I think I may have something that will do this, and your comments are
>> much appreciated.  There is a bit of overhead for the release process,
>> but I think if we are going to have overhead, that's the place to do it.
>>
>> The principle is that for each release, the common package is unique.
>> This is achieved by appending the release number to the package name.
>> So for the 10.1.2 release the package name is org.apache.common_1012.
>> In this way a consumer of the common classes for a given version is
>> guaranteed to load the right classes and no conflicts or confusion occur.
>>
>> As part of the release process, when you make a branch for a release,
>> you run a script that renames the common package to match your release
>> name.  This script does the following:
>>
>> - In the source, replace all instances of
>> org.apache.common_<oldversion>  with org.apache.common_<newversion>
>>
>> - Do an svn move of java/common/org/apache/common_<oldversion> to
>> java/common/org/apache/common_<newversion>
>>
>> Because you are using svn move, although it's a bit confusing, all
>> history is maintained and you can still use svn to do merges and ports
>> of changes.
>>
>> The common package is put into both derby.jar and derby-client.jar,
>> and we do not create a new JAR file.
>>
>> Thanks,
>>
>> David
>
>

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[David Van Couvering]

David W. Van Couvering wrote:
> This proposal was based on the assumption that our common code would
> *never* be exposed to the outside world.
>
> I also never claimed that we should apply this solution to third-party
> components.  That's a fish I don't think we need to fry right now.
>

Although I do want to remind us all that, based on Dan's email a while
back, the JDK itself encountered the mixed version problem and solved/is
solving it through package renaming...

http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c43275363.6000503@...%3e

"The future plan is to rename the org.apache.** packages to be something
like com.sun.org.apache.** to fix this problem."

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Jeremy Boynes]

Daniel John Debrunner wrote:

> Jeremy Boynes wrote:
>
>
>>Think of the carnage if it was java_14208_b13.sql_300.Connection
>
>
> It's actually instructive to look how Java solves this:
>
> - The interface is kept upwards compatible, by following rules such as
> only new methods/fields etc.
>
> Then to look at how a consumer, Derby, deals with the fact of multiple
> versions of the interface:
>
> - Derby only provides and uses the functionality to match the version of
> the interface (java.sql.Connection) loaded, determined at runtime.
>
> I'd thought this was a workable direction being proposed about six
> thousand message ago, upwards compatible apis and the ability for a
> consumer to handle a lower version. Not sure what derailed it.
>

Kathey asked if I would be willing to write something up and I started
thinking about it but haven't had the bandwidth to write anything.

As a quick braindump:
* there are rules that restrict changes to the interface going forward
* Sun (et al) stick to those rules even when it is painful
* no class/method is removed without being deprecated first
* the JVM supports version skew a little by not requiring an
   implementation to implement all methods in a interface (at runtime)
* the interfaces often provide a mechanism to determine what features of
   an API the implementation implements
* frameworks can utilize multiple classloaders to load multiple versions
   of a class into the VM - tools have developed to simplify this

I think we can support a scenario where:
* a comsumer expecting version X.Y will work with any implementation
   X.Y' when Y' >= Y
* a consumer expecting X.Y will work with X.Y' when Y' < Y by version
   detection and degraded functionality (Y' level). If Y level function
   is required then it will gracefully die (able to detect rather
   than AbstractMethodError)
* X.Y defines the interface version, i.e. X.Y.Z works with X.Y.Z' for
   all Z'
* X defines the compatibility level i.e. X and X' are not guaranteed
   to work together (although we will try to ensure they do)

Basic rules:
* at the Z level, no API change is allowed, just implementation changes
* at the Y level, additive API changes are allowed but must be
   accompanied by support in the versioning mechanism so that they can be
   detected. Things can be deprecated
* at the Z level, incompatible changes are allowed. Items deprecated in
   version X can be removed in version X+2 (implicitly (X+2).0.0).

This all applies to operation in a single classloader. Where multiple
classloaders are involved (inside one VM or in multiple VMs) a different
set of versioning behaviours applies to the wire protocol.

--
Jeremy