VOTE: Approach for sharing code

[Kathy Saunders]

Jeremy Boynes wrote:

> Kathy Saunders wrote:
>
>>
>
> <snip/>
>
>>
>> I'm a bit concerned because I see a lot of discussion about what is
>> good from a derby development perspective, but not so much how these
>> changes may affect users of Derby.  Although some Derby users have
>> complex applications (like application servers), many are
>> implementing much more simple solutions.
>
>
> I would argue that we are actually making life easier for people
> implementing simple solutions. To me, a simple environment does not
> need to cater for multiple versions being concurrently loaded or for
> multi-classloader operation; it also means being able to select just
> the functionality you need without having to worry about which jar
> file a class may have come from.
>
> I think for that environment, just adding the component jars to the
> classpath (without any concern for ordering) is reasonable.
>
> To make things even simpler, it has also been proposed that we bundle
> all components together into one jar (containing everything, client
> and server). This gives you less flexibility and a larger footprint
> but is a really simple solution.
>
> <snip/>
>

I have to say that I don't see how adding more jar files to figure out
whether you need to deploy and add to your classpath makes things easier
for the simple case.  And, so far, I don't see what our users would
reasonably be able to  pick and choose--what would they be able to leave
out of our database engine other than how the jar files are already
separated (embedded, network server, tools...)?  Unless I'm missing
something, David is currently working on internationalizing error
messages.  Would it really make sense to tell someone they may not need
that functionality? Will they be able to get error messages for network
server without having those classes in their classpath? I could imagine
scenarios in the future where there may be significant pieces of
functionality that we would want to separate because not everyone
wants/needs that functionality and it would significantly add to
footprint, but I can't think of anything in that category that currently
exists other than what we already have.  For example, we do have a
separate jar file for tools.

Footprint is an interesting argument, but will we really see any
significant differences there yet?  Strictly looking at this from a
usability perspective, I still believe that having a common.jar file
which has no real meaning to a Derby user (since I believe you'll always
need in the network server case at this point), so why have them keep
track of yet another jar file?

If we do have a separate jar file for these classes, I believe that it
should only be one at this point and classpath order should not matter.  
Again, I'm not saying there may not be a need for more jar files in the
future.  I'm only looking at what I believe is proposed right now.

>>
>> In addition, I work on Derby now in the testing area, so I'd also
>> like to understand the implications for what additional testing might
>> need to be done.  If we create more jar files, is there more testing
>> requirements for different combinations?
>>
>
> I don't think there are any more combinations - in fact probably less
> as you would not need to test all possible classpath orderings. We are
> dealing with the same amount of code, just modularizing its structure.
>
> By modularizing the build we also allow for in-depth testing on each
> individual component in isolation. With a clear definition of the API
> contract for each component and testing (unit, functional,
> compatibility) of that contract we can perform more thorough testing
> on each one before integrating into a whole. Integration and system
> testing can focus on the interfaces between components rather than on
> the entire black box.
>
> Add in too that modularization makes it easier for users and
> developers to come up to speed with the design and implementation of
> that component. More eyes on the code with comprehensible component
> leads to better review and higher quality.
>
> Finally, you can see this pattern at work with many open source
> projects: a common core and then a very modular structure that allows
> people to participate at the component level. Examples of projects
> with this type of structure are:
> * Apache HTTPD + mod_*
> * Apache Maven + plugins
> * Eclipse + plugins
> * Apache Jakarta/Tomcat + Commons
> and many more.
>
> --
> Jeremy
>
>
>
>

Thanks for your perspective on the testing issue.

Kathy

[Kathy Saunders]

David W. Van Couvering wrote:

> Hi, Kathy, thanks for your email.  The timing is actually pretty good,
> I was just talking with Francois trying to understand his concerns
> better.
>
> After quickly describing the two approaches, I'd like to summarize the
> experience/impact of these approaches from the perspectives of the end
> user, the developer/maintainer, and the test developer/runner.
>
> Goal:
>  - Reduce code duplication while continuing to support different
> versions of client and embedded driver in the same VM
>
> Approach 1:
>  - Create a common package and put all common code there
> (org.apache.derby.common)
>  - Use compatibility guidelines to ensure backward compatibility and
> some degree of forward compatibility
>  - Common classes are embedded in derby.jar and derby-client.jar
> (based on lots of negative feedback for having a separate jar)
>
> Approach 2:
>  - Create a common package and put all common code there
> (org.apache.derby.common)
>  - During build process, "clone" all common classes into at least two
> generated packages, one for the engine and one for the network client
> (org.apache.derby.engine.common and org.apache.derby.client.common).
>  - This approach avoids having to implement compatiblity
> guidelines/constraints and guarantees, as the engine and client
> continue to be fully isolated
>
> USER EXPERIENCE
>
> Approach 1
>  - No new jar files, everything looks the same
>  - For the vast percentage of users who don't mix versions in the same
> VM, everything works great
>  - For the small percentage of users who mix versions, the order in
> which jar files are loaded.  For example, if they want to use
> derby-10.2.jar and derby-client-11.0.jar, then if derby-10.2.jar is
> first in the classpath, they will get an exception saying that the
> client code is incompatible with the common code and that they need to
> re-order their jar files, whereas it will work fine if the order is
> reversed.
>
> Approach 2
>  - No new jar files, everything looks the same
>  - Ordering of jar files does not matter, everything works fine
>
>
> DEVELOPER EXPERIENCE
>
> Approach 1
>  - Pretty much as it is today, nothing surprising
>
> Approach 2
>  - When navigating code either during source browsing or debugging,
> they will see the *generated* common code, not the master common code.
>  - If a developer is not aware of how things work, or just forgets,
> he/she will try to edit this generated code, and will be confused and
> surprised when his/her changes disappear after a build
>  - Stack traces will point to generated code, and the developer will
> have to remember to translate that back to the master version.
>  - The generation process must be very careful not to adjust line
> numbers or all stack traces will be off and misleading.  This means
> for instance we can't add comments saying "THIS IS GENERATED CODE, DO
> NOT EDIT"
>  - We may need to generate more copies if different types of version
> mixing are required (e.g. if the tools.jar and derby.jar need to be at
> different versions)
>
>
> TESTER EXPERIENCE
> Approach 1
>  - We would have to build and run compatibility tests to make sure
> that compatible versions run correctly and incompatible ones correcty
> raise the exception
>
> Approach 2
>  - No real change, although debugging of tests may be confusing due to
> issues I already listed under developer experience
>
>
> I also am uncomfortable with the "twistiness" of approach 2.  There is
> something to be said for a clean architecture and build environment.  
> I have seen time and again that a good architecture allows you to
> scale and grow, whereas "twisty" architectures tend to wrap you up and
> tie you down at some point.  I think this has to be taken into
> consideration.
>
> My main question is: is it OK to sometimes throw an exception for the
> small set of users who mix versions, and for them to then have to
> rearrange their jar ordering.  If the answer is that this is not
> acceptable and would be considered a showstopper regression for some
> part of our user base, then I think we have no choice but to go with
> Approach 2, even if we do risk painting ourselves into an
> architectural corner.  Otherwise, it is my strong recommendation to go
> with Approach 1.
>
> Thanks!
>
> David
>

Hi David,

Thank you very much for the clear explanation.  From a usability
perspective, I would vote for approach 2.  Requiring a classpath to be
in a particular order is always an issue.  However, the saving grace is
that it sounds like the ordering issue only comes up if you mix versions
of the derby.jar and the derbyclient.jar in the same classpath.  I don't
believe most users put the client and engine in the same classpath
(unless there's a new requirement I don't know about), so that
definitely helps.  Requiring classpath in a specific order can easily
lead to complications though, so I'm not in favor of it in general.

[David Van Couvering]

Hi, Kathy, I am a bit confused by your response.  You seem to be OK with
the classpath ordering issue given that it's an edge case, but you say
you want to go for approach 2.  Approach 2 does not require any
classpath ordering.  

Also, note that in general classpath ordering isn't important except in
the case where the user is already consciously futzing with the
classpath to mix versions in the same VM.

David

>>
> Hi David,
>
> Thank you very much for the clear explanation.  From a usability
> perspective, I would vote for approach 2.  Requiring a classpath to be
> in a particular order is always an issue.  However, the saving grace
> is that it sounds like the ordering issue only comes up if you mix
> versions of the derby.jar and the derbyclient.jar in the same
> classpath.  I don't believe most users put the client and engine in
> the same classpath (unless there's a new requirement I don't know
> about), so that definitely helps.  Requiring classpath in a specific
> order can easily lead to complications though, so I'm not in favor of
> it in general.

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[David Van Couvering]

Hi, Kathey.  My responses below...

Kathey Marsden wrote:

>David W. Van Couvering wrote:
>
>  
>
>>Hi, Kathy, thanks for your email.  The timing is actually pretty good,
>>I was just talking with Francois trying to understand his concerns
>>better.
>>
>>After quickly describing the two approaches, I'd like to summarize the
>>experience/impact of these approaches from the perspectives of the end
>>user, the developer/maintainer, and the test developer/runner.
>>
>>    
>>
>
>Thank you David for the summary.  I thought "modular build"  meant
>adding more jars,  so it was good to have that cleared  With Approach 1
>I personally am not too keen on the new ordering requirements,  testing
>requirements, or the potential for regression with the versioning scheme.
>  
>

Note that we already have to do compatibility testing between client and

server because of protocol changes.  But yes, this does increase the
amount of testing we have to do.

The ordering requirements are for a very small subset of users.  For
most users, ordering does not matter at all.

The regression question to me is the clincher.  If we just are not able
to drop in a new version of Derby that could cause existing code to
break because of their classpath order, then Approach 1 is just not
feasible.  I am hoping we could document this change and the users to
whom this is important could with very little effort adjust how they set
up the order their jars are loaded.

In general I am concerned we are making architectural compromises for a
small subset of customers.  One of my managers at Sybase once told me
his priorities for making decisions around changes to a product: first
the product, then the customer, then the engineers.  What he meant was,
you don't want to compromise the overall "solidity" of the product for
the sake of a small set of customers.

>For the USER EXPERIENCE  for either approach, how much  growth do you
>anticipate in the client due to code sharing  with the first round of
>what you want to share and a guesstimate of how big it might get if we
>utilize all that you think the client should use from  the engine?  
>The earlier thread on the size of the common jar file
>http://mail-archives.apache.org/mod_mbox/db-derby-dev/200507.mbox/%3c42DF40D5.8050605@...%3e
>made it sound significant.
>
>  
>

That thread was assuming I was migrating the entire Monitor/Service

architecture over into common, and it was getting a bit big.  This is
not necessarily going to be required.  I have some thoughts, not yet
fully gelled, about how we might be able to introduce a lightweight
service architecture into common, and have this be compatible with the
heavier-weight service architecture currently in the engine.

I'd like to understand what requirements we have for the footprint of
the client jar.

Thanks,

David

>Kathey
>
>
>  
>

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Daniel John Debrunner]

Kathy Saunders wrote:

>
> Thank you very much for the clear explanation.  From a usability
> perspective, I would vote for approach 2.  Requiring a classpath to be
> in a particular order is always an issue.  However, the saving grace is
> that it sounds like the ordering issue only comes up if you mix versions
> of the derby.jar and the derbyclient.jar in the same classpath.  I don't
> believe most users put the client and engine in the same classpath
> (unless there's a new requirement I don't know about), so that
> definitely helps.  Requiring classpath in a specific order can easily
> lead to complications though, so I'm not in favor of it in general.

Derby's client and engine may be in the same classpath and at different
versions if the JVM is hosting more than one application, or the
application installers have modified the system/user's classpath to add
their required jars.

The issue is that today this is fully supported because the client and
engine do not share code.

Some of the code sharing approaches regress Derby in this area by not
supporting this, or require class path ordering for it to be supported.

While it is true that multiple class loaders solve the issue, this
approach is not always possible, I believe, for example, some of the
major application servers do not support different class loaders for
different JDBC providers (eg. the Derby client at 10.3 and engine and 10.2).

Thus the argument really is, are we willing to accept regression in this
area to gain code sharing, or should the code sharing solution not
regress Derby?

Dan.

[David Van Couvering]

I think Dan has really hit the nail on the head.  All debate will
quickly stop if we reach consensus that it's not acceptable to break
compatibility (it's not really a regression, we never had this
constraint on ordering before) in this way as part of a minor release
(e.g. 10.1 to 10.2).

I don't know the environment that the customers who depend this on work
in.  I don't know how flexible they can be to ordering how jars are
loaded.    It's really hard for us to answer this without their input.  
Could those of us who have these customers possibly talk to them and
discuss this and understand what their constraints are?  I think it's
reasonable to document this for future customers; I don't personally
think it's a major usability issue, the vast majority of users will work
with a single version of Derby in a given VM.

Note that the main reason these customers want to run with different
versions, as I understand it, is so that they can run with a version of
the client that matches the server version.  If we can guarantee
compatibility between client and server versions (so, for instance, you
can upgrade the server without having to upgrade the clients, or vise
versa), does this requirement goes away?  I know at Sybase we never had
this issue because the TDS protocol allowed you to negotiate the version
of the protocol you were going to run at -- the server knew how to
"downgrade" its protocol support if it needed to.  This type of
negotiation and compatibility guarantee is something Rick Hillegas is
already working on...

Thanks,

David

Daniel John Debrunner wrote:

>Kathy Saunders wrote:
>
>
>  
>
>>Thank you very much for the clear explanation.  From a usability
>>perspective, I would vote for approach 2.  Requiring a classpath to be
>>in a particular order is always an issue.  However, the saving grace is
>>that it sounds like the ordering issue only comes up if you mix versions
>>of the derby.jar and the derbyclient.jar in the same classpath.  I don't
>>believe most users put the client and engine in the same classpath
>>(unless there's a new requirement I don't know about), so that
>>definitely helps.  Requiring classpath in a specific order can easily
>>lead to complications though, so I'm not in favor of it in general.
>>    
>>
>
>Derby's client and engine may be in the same classpath and at different
>versions if the JVM is hosting more than one application, or the
>application installers have modified the system/user's classpath to add
>their required jars.
>
>The issue is that today this is fully supported because the client and
>engine do not share code.
>
>Some of the code sharing approaches regress Derby in this area by not
>supporting this, or require class path ordering for it to be supported.
>
>While it is true that multiple class loaders solve the issue, this
>approach is not always possible, I believe, for example, some of the
>major application servers do not support different class loaders for
>different JDBC providers (eg. the Derby client at 10.3 and engine and 10.2).
>
>Thus the argument really is, are we willing to accept regression in this
>area to gain code sharing, or should the code sharing solution not
>regress Derby?
>
>Dan.
>
>
>  
>

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Daniel John Debrunner]

David W. Van Couvering wrote:


> Note that the main reason these customers want to run with different
> versions, as I understand it, is so that they can run with a version of
> the client that matches the server version.  If we can guarantee
> compatibility between client and server versions (so, for instance, you
> can upgrade the server without having to upgrade the clients, or vise
> versa), does this requirement goes away?

No.

Client/server compatibility is good, and soft upgrade is good, and all
help in the area of reducing version compatibility problems.

But I'm not saying anyone *wants* to mix versions, rather that they end
up mixing versions through circumstances.

I'm assuming worst case, end-user A knows nothing about Java, and is
using applications from vendor B (client at version 10.2) and vendor C
(engine at version 10.3).

Dan.

[Kathy Saunders]

Daniel John Debrunner wrote:

>Kathy Saunders wrote:
>
>
>  
>
>>Thank you very much for the clear explanation.  From a usability
>>perspective, I would vote for approach 2.  Requiring a classpath to be
>>in a particular order is always an issue.  However, the saving grace is
>>that it sounds like the ordering issue only comes up if you mix versions
>>of the derby.jar and the derbyclient.jar in the same classpath.  I don't
>>believe most users put the client and engine in the same classpath
>>(unless there's a new requirement I don't know about), so that
>>definitely helps.  Requiring classpath in a specific order can easily
>>lead to complications though, so I'm not in favor of it in general.
>>    
>>
>
>Derby's client and engine may be in the same classpath and at different
>versions if the JVM is hosting more than one application, or the
>application installers have modified the system/user's classpath to add
>their required jars.
>
>The issue is that today this is fully supported because the client and
>engine do not share code.
>
>Some of the code sharing approaches regress Derby in this area by not
>supporting this, or require class path ordering for it to be supported.
>
>While it is true that multiple class loaders solve the issue, this
>approach is not always possible, I believe, for example, some of the
>major application servers do not support different class loaders for
>different JDBC providers (eg. the Derby client at 10.3 and engine and 10.2).
>
>Thus the argument really is, are we willing to accept regression in this
>area to gain code sharing, or should the code sharing solution not
>regress Derby?
>
>Dan.
>
>
>
>
>
>
>  
>

I don't think we should regress and I don't think we should have
classpath ordering issues (sorry if my rambling mail last time was
misleading).  I think that means I would vote for option 2 if we proceed
with code sharing which means no change from a customer
perspective--same number of jars and no ordering issues.

[David Van Couvering]

Ah, I see, yes, you have a point.  Well, big swallow, it seems to me
that this is indeed a regression, and a potential usability headache.

Thinking further, this same issue exists for any platform where shared
code can come in through many fronts.  I was once told that one
distribution of Linux had three or four different copies of Berkeley
DB.  I guess the difference here is that shared libraries are versioned
and you can load a .so of a specific version if you need to, while you
can not do this with JAR files.

Unless someone can convince me otherwise, I am going to have to switch
boats and back Approach 2, the Common Clone approach.  I can hold my
architectural purity nose while writing the cloning code.

Thanks,

David

Daniel John Debrunner wrote:

>David W. Van Couvering wrote:
>
>
>  
>
>>Note that the main reason these customers want to run with different
>>versions, as I understand it, is so that they can run with a version of
>>the client that matches the server version.  If we can guarantee
>>compatibility between client and server versions (so, for instance, you
>>can upgrade the server without having to upgrade the clients, or vise
>>versa), does this requirement goes away?
>>    
>>
>
>No.
>
>Client/server compatibility is good, and soft upgrade is good, and all
>help in the area of reducing version compatibility problems.
>
>But I'm not saying anyone *wants* to mix versions, rather that they end
>up mixing versions through circumstances.
>
>I'm assuming worst case, end-user A knows nothing about Java, and is
>using applications from vendor B (client at version 10.2) and vendor C
>(engine at version 10.3).
>
>Dan.
>
>
>  
>

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Jeremy Boynes]

Daniel John Debrunner wrote:
>
> Derby's client and engine may be in the same classpath and at different
> versions if the JVM is hosting more than one application, or the
> application installers have modified the system/user's classpath to add
> their required jars.
>

In this model, you are assuming that there is a single classpath shared
by all applications run by the system/user. In this mode you cannot have
multiple versions of the client or engine present - you can only have
one version of the client and one version of the engine.

In reality this scenario is useless in anything but the most trivial
installation. As an application can be impacted by any other application
that affects the system/user's classpath reliability requires that they
be isolated from each other. You end up with two modes of operation:

* an application overides the classpath in some script before the JVM
   is started to ensure that only the jars it expects are present
* an application runs inside a server that controls the classpath for
   it and that server ensures the appropriate libraries are present

> The issue is that today this is fully supported because the client and
> engine do not share code.
>
> Some of the code sharing approaches regress Derby in this area by not
> supporting this, or require class path ordering for it to be supported.
>

Some of the others support this by defining compatibility contracts and
eliminate the need for classpath ordering by not duplicating classes.

> While it is true that multiple class loaders solve the issue, this
> approach is not always possible, I believe, for example, some of the
> major application servers do not support different class loaders for
> different JDBC providers (eg. the Derby client at 10.3 and engine and 10.2).
>

Those application servers also define which versions are supported by
the application server vendor. They also do not support client 10.2 and
client 10.3 which is an equally likely combination.

> Thus the argument really is, are we willing to accept regression in this
> area to gain code sharing, or should the code sharing solution not
> regress Derby?
>

Let's recharacterize this a little. What we are contemplating with code
sharing is extracting common functionality out into a library. By saying
that we are not willing to accept any solution where a component depends
on a library we are shutting ourselves off from using any external
library or any functionality not provided by Derby itself. This dooms us
forever to reinvent any functionality that could be provided by other
projects.

For example, there are libraries out there that support bytecode
generation, JMX for management, high-performance concurrency on Java
1.4, regexp processing to support SQL patterns, ... By saying we are not
prepared to incorporate them but instead need our own versions that can
be morphed for client and server we dramatically reduce the
functionality that can be made available to users.

So let me ask this: do our users want more functionality faster by
allowing the use of libraries, or a completely standalone solution with
tight control over the entire implementation?

--
Jeremy

[Jeremy Boynes]

Kathy Saunders wrote:

> Jeremy Boynes wrote:
>
>> I would argue that we are actually making life easier for people
>> implementing simple solutions. To me, a simple environment does not
>> need to cater for multiple versions being concurrently loaded or for
>> multi-classloader operation; it also means being able to select just
>> the functionality you need without having to worry about which jar
>> file a class may have come from.
>>
>> I think for that environment, just adding the component jars to the
>> classpath (without any concern for ordering) is reasonable.
>>
>> To make things even simpler, it has also been proposed that we bundle
>> all components together into one jar (containing everything, client
>> and server). This gives you less flexibility and a larger footprint
>> but is a really simple solution.
>>
>> <snip/>
>>
> I have to say that I don't see how adding more jar files to figure out
> whether you need to deploy and add to your classpath makes things easier
> for the simple case.  

For the simplest case, the proposal was to package everything into one
jar. What you don't get is minimal footprint and the ability to support
multiple versions in the same JVM, but then those aren't really
high-priority features for the simple case.

> And, so far, I don't see what our users would
> reasonably be able to  pick and choose--what would they be able to leave
> out of our database engine other than how the jar files are already
> separated (embedded, network server, tools...)?  
>
> Unless I'm missing
> something, David is currently working on internationalizing error
> messages.  Would it really make sense to tell someone they may not need
> that functionality? Will they be able to get error messages for network
> server without having those classes in their classpath? I could imagine
> scenarios in the future where there may be significant pieces of
> functionality that we would want to separate because not everyone
> wants/needs that functionality and it would significantly add to
> footprint, but I can't think of anything in that category that currently
> exists other than what we already have.  For example, we do have a
> separate jar file for tools.
>
> Footprint is an interesting argument, but will we really see any
> significant differences there yet?  Strictly looking at this from a
> usability perspective, I still believe that having a common.jar file
> which has no real meaning to a Derby user (since I believe you'll always
> need in the network server case at this point), so why have them keep
> track of yet another jar file?
>
> If we do have a separate jar file for these classes, I believe that it
> should only be one at this point and classpath order should not matter.  
> Again, I'm not saying there may not be a need for more jar files in the
> future.  I'm only looking at what I believe is proposed right now.
>

With the current setup, perhaps not - at this time we are only talking
about a few common classes. However, we are talking about an
architecture and philosophy that relies on duplication and isolation
from any libraries, internal or external.

David has raised this issue in conjunction with messages and some common
network functionality. I have raised it previously in conjunction with
the implementation of our DataSources and user-visible API. That this
keeps coming up indicates that there is something here we need to address.

--
Jeremy

[David Van Couvering]

Jeremy Boynes wrote:
[snip]

> Daniel John Debrunner wrote:
>
>> The issue is that today this is fully supported because the client and
>> engine do not share code.
>>
>> Some of the code sharing approaches regress Derby in this area by not
>> supporting this, or require class path ordering for it to be supported.
>>
>
> Some of the others support this by defining compatibility contracts
> and eliminate the need for classpath ordering by not duplicating classes.

Can't you have the situation where common 10.2 and common 10.3 are both
included in the classpath (by accident, as Dan brings up)?  Wouldn't you
end up with order dependencies then?

[snip]

> Let's recharacterize this a little. What we are contemplating with
> code sharing is extracting common functionality out into a library. By
> saying that we are not willing to accept any solution where a
> component depends on a library we are shutting ourselves off from
> using any external library or any functionality not provided by Derby
> itself. This dooms us forever to reinvent any functionality that could
> be provided by other projects.
>
> For example, there are libraries out there that support bytecode
> generation, JMX for management, high-performance concurrency on Java
> 1.4, regexp processing to support SQL patterns, ... By saying we are
> not prepared to incorporate them but instead need our own versions
> that can be morphed for client and server we dramatically reduce the
> functionality that can be made available to users.
>
> So let me ask this: do our users want more functionality faster by
> allowing the use of libraries, or a completely standalone solution
> with tight control over the entire implementation?

You make a compelling argument here.  I already would like to use stuff
in Jakarta Commons (I haven't brought it up yet, one thing at a time).  
It seems a good Apache Java citizen should make use of what's in Jakarta
Commons rather than build stuff themselves.  And I think Jeremy's right
that we will run into this same configuration situation with these guys.

Jeremy, how *do* the users of commons avoid accidentally using a version
they are not compatible with (e.g. a consumer depends on new features
that aren't available in an older version of the common jar file)?

David

>
> --
> Jeremy


[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Jeremy Boynes]

David W. Van Couvering wrote:

>
>
> Jeremy Boynes wrote:
> [snip]
>
>> Daniel John Debrunner wrote:
>>
>>> The issue is that today this is fully supported because the client and
>>> engine do not share code.
>>>
>>> Some of the code sharing approaches regress Derby in this area by not
>>> supporting this, or require class path ordering for it to be supported.
>>>
>>
>> Some of the others support this by defining compatibility contracts
>> and eliminate the need for classpath ordering by not duplicating classes.
>
>
> Can't you have the situation where common 10.2 and common 10.3 are both
> included in the classpath (by accident, as Dan brings up)?  Wouldn't you
> end up with order dependencies then?
>

To what extent do we need to cater for accidents? But, yes if you do end
up with both then the (actual) order would determine which one you ended
up with. In many cases the user would not notice due to the API
compatibility between 10.2 and 10.3 (with 10.3 being a pure superset by
the rules defined). The key thing is that there is no overlap in
functionality between jars (so you are not getting 10.2 (containing
client + common) merged with 10.3 (containing engine + common)).

> [snip]
>
>> Let's recharacterize this a little. What we are contemplating with
>> code sharing is extracting common functionality out into a library. By
>> saying that we are not willing to accept any solution where a
>> component depends on a library we are shutting ourselves off from
>> using any external library or any functionality not provided by Derby
>> itself. This dooms us forever to reinvent any functionality that could
>> be provided by other projects.
>>
>> For example, there are libraries out there that support bytecode
>> generation, JMX for management, high-performance concurrency on Java
>> 1.4, regexp processing to support SQL patterns, ... By saying we are
>> not prepared to incorporate them but instead need our own versions
>> that can be morphed for client and server we dramatically reduce the
>> functionality that can be made available to users.
>>
>> So let me ask this: do our users want more functionality faster by
>> allowing the use of libraries, or a completely standalone solution
>> with tight control over the entire implementation?
>
>
> You make a compelling argument here.  I already would like to use stuff
> in Jakarta Commons (I haven't brought it up yet, one thing at a time).  
> It seems a good Apache Java citizen should make use of what's in Jakarta
> Commons rather than build stuff themselves.  And I think Jeremy's right
> that we will run into this same configuration situation with these guys.
>
> Jeremy, how *do* the users of commons avoid accidentally using a version
> they are not compatible with (e.g. a consumer depends on new features
> that aren't available in an older version of the common jar file)?
>

This can be a problem - Java's variant of DLL hell. The solution
generally relies on two factors:
1) upward compatibility between versions - because they are small-ish
    components then they tend to do one thing and not suffer from API
    scope creep. You just use the latest version and add it to the
    classpath once

2) multi-classloader loading mechanisms, which are commonplace in
    open source projects, even command line utilities like Ant and Maven
    not just the application servers

It is kind of self-fulfilling - when you adopt a modular structure then
you expect to be loading modules with libraries at different version
levels and so use a classloading architecture that fits.

There is also a cultural thing here. For example, suppose I am writing a
database because I happen to think writing database stuff is fun (yes, I
  do realize that doing this for fun makes one appear somewhat odd). I
come up with the idea that converting the query plan to executable code
is a neat idea and so need to generate that code. I can implement a code
generator myself, or look for one out there that is already written and
is available under a suitable license. Given my main interest is in
databases, using someone else's code to generate byte code save me a lot
of effort (especially when the class format changes). If it doesn't do
what I want, I could hack my own but it is generally easier to donate
those changes back so that someone else can maintain them. After all, my
interest is in databases not in byte code generation.

Basically, a volunteer community scales better by incorporating each
other's code rather than trying to do it all.

--
Jeremy

[David Van Couvering]

One thing I'd like to raise is that the code cloning is a build process.
  If in the future we realize we have painted ourselves into a corner,
unless I'm missing something we can just remove the cloning step and we
have the module-based approach that Jeremy and I have been backing.

Right?

David

Jeremy Boynes wrote:

> David W. Van Couvering wrote:
>
>>
>>
>> Jeremy Boynes wrote:
>> [snip]
>>
>>> Daniel John Debrunner wrote:
>>>
>>>> The issue is that today this is fully supported because the client and
>>>> engine do not share code.
>>>>
>>>> Some of the code sharing approaches regress Derby in this area by not
>>>> supporting this, or require class path ordering for it to be supported.
>>>>
>>>
>>> Some of the others support this by defining compatibility contracts
>>> and eliminate the need for classpath ordering by not duplicating
>>> classes.
>>
>>
>>
>> Can't you have the situation where common 10.2 and common 10.3 are
>> both included in the classpath (by accident, as Dan brings up)?  
>> Wouldn't you end up with order dependencies then?
>>
>
> To what extent do we need to cater for accidents? But, yes if you do end
> up with both then the (actual) order would determine which one you ended
> up with. In many cases the user would not notice due to the API
> compatibility between 10.2 and 10.3 (with 10.3 being a pure superset by
> the rules defined). The key thing is that there is no overlap in
> functionality between jars (so you are not getting 10.2 (containing
> client + common) merged with 10.3 (containing engine + common)).
>
>> [snip]
>>
>>> Let's recharacterize this a little. What we are contemplating with
>>> code sharing is extracting common functionality out into a library.
>>> By saying that we are not willing to accept any solution where a
>>> component depends on a library we are shutting ourselves off from
>>> using any external library or any functionality not provided by Derby
>>> itself. This dooms us forever to reinvent any functionality that
>>> could be provided by other projects.
>>>
>>> For example, there are libraries out there that support bytecode
>>> generation, JMX for management, high-performance concurrency on Java
>>> 1.4, regexp processing to support SQL patterns, ... By saying we are
>>> not prepared to incorporate them but instead need our own versions
>>> that can be morphed for client and server we dramatically reduce the
>>> functionality that can be made available to users.
>>>
>>> So let me ask this: do our users want more functionality faster by
>>> allowing the use of libraries, or a completely standalone solution
>>> with tight control over the entire implementation?
>>
>>
>>
>> You make a compelling argument here.  I already would like to use
>> stuff in Jakarta Commons (I haven't brought it up yet, one thing at a
>> time).  It seems a good Apache Java citizen should make use of what's
>> in Jakarta Commons rather than build stuff themselves.  And I think
>> Jeremy's right that we will run into this same configuration situation
>> with these guys.
>>
>> Jeremy, how *do* the users of commons avoid accidentally using a
>> version they are not compatible with (e.g. a consumer depends on new
>> features that aren't available in an older version of the common jar
>> file)?
>>
>
> This can be a problem - Java's variant of DLL hell. The solution
> generally relies on two factors:
> 1) upward compatibility between versions - because they are small-ish
>    components then they tend to do one thing and not suffer from API
>    scope creep. You just use the latest version and add it to the
>    classpath once
>
> 2) multi-classloader loading mechanisms, which are commonplace in
>    open source projects, even command line utilities like Ant and Maven
>    not just the application servers
>
> It is kind of self-fulfilling - when you adopt a modular structure then
> you expect to be loading modules with libraries at different version
> levels and so use a classloading architecture that fits.
>
> There is also a cultural thing here. For example, suppose I am writing a
> database because I happen to think writing database stuff is fun (yes, I
>  do realize that doing this for fun makes one appear somewhat odd). I
> come up with the idea that converting the query plan to executable code
> is a neat idea and so need to generate that code. I can implement a code
> generator myself, or look for one out there that is already written and
> is available under a suitable license. Given my main interest is in
> databases, using someone else's code to generate byte code save me a lot
> of effort (especially when the class format changes). If it doesn't do
> what I want, I could hack my own but it is generally easier to donate
> those changes back so that someone else can maintain them. After all, my
> interest is in databases not in byte code generation.
>
> Basically, a volunteer community scales better by incorporating each
> other's code rather than trying to do it all.
>
> --
> Jeremy

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Kathey Marsden]

Jeremy Boynes wrote:

>
> Let's recharacterize this a little. What we are contemplating with
> code sharing is extracting common functionality out into a library. By
> saying that we are not willing to accept any solution where a
> component depends on a library we are shutting ourselves off from
> using any external library or any functionality not provided by Derby
> itself. This dooms us forever to reinvent any functionality that could
> be provided by other projects.
>
We are not "doomed forever". Requiring a new jar file for new
functionality seems an entirely reasonable thing to me and at that time
we can impose whatever restrictions the community sees fit.  Requiring a
new jar file to have the product continue work, is another matter all
together.

We can deprecate product functionality that we don't like for some
reason, but we can't just up and one day take away something that
deployed products depend upon.   There are products that are out in the
field now which support and encourage use of Derby and expect the
current separation of jars that we have now.  Users should be able to
install  applications that embed future versions of Derby and have
things still work as they do today.  If they don't it's a *regression*.

I was fascinated by David's priority list:  first Product, then User,
then Engineer and think it is really great that we actually seem to have
all permutations of this priority list represented here.   All are
important  and need advocates, but  when you have an installed user
base, sometimes the engineers have to be a little careful and patient.

 Of all the approaches presented thus far, I like.

Approach 3) Wait until we have some new set of  code that  we can use as our first code sharing test case and untangle the existing code later.

This way the big thinking engineers can have fun and do pretty what they want without the installed user base getting in the way.


Kathey

[Daniel John Debrunner]

Jeremy Boynes wrote:

> David W. Van Couvering wrote:

>> Can't you have the situation where common 10.2 and common 10.3 are
>> both included in the classpath (by accident, as Dan brings up)?
>> Wouldn't you end up with order dependencies then?

I feel my scenario keeps being misrepresented by the choice of terms
used to decribe it. Using 'accident' makes it sounds as though it's not
an important problem to deal with, as seen in Jeremy's reponse here:

> To what extent do we need to cater for accidents?

The end user didn't accidently install two applications, they chose to
and didn't realise/know that one used client at version 10.2 and one
used engine at version 10.3. In many cases the use of Derby engine is
hidden by the application developer.

Dan.

[Jeremy Boynes]

Daniel John Debrunner wrote:

> Jeremy Boynes wrote:
>
>
>>David W. Van Couvering wrote:
>
>
>>>Can't you have the situation where common 10.2 and common 10.3 are
>>>both included in the classpath (by accident, as Dan brings up)?
>>>Wouldn't you end up with order dependencies then?
>
>
> I feel my scenario keeps being misrepresented by the choice of terms
> used to decribe it. Using 'accident' makes it sounds as though it's not
> an important problem to deal with, as seen in Jeremy's reponse here:
>
>
>>To what extent do we need to cater for accidents?
>
>
> The end user didn't accidently install two applications, they chose to
> and didn't realise/know that one used client at version 10.2 and one
> used engine at version 10.3. In many cases the use of Derby engine is
> hidden by the application developer.
>

I actually thought "accident" was appropriate here :-) The collision in
Derby versions is "an unexpected and undesirable event, especially one
resulting in damage or harm"[1] arising from the installation of two
applications.

I would assign fault here to the application providers who are not
allowing for other applications using a common shared resource - the
system or user classpath; it's like if they both insisted in being
installed in the same directory.

I would also say that most application providers are aware of this
problem (usually from some unfortunate prior experience) and take
explicit control of the classpath using mechanisms described elsewhere.
Their end-users are unaffected by this scenario.

So whilst there is high impact from your scenario, it applies to
end-users who install multiple applications that do not exhibit
classpath discipline and which happen to use different versions of Derby
client and engine (but not two different engine versions or two
different client versions). And never mind that these users may be
affected by any other libraries these applications use.

Rather than address this ourselves (and just for Derby) by convoluted
code-rewrites and avoidance of third party libraries we should encourage
the application providers avoid this entirely by exhibiting discipline
when using shared resources. We can even point them at something like:
http://classworlds.codehaus.org/

--
Jeremy

[1] http://dictionary.reference.com/search?q=accident

[David Van Couvering]

On thinking further about this, I have a couple more questions/concerns

- I think Jeremy has a very good point that has not been directly
answered.  If we do not allow potential compatibility issues by running
multiple versions in the same VM, then doesn't that mean we are not
going to be willing to incorporate third-party components like from
Jakarta Commons?

I think this is a real problem, and I think a stance like this would not
be looked upon positively from the rest of the Apache community.

- It seems to me that the environments in which mixed versions in the
same VM occur are:

    * app servers, which already deal with this through multiple
classloaders

    * home-built servers in which case we can expect the user to be
expert enough to know how to affect the order in which jars are loaded

- I agree with Jeremy that dealing with shared resources is a well-known
problem and although mistakes can happen good app providers know how to
deal with this.  Is it our responsibility to munge our code and paint
ourselves into a corner regarding use of shared components to deal with
app providers who don't know how to work with this?

Thanks,

David


Jeremy Boynes wrote:

> Daniel John Debrunner wrote:
>
>> Jeremy Boynes wrote:
>>
>>
>>> David W. Van Couvering wrote:
>>
>>
>>
>>>> Can't you have the situation where common 10.2 and common 10.3 are
>>>> both included in the classpath (by accident, as Dan brings up)?
>>>> Wouldn't you end up with order dependencies then?
>>
>>
>>
>> I feel my scenario keeps being misrepresented by the choice of terms
>> used to decribe it. Using 'accident' makes it sounds as though it's not
>> an important problem to deal with, as seen in Jeremy's reponse here:
>>
>>
>>> To what extent do we need to cater for accidents?
>>
>>
>>
>> The end user didn't accidently install two applications, they chose to
>> and didn't realise/know that one used client at version 10.2 and one
>> used engine at version 10.3. In many cases the use of Derby engine is
>> hidden by the application developer.
>>
>
> I actually thought "accident" was appropriate here :-) The collision in
> Derby versions is "an unexpected and undesirable event, especially one
> resulting in damage or harm"[1] arising from the installation of two
> applications.
>
> I would assign fault here to the application providers who are not
> allowing for other applications using a common shared resource - the
> system or user classpath; it's like if they both insisted in being
> installed in the same directory.
>
> I would also say that most application providers are aware of this
> problem (usually from some unfortunate prior experience) and take
> explicit control of the classpath using mechanisms described elsewhere.
> Their end-users are unaffected by this scenario.
>
> So whilst there is high impact from your scenario, it applies to
> end-users who install multiple applications that do not exhibit
> classpath discipline and which happen to use different versions of Derby
> client and engine (but not two different engine versions or two
> different client versions). And never mind that these users may be
> affected by any other libraries these applications use.
>
> Rather than address this ourselves (and just for Derby) by convoluted
> code-rewrites and avoidance of third party libraries we should encourage
> the application providers avoid this entirely by exhibiting discipline
> when using shared resources. We can even point them at something like:
> http://classworlds.codehaus.org/
>
> --
> Jeremy
>
> [1] http://dictionary.reference.com/search?q=accident

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Daniel John Debrunner]

David W. Van Couvering wrote:

> On thinking further about this, I have a couple more questions/concerns
>
> - I think Jeremy has a very good point that has not been directly
> answered.  If we do not allow potential compatibility issues by running
> multiple versions in the same VM, then doesn't that mean we are not
> going to be willing to incorporate third-party components like from
> Jakarta Commons?

No. If we have common code, I don't see how it matters if it's written
as part of Derby or elsewhere. We would take the same approach for both.

> I think this is a real problem, and I think a stance like this would not
> be looked upon positively from the rest of the Apache community.

No-one has suggested such a stance.

But as an aside, I once downloaded an open source project, which then
instructed me to fetch a further five or six common Apache libraries,
either at a specific version numbers or higher than a given version. I
followed all this correctly, set up the classpath and immediately got an
abstract method error.

That's an example of badly managed common versions, and I do not wish
Derby to fall into the same trap.

1) The ease to download goes away in that case where I had to spend time
finding and downloading N additional jars.

2) The ease to use goes away when I have to modify the classpath to add
six or seven jars.

3) The desire to use goes away completely when having followed the
instructions the product fails.

This e-mail isn't to solict ideas on how specifics can be solved, but to
 understand that Derby's ease of use in all aspects is one of its
greatest strengths, we cannot forget that, or lose it.

Dan.

[Jeremy Boynes]

Daniel John Debrunner wrote:

>
> But as an aside, I once downloaded an open source project, which then
> instructed me to fetch a further five or six common Apache libraries,
> either at a specific version numbers or higher than a given version. I
> followed all this correctly, set up the classpath and immediately got an
> abstract method error.
>
> That's an example of badly managed common versions, and I do not wish
> Derby to fall into the same trap.
>
> 1) The ease to download goes away in that case where I had to spend time
> finding and downloading N additional jars.
>
> 2) The ease to use goes away when I have to modify the classpath to add
> six or seven jars.
>
> 3) The desire to use goes away completely when having followed the
> instructions the product fails.
>
> This e-mail isn't to solict ideas on how specifics can be solved, but to
>  understand that Derby's ease of use in all aspects is one of its
> greatest strengths, we cannot forget that, or lose it.
>

And how many open source projects have you downloaded or used without
such issues? Please don't taint this from one bad experience.

For build-time dependencies, many of these issues are addressed by Maven
which automatically downloads the dependencies specified in the project.
  This would be an immediate solution to the problem raised on the
recent junit thread.

For run-time dependencies, these can be addressed with solutions like
uberjar:
1) for Derby as a standalone server, we can have one executable jar that
    loads all dependencies from an internal classloader (no classpath
    issues at all)
2) for Derby as a library, the application embedding us should be using
    the same type of solution and we would fit in there just like
    clogging. With it controlling the classpath there is nothing for the
    user to (mis-)configure

All of these lead to a simple end-user experience - whether they are
trying to build from source, use a standalone server, or embed inside
another application.

--
Jeremy