VOTE: Approach for sharing code

[Jeremy Boynes]

David W. Van Couvering wrote:
> I thought Kathey/Dan's idea of generating copies of the common code
> into two separate directories was interesting and solved a lot of problems,
> and I thought it would be worthwhile to walk through this in a bit more
> detail.
>

This seems like a lot of build time complexity for very little benefit.
We are doing all this pre-processing (in source or, please no, in
binary) just to avoid the need for users to include two jars on the
classpath instead of one.

This is also reinforcing Derby's not-invented-here attitude that
requires implementation of everything rather than using libraries freely
available from other open source projects.

I'd like to crack this discussion wide open and say that we should move
away from the one-jar-to-rule-them model and switch to a model where the
different components inside Derby can be built and versioned
independently and where we are comfortable depending on external
libraries to provide core functionality.

In that context, components that come to mind are engine, client, net,
tools and common and external dependencies for consideration include
logging, configuration and thread management.

I would like to discuss moving to a modular structure where these are
built separately and can be released separately. That gives us the
benfits of simplicity, shared code, and reduced maintenance without the
wierdness of code templates and preprocessing.

For users that want one single jar, we can merge them togther either by
combining jars or through the use of Class-Path references in the
manifests. But that should be a special distribution rather than the norm.

--
Jeremy

[Kathey Marsden]

Jeremy Boynes wrote:

> In that context, components that come to mind are engine, client, net,
> tools and common and external dependencies for consideration include
> logging, configuration and thread management.
>

What type of version  interoperability do you propose for these
components and how would that be managed?

[Jeremy Boynes]

Kathey Marsden wrote:
> Jeremy Boynes wrote:
>
>>In that context, components that come to mind are engine, client, net,
>>tools and common and external dependencies for consideration include
>>logging, configuration and thread management.
>
> What type of version  interoperability do you propose for these
> components and how would that be managed?
>

I think the same as what we have proposed earlier on this thread.

The public interface for any component can only be extended at the same
major.minor level. So 10.3 may have stuff that 10.1 does not have but a
consumer built against 10.1 is guaranteed to work with any 10.X
implementation provided X >= 1. Any incompatible changes require a new
major version and no compatibility is implied.

Implementation can provide version information about themselves and
about the interfaces they support; this might be through the Package
info or through some custom mechanisn. Consumers can query that and
adapt as appropriate.

This applies within a single classloader; if a user needs a more
esoteric scheme within a single JVM then they need to isolate the
versions from each other by using different classloaders. I do not
consider this an unrealistic expectation given this is a non-trivial
case in the first place.

One thing I think we should add is separation between in-VM client (the
JDBC implementation) and the engine implementation. This should allow a
program in one classloader to access an engine in another; this may
involve data copies which would make it less performant than an
in-classloader configuration but it will be better than using a network
connection which is the only option now.

Across JVMs we support up-down compatibility across a wider spread of
versions but that can be done with the wire protocol and does not need
class-level compatibility.

--
Jeremy

[Rick Hillegas-2]

I'm afraid I was too elliptical in my first attempt to describe this
problem. Here's my next attempt:

Suppose that we have a class common.A with encode() and decode()
methods. In 10.1 and 10.2, these methods have identical signatures but
implementations which differ so that the 10.1 decode() does not invert
the 10.2 encode().

Now let's mix a 10.1 client with a 10.2 server in the same VM.

If we don't clone the code, then encode() and decode() will invert one
another regardless of how we wire 10.1 and 10.2 jars together in the
classpath.

However, suppose we clone the code to arrive at client.common.A and
server.common.A. Suppose further that the client jar, reasonably, does
not contain server classes and vice versa. In this case, the client's
decode() will not invert the server's encode(). Somehow, we need to
detect this incompatibility.

The point I am trying to make is this: It seemed to me that code cloning
was put forward as an alternative to the compatiblity checking in
David's first proposal. I don't think cloning solves this problem. It
merely pushes the problem around a bit. We still need run-time
compatibility checking.

Regards,
-Rick

Daniel John Debrunner wrote:

>Rick Hillegas wrote:
>
>  
>
>>I would like to add a few comments to this very useful discussion:
>>    
>>
>
>[snip]
>
>  
>
>>(2) A possible solution was raised: instead of cloning the code in the
>>source tree, clone it during the build process. I have written similar
>>kinds of ant-based code pre-processors and can give some advice if we
>>adopt this approach. Done right, it's a pleasant afternoon's work.
>>However, I have reservations about this approach. It seems to me that it
>>subverts the meaning of "common" code. To me "common" means "behaves the
>>same way." It's not just a matter of preserving APIs that the compiler
>>can check. It comes down to actually doing the same thing internally. As
>>an example, consider a pair of encoding/decoding methods. You want the
>>internal encoding/decoding logic to be the same everywhere. I don't
>>think that build-time cloning saves us from the problem of having to do
>>a run-time compatibility check.
>>    
>>
>
>Not sure what point you are making here, how does copying the code into
>different packages change its behaviour?
>
>With what ever approach, if the client is at version X and the
>engine/network server at version Y then by definition the common code
>will be different. Could be the same api, but maybe bug fixes in Y that
>don't exist in X.
>
>Dan.
>
>
>  
>

[Daniel John Debrunner]

Rick Hillegas wrote:

> I'm afraid I was too elliptical in my first attempt to describe this
> problem. Here's my next attempt:
>
[snip]
>
> The point I am trying to make is this: It seemed to me that code cloning
> was put forward as an alternative to the compatiblity checking in
> David's first proposal. I don't think cloning solves this problem. It
> merely pushes the problem around a bit. We still need run-time
> compatibility checking.

I think you are descibing a different problem, communication between the
client and the server. In that case yes, version compatibility checking
is needed.

But what David is describing with the common code in separate packages
does not need versioning, as the code is self contained within its
domain of engine, client or network server.

You are describing the case where the common code from one domain
creates something that's consumed by common code in another domain (java
object in serialized form, drda protocol etc.).

That style would need versioning even if we didn't have common code.

Dan.

[David Van Couvering]

Well, this is an interesting wrench you have thrown in here, Jeremy.  I
personally have always been a believer in the scheme you suggest, with
modules of code being built independently rather than a single
monolithic code base.  It allows for greater flexibility, but it does
require a lot more discipline in terms of version compatibility.  This
is the way Solaris works, for instance, and they have been quite
successful at it.  But you should see the level of rigor they apply here
at Sun to make sure that works.  Before this level of rigor was applied,
Solaris was a nasty ball of yarn indeed.

Now, we do have a guide for us in this approach.  Jakarta Commons is a
collection of independent modules that are released independently, and
which must meet strong compatibility rules.  Take a look at their
versioning guidelines here:

http://jakarta.apache.org/commons/releases/versioning.html

The question is, what are the benefits of the approach, and do they
merit the rigor that we would have to follow to make sure things don't
break?  I have to think about this myself.  What do others think?

P.S. Jeremy I don't understand why separate classloaders are needed to
work with different versions in the same VM, if we meet compatibility rules.

David

Jeremy Boynes wrote:

> Kathey Marsden wrote:
>
>> Jeremy Boynes wrote:
>>
>>> In that context, components that come to mind are engine, client, net,
>>> tools and common and external dependencies for consideration include
>>> logging, configuration and thread management.
>>
>>
>> What type of version  interoperability do you propose for these
>> components and how would that be managed?
>>
>
> I think the same as what we have proposed earlier on this thread.
>
> The public interface for any component can only be extended at the
> same major.minor level. So 10.3 may have stuff that 10.1 does not have
> but a consumer built against 10.1 is guaranteed to work with any 10.X
> implementation provided X >= 1. Any incompatible changes require a new
> major version and no compatibility is implied.
>
> Implementation can provide version information about themselves and
> about the interfaces they support; this might be through the Package
> info or through some custom mechanisn. Consumers can query that and
> adapt as appropriate.
>
> This applies within a single classloader; if a user needs a more
> esoteric scheme within a single JVM then they need to isolate the
> versions from each other by using different classloaders. I do not
> consider this an unrealistic expectation given this is a non-trivial
> case in the first place.
>
> One thing I think we should add is separation between in-VM client
> (the JDBC implementation) and the engine implementation. This should
> allow a program in one classloader to access an engine in another;
> this may involve data copies which would make it less performant than
> an in-classloader configuration but it will be better than using a
> network connection which is the only option now.
>
> Across JVMs we support up-down compatibility across a wider spread of
> versions but that can be done with the wire protocol and does not need
> class-level compatibility.
>
> --
> Jeremy

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Rick Hillegas-2]

Thanks, Dan. You are right, it's best to separate these issues.

Cheers,
-Rick

Daniel John Debrunner wrote:

>Rick Hillegas wrote:
>
>  
>
>>I'm afraid I was too elliptical in my first attempt to describe this
>>problem. Here's my next attempt:
>>
>>    
>>
>[snip]
>  
>
>>The point I am trying to make is this: It seemed to me that code cloning
>>was put forward as an alternative to the compatiblity checking in
>>David's first proposal. I don't think cloning solves this problem. It
>>merely pushes the problem around a bit. We still need run-time
>>compatibility checking.
>>    
>>
>
>I think you are descibing a different problem, communication between the
>client and the server. In that case yes, version compatibility checking
>is needed.
>
>But what David is describing with the common code in separate packages
>does not need versioning, as the code is self contained within its
>domain of engine, client or network server.
>
>You are describing the case where the common code from one domain
>creates something that's consumed by common code in another domain (java
>object in serialized form, drda protocol etc.).
>
>That style would need versioning even if we didn't have common code.
>
>Dan.
>
>  
>

[Jeremy Boynes]

David W. Van Couvering wrote:
> Well, this is an interesting wrench you have thrown in here, Jeremy.  

Wrenches are a specialty of the house :-)

<snip/>

>
> The question is, what are the benefits of the approach, and do they
> merit the rigor that we would have to follow to make sure things don't
> break?  I have to think about this myself.  What do others think?
>

The big benefit I see is not technical but social. AIUI Jakarta Commons
evolved out of the recognition that large projects (specifically Tomcat)
provided many libraries that would be valuable for other projects to
use. By separating them out it made is possible for people to contribute
in a meaningful way without needing to grok the whole of a large project
in its entirity.

Commons itself went through a process of evolution, both in terms of
stability and in terms of modularity (decoupling the libraries from each
  other). It now provides a useful resource for open source and
commercial projects alike.

In many cases the rigour is not there to the same extent as in typical
closed source implementations. Instead, the use of classloader
hierarchies allows modules to be loaded into separate domains with
multiple versions present in memory concurrently.

This is analogous to the J2EE model where the application server needs
to be able to load different class versions for the different
applications it is hosting. However, it is also used by frameworks like
Spring, Eclipse, Plexus (Maven), Ant etc. in more specialized niches
than a general purpose application server.

> P.S. Jeremy I don't understand why separate classloaders are needed to
> work with different versions in the same VM, if we meet compatibility
> rules.
>

I was thinking of the case where they do not meet compatibility
requirements for co-location, for example a 11.X client talking to a
10.X engine. The in-VM, cross-classloader mode would allow these to work
together without having to use a network connection.

For most co-location situations though I would anticipate that the
compatibility rules combined with the clear separation of classes into
distinct modules would allow the user to assemble an unordered list of
jars that could be loaded by a single classloader.

--
Jeremy

[David Van Couvering]

Well, we're at a bit of a standoff here.  What I'm looking for is a
nail-in-the-coffin data point that would move us in one direction or
the other.

Here are my feelings on the issue.  Although the cloning approach
helps us avoid having to maintain compatibility, my gut feel is that
it is a hack and it will come back to bite us in the future.  I also
think it's not extensible -- it's a particular solution to a particular
problem of sharing code between client and engine.  What if we want to
share code in other ways in the future?  We'd have to keep trying to
extend this model, making more and more copies.   I also don't like the
debugging model of the cloning approach, where what you're looking at
and debugging is not the "real" code but "shadow" code.  It just overall
feels like a bandaid type of approach; when I've gone for these types
of solutions in the past I've almost always regretted it (or somebody
else who had to deal with it four years later).

The module principle as Jeremy describes it, is a well-understood approach,
it's already in use within Jakarta commons, and has been used to great
effect in Solaris.  It's extensible, and it has been shown to be effective
in allowing a project to be more flexible and scale up better as a project
grows in size and complexity.  It's also in line with the approach we
already have in place with the engine code, with the Monitor and Service
pattern.

That said, I recognize that it requires a certain level of discipline
around version compatibility to make this scheme work.  For this reason,
I think we should be careful how many "modules" we create.
I don't think it helps to take every subsystem and turn it into a module
unless there is a need for it to be used in multiple places.  For now I
would like to propose we stick with just the "common module" as the only
one; if necessary we can break this into smaller pieces in the future,
for example splitting out the DRDA code, but so far I don't see a need
for that.

So, unless someone can show me a nail that puts the module approach in
the grave, this is the approach I'd like to take.

Thanks,

David


Jeremy Boynes wrote:

> Kathey Marsden wrote:
>
>> Jeremy Boynes wrote:
>>
>>> In that context, components that come to mind are engine, client, net,
>>> tools and common and external dependencies for consideration include
>>> logging, configuration and thread management.
>>
>>
>> What type of version  interoperability do you propose for these
>> components and how would that be managed?
>>
>
> I think the same as what we have proposed earlier on this thread.
>
> The public interface for any component can only be extended at the
> same major.minor level. So 10.3 may have stuff that 10.1 does not have
> but a consumer built against 10.1 is guaranteed to work with any 10.X
> implementation provided X >= 1. Any incompatible changes require a new
> major version and no compatibility is implied.
>
> Implementation can provide version information about themselves and
> about the interfaces they support; this might be through the Package
> info or through some custom mechanisn. Consumers can query that and
> adapt as appropriate.
>
> This applies within a single classloader; if a user needs a more
> esoteric scheme within a single JVM then they need to isolate the
> versions from each other by using different classloaders. I do not
> consider this an unrealistic expectation given this is a non-trivial
> case in the first place.
>
> One thing I think we should add is separation between in-VM client
> (the JDBC implementation) and the engine implementation. This should
> allow a program in one classloader to access an engine in another;
> this may involve data copies which would make it less performant than
> an in-classloader configuration but it will be better than using a
> network connection which is the only option now.
>
> Across JVMs we support up-down compatibility across a wider spread of
> versions but that can be done with the wire protocol and does not need
> class-level compatibility.
>
> --
> Jeremy

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[francois.orsini]

Copying common classes to specific (tier) packages does sound good - However, it is true that there could be confusion during development and debugging if the engine and client common classes are not up to date (did not get re-generated) and if some developers do not necessarily understand how the 'common' package is managed in the codeline.  I guess this is true for some other packages we have but this one is going to be fairly visible due to the fact we are sharing code functionality here between 2 tiers.

Due to the requirement of supporting a different version of the derby client and server in the same JVM, it's not like there are a lot of other and simpler alternatives out there indeed...Am guessing we're not going to support the loading of 2 different DerbyClient versions in the same JVM...(although one could envision db2jcc.jar and DerbyClient.jar running in the same JVM...we should not be seeing any conflict if the package structure is different)

--francois

[Jeremy Boynes]

Francois Orsini wrote:
<snip/>

> Due to the requirement of supporting a different version of the derby client
> and server in the same JVM, it's not like there are a lot of other and
> simpler alternatives out there indeed...Am guessing we're not going to
> support the loading of 2 different DerbyClient versions in the same
> JVM...(although one could envision db2jcc.jar and DerbyClient.jar running in
> the same JVM...we should not be seeing any conflict if the package structure
> is different)

I see no reason why we should not support that assuming the
infrastructure allows them to be loaded into different classloaders. For
example, each could be packaged inside a different RAR file and deployed
to an application server. A use case for this is different applications
talking to different databases.

Similarly I think we should also allow multiple versions of the engine
to be loaded as well (although not accessing the same data files).
Support for this though requires evolution away from the reliance on
system properties.

The use case here is isolation between the engines - for example, if an
application server is being used to host applications from different
organizations.

--
Jeremy

[francois.orsini]

Yes many things are possible technically - I did not mention the classloader thing because 1) it's very likely not going to be the most common use case (80-20 rule), 2) I think the original idea is to come up with a simple solution to the issue of code sharing between the client and the derby engine - if people want to come up with sophisticated frameworks to allow loading of specific classpath/archives, then it is fine; but I don't think we should make this the default case/solution - to me this belongs to the real of application server framework, not the database one which is Derby's main focus. This is something that can be done even today outside of Derby.

--francois

On 9/12/05, Jeremy Boynes <jboynes@...> wrote:

Francois Orsini wrote:
<snip/>

> Due to the requirement of supporting a different version of the derby client
> and server in the same JVM, it's not like there are a lot of other and
> simpler alternatives out there indeed...Am guessing we're not going to
> support the loading of 2 different DerbyClient versions in the same
> JVM...(although one could envision db2jcc.jar and DerbyClient.jar running in
> the same JVM...we should not be seeing any conflict if the package structure
> is different)

I see no reason why we should not support that assuming the
infrastructure allows them to be loaded into different classloaders. For
example, each could be packaged inside a different RAR file and deployed
to an application server. A use case for this is different applications
talking to different databases.

Similarly I think we should also allow multiple versions of the engine
to be loaded as well (although not accessing the same data files).
Support for this though requires evolution away from the reliance on
system properties.

The use case here is isolation between the engines - for example, if an
application server is being used to host applications from different
organizations.

--
Jeremy

[Kathy Saunders]

David W. Van Couvering wrote:

> Well, we're at a bit of a standoff here.  What I'm looking for is a
> nail-in-the-coffin data point that would move us in one direction or
> the other.

I've been following this discussion with some interest as my background
is technical support.  In fact, I supported Cloudscape for the first 4
years (from release 1.0 of Cloudscape).  My experience is that
developers (particularly Java developers) really liked Cloudscape (now
Derby) because it was so easy to use and deploy.  And, I found
historically that one of the most common issues to come up were
classpath issues (in particular we got in trouble a few times when we
introduced something that caused the order of the jar files to be
important).  You should note that I'm not, and never have been, a Derby
developer, so I don't claim to be an expert on what's correct and best
from a development perspective.

I'm a bit concerned because I see a lot of discussion about what is good
from a derby development perspective, but not so much how these changes
may affect users of Derby.  Although some Derby users have complex
applications (like application servers), many are implementing much more
simple solutions.

Having said that, I'm a bit lost in what is being proposed from the
user/functional perspective.  David, as soon as you have a more concrete
proposal (may not be the time yet), can you post that information?  
Could you  provide information on what the users of Derby will have to
do with this change (how would our documentation need to be changed) and
maybe footprint, performance, etc. impacts vs. the benefits  from making
this change.  I'd like to be able to provide my input from a
usability/documentation perspective.

In addition, I work on Derby now in the testing area, so I'd also like
to understand the implications for what additional testing might need to
be done.  If we create more jar files, is there more testing
requirements for different combinations?

Thanks,
Kathy

[Daniel John Debrunner]

Francois Orsini wrote:

> Copying common classes to specific (tier) packages does sound good -
> However, it is true that there could be confusion during development and
> debugging if the engine and client common classes are not up to date
> (did not get re-generated) and if some developers do not necessarily
> understand how the 'common' package is managed in the codeline.

Sun seems to be thinking about this approach for J2SE, namely changing
the package names of the Apache parser into the com.sun domain to avoid
version confusion. :-)

Here's an FAQ from
http://java.sun.com/xml/jaxp/faq.html

<quote>

Q. Why are there Apache classes in the J2SE 1.4 RI?

The J2SE 1.4 RI is the first version of the JDK that bundles in an
implementation of JAXP 1.1. This allows developers to write applications
without having to provide a parser and XSLT processor with their
application. However, in some cases, it may create additional problems.

The Sun J2SE 1.4 RI uses Apache software for its implemenation of JAXP
1.1 with package names unchanged from Apache software distributions.
This can cause problems, for example, if your application wants to use a
newer version of Apache software. Under the Java 2 class loader
delegation model, the java launcher's ClassLoader will load the bundled
version of a class before any other version. Thus, if you place a newer
version of xalan.jar on your CLASSPATH, then that version will be
ignored since the runtime will use the older bundled version instead. As
a workaround, see the question on overriding the implementation in J2SE
SDK 1.4.

The future plan is to rename the org.apache.** packages to be something
like com.sun.org.apache.** to fix this problem. In addition, other
package-dependent parts of the software may also need to be modified.
However, because of lack of resources, this may not be done until after
J2SE SDK 1.4.1.

</quote>

Dan.

[David Van Couvering]

Hi, Kathy, thanks for your email.  The timing is actually pretty good, I
was just talking with Francois trying to understand his concerns better.

After quickly describing the two approaches, I'd like to summarize the
experience/impact of these approaches from the perspectives of the end
user, the developer/maintainer, and the test developer/runner.

Goal:
  - Reduce code duplication while continuing to support different
versions of client and embedded driver in the same VM

Approach 1:
  - Create a common package and put all common code there
(org.apache.derby.common)
  - Use compatibility guidelines to ensure backward compatibility and
some degree of forward compatibility
  - Common classes are embedded in derby.jar and derby-client.jar (based
on lots of negative feedback for having a separate jar)

Approach 2:
  - Create a common package and put all common code there
(org.apache.derby.common)
  - During build process, "clone" all common classes into at least two
generated packages, one for the engine and one for the network client
(org.apache.derby.engine.common and org.apache.derby.client.common).
  - This approach avoids having to implement compatiblity
guidelines/constraints and guarantees, as the engine and client continue
to be fully isolated

USER EXPERIENCE

Approach 1
  - No new jar files, everything looks the same
  - For the vast percentage of users who don't mix versions in the same
VM, everything works great
  - For the small percentage of users who mix versions, the order in
which jar files are loaded.  For example, if they want to use
derby-10.2.jar and derby-client-11.0.jar, then if derby-10.2.jar is
first in the classpath, they will get an exception saying that the
client code is incompatible with the common code and that they need to
re-order their jar files, whereas it will work fine if the order is
reversed.

Approach 2
  - No new jar files, everything looks the same
  - Ordering of jar files does not matter, everything works fine


DEVELOPER EXPERIENCE

Approach 1
  - Pretty much as it is today, nothing surprising

Approach 2
  - When navigating code either during source browsing or debugging,
they will see the *generated* common code, not the master common code.
  - If a developer is not aware of how things work, or just forgets,
he/she will try to edit this generated code, and will be confused and
surprised when his/her changes disappear after a build
  - Stack traces will point to generated code, and the developer will
have to remember to translate that back to the master version.
  - The generation process must be very careful not to adjust line
numbers or all stack traces will be off and misleading.  This means for
instance we can't add comments saying "THIS IS GENERATED CODE, DO NOT EDIT"
  - We may need to generate more copies if different types of version
mixing are required (e.g. if the tools.jar and derby.jar need to be at
different versions)


TESTER EXPERIENCE
Approach 1
  - We would have to build and run compatibility tests to make sure that
compatible versions run correctly and incompatible ones correcty raise
the exception

Approach 2
  - No real change, although debugging of tests may be confusing due to
issues I already listed under developer experience


I also am uncomfortable with the "twistiness" of approach 2.  There is
something to be said for a clean architecture and build environment.  I
have seen time and again that a good architecture allows you to scale
and grow, whereas "twisty" architectures tend to wrap you up and tie you
down at some point.  I think this has to be taken into consideration.

My main question is: is it OK to sometimes throw an exception for the
small set of users who mix versions, and for them to then have to
rearrange their jar ordering.  If the answer is that this is not
acceptable and would be considered a showstopper regression for some
part of our user base, then I think we have no choice but to go with
Approach 2, even if we do risk painting ourselves into an architectural
corner.  Otherwise, it is my strong recommendation to go with Approach 1.

Thanks!

David

Kathy Saunders wrote:

> David W. Van Couvering wrote:
>
>> Well, we're at a bit of a standoff here.  What I'm looking for is a
>> nail-in-the-coffin data point that would move us in one direction or
>> the other.
>
>
> I've been following this discussion with some interest as my
> background is technical support.  In fact, I supported Cloudscape for
> the first 4 years (from release 1.0 of Cloudscape).  My experience is
> that developers (particularly Java developers) really liked Cloudscape
> (now Derby) because it was so easy to use and deploy.  And, I found
> historically that one of the most common issues to come up were
> classpath issues (in particular we got in trouble a few times when we
> introduced something that caused the order of the jar files to be
> important).  You should note that I'm not, and never have been, a
> Derby developer, so I don't claim to be an expert on what's correct
> and best from a development perspective.
>
> I'm a bit concerned because I see a lot of discussion about what is
> good from a derby development perspective, but not so much how these
> changes may affect users of Derby.  Although some Derby users have
> complex applications (like application servers), many are implementing
> much more simple solutions.
> Having said that, I'm a bit lost in what is being proposed from the
> user/functional perspective.  David, as soon as you have a more
> concrete proposal (may not be the time yet), can you post that
> information?  Could you  provide information on what the users of
> Derby will have to do with this change (how would our documentation
> need to be changed) and maybe footprint, performance, etc. impacts vs.
> the benefits  from making this change.  I'd like to be able to provide
> my input from a usability/documentation perspective.
>
> In addition, I work on Derby now in the testing area, so I'd also like
> to understand the implications for what additional testing might need
> to be done.  If we create more jar files, is there more testing
> requirements for different combinations?
>
> Thanks,
> Kathy
>

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[David Van Couvering]

Oooh, you are good! :)

David

Daniel John Debrunner wrote:

>Francois Orsini wrote:
>
>  
>
>>Copying common classes to specific (tier) packages does sound good -
>>However, it is true that there could be confusion during development and
>>debugging if the engine and client common classes are not up to date
>>(did not get re-generated) and if some developers do not necessarily
>>understand how the 'common' package is managed in the codeline.
>>    
>>
>
>Sun seems to be thinking about this approach for J2SE, namely changing
>the package names of the Apache parser into the com.sun domain to avoid
>version confusion. :-)
>
>Here's an FAQ from
>http://java.sun.com/xml/jaxp/faq.html
>
><quote>
>
>Q. Why are there Apache classes in the J2SE 1.4 RI?
>
>The J2SE 1.4 RI is the first version of the JDK that bundles in an
>implementation of JAXP 1.1. This allows developers to write applications
>without having to provide a parser and XSLT processor with their
>application. However, in some cases, it may create additional problems.
>
>The Sun J2SE 1.4 RI uses Apache software for its implemenation of JAXP
>1.1 with package names unchanged from Apache software distributions.
>This can cause problems, for example, if your application wants to use a
>newer version of Apache software. Under the Java 2 class loader
>delegation model, the java launcher's ClassLoader will load the bundled
>version of a class before any other version. Thus, if you place a newer
>version of xalan.jar on your CLASSPATH, then that version will be
>ignored since the runtime will use the older bundled version instead. As
>a workaround, see the question on overriding the implementation in J2SE
>SDK 1.4.
>
>The future plan is to rename the org.apache.** packages to be something
>like com.sun.org.apache.** to fix this problem. In addition, other
>package-dependent parts of the software may also need to be modified.
>However, because of lack of resources, this may not be done until after
>J2SE SDK 1.4.1.
>
></quote>
>
>Dan.
>
>  
>

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[David Van Couvering]

OK, smileys aside, I think this case is different because they are
dealing with (a) third-party code that may be upgraded independently
and  (b) something that is bundled into J2SE.  We aren't bundling into
J2SE and we won't allow independent upgrades of our common code.  So I
am not convinced we have to go through this exercise.  Also, the mixed
version issue is an *edge case* with us, whereas for them it's likely a
very common case.

Thanks,

David

David W. Van Couvering wrote:

> Oooh, you are good! :)
>
> David
>
> Daniel John Debrunner wrote:
>
>> Francois Orsini wrote:
>>
>>  
>>
>>> Copying common classes to specific (tier) packages does sound good -
>>> However, it is true that there could be confusion during development
>>> and
>>> debugging if the engine and client common classes are not up to date
>>> (did not get re-generated) and if some developers do not necessarily
>>> understand how the 'common' package is managed in the codeline.
>>>  
>>
>>
>> Sun seems to be thinking about this approach for J2SE, namely changing
>> the package names of the Apache parser into the com.sun domain to avoid
>> version confusion. :-)
>>
>> Here's an FAQ from
>> http://java.sun.com/xml/jaxp/faq.html
>>
>> <quote>
>>
>> Q. Why are there Apache classes in the J2SE 1.4 RI?
>>
>> The J2SE 1.4 RI is the first version of the JDK that bundles in an
>> implementation of JAXP 1.1. This allows developers to write applications
>> without having to provide a parser and XSLT processor with their
>> application. However, in some cases, it may create additional problems.
>>
>> The Sun J2SE 1.4 RI uses Apache software for its implemenation of JAXP
>> 1.1 with package names unchanged from Apache software distributions.
>> This can cause problems, for example, if your application wants to use a
>> newer version of Apache software. Under the Java 2 class loader
>> delegation model, the java launcher's ClassLoader will load the bundled
>> version of a class before any other version. Thus, if you place a newer
>> version of xalan.jar on your CLASSPATH, then that version will be
>> ignored since the runtime will use the older bundled version instead. As
>> a workaround, see the question on overriding the implementation in J2SE
>> SDK 1.4.
>>
>> The future plan is to rename the org.apache.** packages to be something
>> like com.sun.org.apache.** to fix this problem. In addition, other
>> package-dependent parts of the software may also need to be modified.
>> However, because of lack of resources, this may not be done until after
>> J2SE SDK 1.4.1.
>>
>> </quote>
>>
>> Dan.
>>
>>  
>>

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[djencks]

Admittedly I approach this mostly from an app server perspective, but I
prefer

--Create a common package and put all common code there
--Make a separate jar with this stuff in it
--Leave the common classes out of derby.jar and derby-client.jar
--For users convenience in standalone apps produce an additional jar
that includes everything from common, derby, derby-client, network,
drda, etc etc.  This would not be used if you are concerned about
classpaths and versions or the smallest possible footprint.
--Make sure that all the jars have clear version numbers in their names.

I think you are asking for major classloader trouble including the same
classes in 2 different jars (doesn't this produce a sealing or signing
problem?) and I think the package-name munging approach is really ugly
and twisted.

thanks
david jencks

On Sep 13, 2005, at 3:34 PM, David W. Van Couvering wrote:

> Hi, Kathy, thanks for your email.  The timing is actually pretty good,
> I was just talking with Francois trying to understand his concerns
> better.
>
> After quickly describing the two approaches, I'd like to summarize the
> experience/impact of these approaches from the perspectives of the end
> user, the developer/maintainer, and the test developer/runner.
>
> Goal:
>  - Reduce code duplication while continuing to support different
> versions of client and embedded driver in the same VM
>
> Approach 1:
>  - Create a common package and put all common code there
> (org.apache.derby.common)
>  - Use compatibility guidelines to ensure backward compatibility and
> some degree of forward compatibility
>  - Common classes are embedded in derby.jar and derby-client.jar
> (based on lots of negative feedback for having a separate jar)
>
> Approach 2:
>  - Create a common package and put all common code there
> (org.apache.derby.common)
>  - During build process, "clone" all common classes into at least two
> generated packages, one for the engine and one for the network client
> (org.apache.derby.engine.common and org.apache.derby.client.common).  
> - This approach avoids having to implement compatiblity
> guidelines/constraints and guarantees, as the engine and client
> continue to be fully isolated
>
> USER EXPERIENCE
>
> Approach 1
>  - No new jar files, everything looks the same
>  - For the vast percentage of users who don't mix versions in the same
> VM, everything works great
>  - For the small percentage of users who mix versions, the order in
> which jar files are loaded.  For example, if they want to use
> derby-10.2.jar and derby-client-11.0.jar, then if derby-10.2.jar is
> first in the classpath, they will get an exception saying that the
> client code is incompatible with the common code and that they need to
> re-order their jar files, whereas it will work fine if the order is
> reversed.
>
> Approach 2
>  - No new jar files, everything looks the same
>  - Ordering of jar files does not matter, everything works fine
>
>
> DEVELOPER EXPERIENCE
>
> Approach 1
>  - Pretty much as it is today, nothing surprising
>
> Approach 2
>  - When navigating code either during source browsing or debugging,
> they will see the *generated* common code, not the master common code.
>  - If a developer is not aware of how things work, or just forgets,
> he/she will try to edit this generated code, and will be confused and
> surprised when his/her changes disappear after a build
>  - Stack traces will point to generated code, and the developer will
> have to remember to translate that back to the master version.
>  - The generation process must be very careful not to adjust line
> numbers or all stack traces will be off and misleading.  This means
> for instance we can't add comments saying "THIS IS GENERATED CODE, DO
> NOT EDIT"
>  - We may need to generate more copies if different types of version
> mixing are required (e.g. if the tools.jar and derby.jar need to be at
> different versions)
>
>
> TESTER EXPERIENCE
> Approach 1
>  - We would have to build and run compatibility tests to make sure
> that compatible versions run correctly and incompatible ones correcty
> raise the exception
>
> Approach 2
>  - No real change, although debugging of tests may be confusing due to
> issues I already listed under developer experience
>
>
> I also am uncomfortable with the "twistiness" of approach 2.  There is
> something to be said for a clean architecture and build environment.  
> I have seen time and again that a good architecture allows you to
> scale and grow, whereas "twisty" architectures tend to wrap you up and
> tie you down at some point.  I think this has to be taken into
> consideration.
>
> My main question is: is it OK to sometimes throw an exception for the
> small set of users who mix versions, and for them to then have to
> rearrange their jar ordering.  If the answer is that this is not
> acceptable and would be considered a showstopper regression for some
> part of our user base, then I think we have no choice but to go with
> Approach 2, even if we do risk painting ourselves into an
> architectural corner.  Otherwise, it is my strong recommendation to go
> with Approach 1.
>
> Thanks!
>
> David
>
> Kathy Saunders wrote:
>
>> David W. Van Couvering wrote:
>>
>>> Well, we're at a bit of a standoff here.  What I'm looking for is a
>>> nail-in-the-coffin data point that would move us in one direction or
>>> the other.
>>
>>
>> I've been following this discussion with some interest as my
>> background is technical support.  In fact, I supported Cloudscape for
>> the first 4 years (from release 1.0 of Cloudscape).  My experience is
>> that developers (particularly Java developers) really liked
>> Cloudscape (now Derby) because it was so easy to use and deploy.  
>> And, I found historically that one of the most common issues to come
>> up were classpath issues (in particular we got in trouble a few times
>> when we introduced something that caused the order of the jar files
>> to be important).  You should note that I'm not, and never have been,
>> a Derby developer, so I don't claim to be an expert on what's correct
>> and best from a development perspective.
>>
>> I'm a bit concerned because I see a lot of discussion about what is
>> good from a derby development perspective, but not so much how these
>> changes may affect users of Derby.  Although some Derby users have
>> complex applications (like application servers), many are
>> implementing much more simple solutions.
>> Having said that, I'm a bit lost in what is being proposed from the
>> user/functional perspective.  David, as soon as you have a more
>> concrete proposal (may not be the time yet), can you post that
>> information?  Could you  provide information on what the users of
>> Derby will have to do with this change (how would our documentation
>> need to be changed) and maybe footprint, performance, etc. impacts
>> vs. the benefits  from making this change.  I'd like to be able to
>> provide my input from a usability/documentation perspective.
>>
>> In addition, I work on Derby now in the testing area, so I'd also
>> like to understand the implications for what additional testing might
>> need to be done.  If we create more jar files, is there more testing
>> requirements for different combinations?
>>
>> Thanks,
>> Kathy
>>
> <david.vancouvering.vcf>

[Jeremy Boynes]

Kathy Saunders wrote:
>

<snip/>

>
> I'm a bit concerned because I see a lot of discussion about what is good
> from a derby development perspective, but not so much how these changes
> may affect users of Derby.  Although some Derby users have complex
> applications (like application servers), many are implementing much more
> simple solutions.

I would argue that we are actually making life easier for people
implementing simple solutions. To me, a simple environment does not need
to cater for multiple versions being concurrently loaded or for
multi-classloader operation; it also means being able to select just the
functionality you need without having to worry about which jar file a
class may have come from.

I think for that environment, just adding the component jars to the
classpath (without any concern for ordering) is reasonable.

To make things even simpler, it has also been proposed that we bundle
all components together into one jar (containing everything, client and
server). This gives you less flexibility and a larger footprint but is a
really simple solution.

<snip/>

>
> In addition, I work on Derby now in the testing area, so I'd also like
> to understand the implications for what additional testing might need to
> be done.  If we create more jar files, is there more testing
> requirements for different combinations?
>

I don't think there are any more combinations - in fact probably less as
you would not need to test all possible classpath orderings. We are
dealing with the same amount of code, just modularizing its structure.

By modularizing the build we also allow for in-depth testing on each
individual component in isolation. With a clear definition of the API
contract for each component and testing (unit, functional,
compatibility) of that contract we can perform more thorough testing on
each one before integrating into a whole. Integration and system testing
can focus on the interfaces between components rather than on the entire
black box.

Add in too that modularization makes it easier for users and developers
to come up to speed with the design and implementation of that
component. More eyes on the code with comprehensible component leads to
better review and higher quality.

Finally, you can see this pattern at work with many open source
projects: a common core and then a very modular structure that allows
people to participate at the component level. Examples of projects with
this type of structure are:
* Apache HTTPD + mod_*
* Apache Maven + plugins
* Eclipse + plugins
* Apache Jakarta/Tomcat + Commons
and many more.

--
Jeremy

[Kathey Marsden]

David W. Van Couvering wrote:

> Hi, Kathy, thanks for your email.  The timing is actually pretty good,
> I was just talking with Francois trying to understand his concerns
> better.
>
> After quickly describing the two approaches, I'd like to summarize the
> experience/impact of these approaches from the perspectives of the end
> user, the developer/maintainer, and the test developer/runner.
>

Thank you David for the summary.  I thought "modular build"  meant
adding more jars,  so it was good to have that cleared  With Approach 1
I personally am not too keen on the new ordering requirements,  testing
requirements, or the potential for regression with the versioning scheme.

For the USER EXPERIENCE  for either approach, how much  growth do you
anticipate in the client due to code sharing  with the first round of
what you want to share and a guesstimate of how big it might get if we
utilize all that you think the client should use from  the engine?  
The earlier thread on the size of the common jar file
http://mail-archives.apache.org/mod_mbox/db-derby-dev/200507.mbox/%3c42DF40D5.8050605@...%3e
made it sound significant.

Kathey