VOTE: Approach for sharing code

[David Van Couvering]

I added a comment to DERBY-289
(http://issues.apache.org/jira/browse/DERBY-289) that is my official
proposal for an approach to sharing code.

I am including it in this email as well for easy reference.

Please vote and/or comment on this proposal.

Thanks,

David

=====

PROPOSAL FOR SHARING CODE IN DERBY

Here are the outlines of how I plan to share code in Derby. This first
version is a high-level description of the approach. After incorporating
your feedback and piloting with the internationalization code, I will write
up a more detailed proposal.


PRINCIPLES AND REQUIREMENTS

It's important to understand and agree upon the principles and requirements
of this feature

- Allow sharing of common code across all parts of the Derby codeline,
  in particular between the network client and the engine.
 
- Make it easy to code agains common components (avoid onerous overhead)

- Support the following binary compatibility rules. This is based
  off of the compatibility rules defined for the Apache Portable
  Runtime (APR) project (see http://apr.apache.org/versioning.html)
<http://apr.apache.org/versioning.html%29>:

   * compatibility guaranteed against later versions until
     the major version number changes (e.g. a 10.1 consumer will
     work with with 10.2 common classes, but a 10.2 consumer is
     not guaranteed to work with 11.0 common classes).

   * compatibility guaranteed for all previous patch versions
     (e.g. a 10.1.2 consumer will work with 10.1.1 common classes).
 
   * compatibility will be strongly encouraged but not guaranteed
     against previous minor versions (e.g. a 10.2 consumer works
     with 10.1 common classes, but a 10.3 consumer has a hard
     dependency on new methods, it can not work with 10.2
     common classes).

   * no expectation for compatibility for previous major versions
     (e.g. 10.1 is not guaranteed to work with 11.0).


- Support for both direct creation of classes and pluggable infrastructure
  where this is needed. Some common classes are very simple, and not
  everything should require a pluggable infrastructure.




IMPLEMENTATION PLAN

- Create a new directory java/common

- Create a new package directory under common, org/apache/derby/common

- Classes created within the common package hierarchy needs to follow the
  guidelines for common components as described below

- Provide a CommonVersion helper class (described further below)
  that allows consumers to detect what version of the common components
  is available and determine whether the version is compatible.

- Modify the build script to create a new jar file, derby-common.jar,
  which contains all the common classes.

- Update the documentation to describe the need for this new jar file
  in the classpath/ext directory/etc.
 
   
COMMON CLASS CODING GUIDELINES

These guidelines are for any classes in the common package with
public methods. These guidelines should actually be applied to major
releases as well, as much as possible, but compatibility is not
required between major releases.

Compatibility will be tested with a new suite of compatibility
tests (currently being devised by Rick Hillegas)

- Class names and package names should not change between patch or
  minor releases.

- Existing public method names, method signatures, and public field
  names should not change between patch or minor releases.

- Existing public fields or methods should not be removed between patch
  or minor releases.

- Although the code of existing methods can change, the defined behavior,
  in terms of semantics and observable effects, should not change
  between patch and minor releases.

- New classes, packages, public fields and methods can be added between
  minor releases, but not between patch releases, as this would break
  forward-compatibility for patch releases.


VERSION DETECTION

The Version class will be defined as a common class and is used to
define a version and check for compatibility between a consumer and
the common package. Here is a first pass at the methods on this class.
 
public class Version
{
  /**
    * return the String representation of the version of the
    * common package
    */
  static Version getCommonVersion()

  /**
    * Create a version object. For 10.1.2 you would say
    * <code>new Version(10, 1, 2)</code>
    */
  public Version(int major, int minor, int patch)

  int getMajor()

  int getMinor()

  int getPatch()

  /**
   * Check to see if a consumer is compatible with the common package.
   * This method should be called when the consumer is being initialized,
   * and an exception should be thrown if they are not compatible.
   *
   * If the consumer version is older than or the same as the common
   * package version, we have all the knowledge we need to determine
   * compatibility.
   *
   * If the consumer is newer than the common package, then we need
   * to check for forward compatibility (e.g. to see if the older common
   * package can work with the newer consumer). If only the patch
   * versions differ, then by our compatibility rules the two versions
   * must be compatible. If the major or minor versions differ then we
   * use the firstIncompatibleVersion parameter to determine forward
   * compatibility. If firstIncompatibleVersion is null then we
   * assume full forward compatibility (the consumer is saying "I can
   * work with anything").
   *
   *
   * Usage example:
   *
   * checkCommonCompatibility(new Version(10, 1, 2), null);
   * checkCommonCompatibility(new Version(11, 0, 1), new Version(10, 9, 0)
   */
  boolean checkCommonCompatibility(Version consumer,
    Version firstIncompatibleVersion)
}


[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Daniel John Debrunner]

David W. Van Couvering wrote:

> I added a comment to DERBY-289
> (http://issues.apache.org/jira/browse/DERBY-289) that is my official
> proposal for an approach to sharing code.
>
> I am including it in this email as well for easy reference.
>
> Please vote and/or comment on this proposal.
>

I'm a little confused on the versioning. Is the version number for a
common component in-line with the Derby version or independent? I was
assuming independent, but it seems from your examples that the versions
are in-line with the Derby version.

As an example, I imagine code for localization would be fairly static,
and the initial version may well remain unchanged until several Derby
releases from now (if ever). Thus I wouldn't expect the version number
of such common code to change on every Derby release.

Are you saying that all common code is at a single version, rather than
versions within the common code. Eg. I was imagining a version for
i18n/l10n code, another for say drda encoding, etc.

Dan.

[David Van Couvering]

I was assuming that there was one version for all common code.  I am
hesitant to version separately each common component.  It seems like
that's a lot of overhead for component users -- they would have to track
version compatibility for each component independently.

It seems to me that since the common classes are deployed as a unit,
what you really want to know is if the network client or engine can work
with the common components/classes as a whole.  It's not very useful if
say the 10.2 network client can work with the 10.1 i18n code but not the
10.1 DRDA encoding.  It's kind of all-or-nothing.

Thanks,

David

Daniel John Debrunner wrote:

>David W. Van Couvering wrote:
>
>  
>
>>I added a comment to DERBY-289
>>(http://issues.apache.org/jira/browse/DERBY-289) that is my official
>>proposal for an approach to sharing code.
>>
>>I am including it in this email as well for easy reference.
>>
>>Please vote and/or comment on this proposal.
>>
>>    
>>
>
>I'm a little confused on the versioning. Is the version number for a
>common component in-line with the Derby version or independent? I was
>assuming independent, but it seems from your examples that the versions
>are in-line with the Derby version.
>
>As an example, I imagine code for localization would be fairly static,
>and the initial version may well remain unchanged until several Derby
>releases from now (if ever). Thus I wouldn't expect the version number
>of such common code to change on every Derby release.
>
>Are you saying that all common code is at a single version, rather than
>versions within the common code. Eg. I was imagining a version for
>i18n/l10n code, another for say drda encoding, etc.
>
>Dan.
>
>
>  
>

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Daniel John Debrunner]

David W. Van Couvering wrote:

> I was assuming that there was one version for all common code.  I am
> hesitant to version separately each common component.  It seems like
> that's a lot of overhead for component users -- they would have to track
> version compatibility for each component independently.
> It seems to me that since the common classes are deployed as a unit,
> what you really want to know is if the network client or engine can work
> with the common components/classes as a whole.  It's not very useful if
> say the 10.2 network client can work with the 10.1 i18n code but not the
> 10.1 DRDA encoding.  It's kind of all-or-nothing.

Most likely a single version is fine when there are only two users of
common code (client & engine). If/once there are more users (e.g. tools)
it may get interesting. E.g. the client is coded against version 4, but
new shared code added for tools/engine bumps the version up to 9. As
long as the client continues to work with 9 (as none of the code it is
using has changed) and there are no rules like version+N is only
supported if N <=2, it should be ok.

I think what will typically happen is that sets of shared code will be
added, but their api ("version") is unlikely to change. The code may get
bug fixes, but its offered api remains the same. Thus the version
bumping (with a single version) will most likely be due to added code.

Dan.

[Kathey Marsden]

David W. Van Couvering wrote:

> I added a comment to DERBY-289
> (http://issues.apache.org/jira/browse/DERBY-289) that is my official
> proposal for an approach to sharing code.
>
For my three externals concerns, this is what I see with this proposal.

1) We need to allow mixing of client and server versions.
   This proposal theoretically achieves this but the  COMMON CLASS
CODING GUIDELINES
    seem error prone hard to manage in an open source environment and
you have to wait for those opportune moments before major releases to
refactor code.

2)  We should keep jar file growth commensurate with functionality
improvement.
    Since derby-common.jar holds all the shared classes,   it adds
footprint to the jars in many
configurations. For example, If  I just wanted the embedded server,  I
would need derby.jar + derby-common.jar which would include the drda
classes which are not needed.

3) We should try to avoid asking every user in the world to change their
classpath.
I still  think we should try to avoid it.

I liked your idea of adding just the needed classes to the  existing
jars.  The trick  is to find a way to get Derby to always load the class
from the same jar file first, then no versioning is needed.  
Suddenly, creating  separate package namespaces for the common package
in the jars as last step of the jar build doesn't seem so weird to me.
Obfuscators rename packages all the time and are widely accepted.
It could even happen as a special  releasejar target so it wouldn't
confuse day to day development.

Kathey

[Daniel John Debrunner]

Kathey Marsden wrote:


>
> I liked your idea of adding just the needed classes to the  existing
> jars.  The trick  is to find a way to get Derby to always load the class
> from the same jar file first, then no versioning is needed.  
> Suddenly, creating  separate package namespaces for the common package
> in the jars as last step of the jar build doesn't seem so weird to me.

I've been thinking about suggesting that as well, e.g. the same code
would be generated into two source java files in two (etc.) packages:

org.apache.derby.engine.common
org.apache.derby.client.common

As I said in an earlier e-mail, you can share at many levels, a lot
depends on why you are sharing? Is it to reduce development effort,
reduce static/runtime footprint, add confusion for the user?

> Obfuscators rename packages all the time and are widely accepted.

And we already have generated code, so we handle that currently, ie. the
java files from the parsers.

> It could even happen as a special  releasejar target so it wouldn't
> confuse day to day development.

I don't agree with this, a single build process is much better, it
nmeans the normal development testing is in line with the released builds.


Dan.

[David Van Couvering]

Hm, interesting ideas.  I would be much happier generating two source
files from a single initial source, the reason being that if we
transmogrify package names or generate bytecode then debugging using
stack traces and/or visual debuggers becomes much more difficult.

I would say the motivation for sharing is primarily to confuse users...  :)

Seriously, the motivation is primarily to reduce code duplication, which
requires extra engineering resources both during development and
particularly during maintenance.  I used to work in maintenance for
connectivity at Sybase and we had two exact copies of the same network
code for Unix and VMS and it was nasty nasty nasty.  I am not so
concerned about reducing static/runtime footprint, although it would be
nice.

We would have to find a way to ensure naive developers don't
accidentally modify the generated code when they are doing maintenance,
vs. modifying the primary source code.

I'd like to hear what others have to say about this.

After another day of emails, I'm going to try and capture the issues
brought out and their resolutions and see if we're ready for a final vote.

David

Daniel John Debrunner wrote:

>Kathey Marsden wrote:
>
>
>  
>
>>I liked your idea of adding just the needed classes to the  existing
>>jars.  The trick  is to find a way to get Derby to always load the class
>>from the same jar file first, then no versioning is needed.  
>>Suddenly, creating  separate package namespaces for the common package
>>in the jars as last step of the jar build doesn't seem so weird to me.
>>    
>>
>
>I've been thinking about suggesting that as well, e.g. the same code
>would be generated into two source java files in two (etc.) packages:
>
>org.apache.derby.engine.common
>org.apache.derby.client.common
>
>As I said in an earlier e-mail, you can share at many levels, a lot
>depends on why you are sharing? Is it to reduce development effort,
>reduce static/runtime footprint, add confusion for the user?
>
>  
>
>>Obfuscators rename packages all the time and are widely accepted.
>>    
>>
>
>And we already have generated code, so we handle that currently, ie. the
>java files from the parsers.
>
>  
>
>>It could even happen as a special  releasejar target so it wouldn't
>>confuse day to day development.
>>    
>>
>
>I don't agree with this, a single build process is much better, it
>nmeans the normal development testing is in line with the released builds.
>
>
>Dan.
>
>  
>

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[David Van Couvering]

I thought Kathey/Dan's idea of generating copies of the common code
into two separate directories was interesting and solved a lot of problems,
and I thought it would be worthwhile to walk through this in a bit more
detail.

I took a look at all the relevant use cases I could think of and describe
the steps involved and what the user experience will be like.  Through
this effort I did find a couple of possible issues that may make us want
to think twice about this approach.  Perhaps others can think of ways around
these issues.  I have labelled issues uncovered with the tag <ISSUE> in
the text below, rather than try and summarize them here.


CREATING A COMMON CLASS

Say a developer wants to add a new class for logging.

The developer creates the package directory
java/common/org/apache/derby/common/logging

Under this directory she creates the new class Logger.  A simple
version is shown here.

===========

/**
   Derby class org.apache.derby.common.logging.Logger

   Copyright  yada yada
*/

 
package org.apache.derby.common.logging;

/**
* Class description
*/
public class Logger
{
  public void log(int level, String message)
  {
    // log this message
  }
}

===========

Note that this looks like a regular class.  This is so you
can edit it as a Java class in Java-aware IDEs.


BUILDING THE COMMON PACKAGE

The build script for the org/apache/derby/common directories
will not compile the class.  Instead, it invokes a build-time
tool that will perform the following actions for each class
in the org/apache/derby/common hierarchy:

- Create a copy of the file in java/common/org/apache/derby/engine/common
  and change the package name in this copy to
org.apache.derby.engine.common.

- Create a copy of the file in java/common/org/apache/derby/client/common
  and change the package name in this copy to org.apache.derby.client.common

- No lines will be added or removed.  This ensures that during debugging
  the line numbers in the generated code match the line numbers in the
  original source.

- Compile both copies

The client/common and engine/common directories will be created if
they don't exist and will NOT be included in the svn directory structure.

ant clobber/clean will remove the client/common and engine/common
directories.

The generated source files will NOT be removed after compilation so
that developers can use a source debugger and so that you can navigate
to the source for browsing in Java-aware IDEs.  More on debugging below.

The classes under engine/common will be placed into derby.jar and
derbytools.jar.  The classes under client/common will be placed into
derbyclient.jar.

<ISSUE>
QUESTION: is there a need for mixed versions between the tools and
engine code?  If so we will need to generate a third package hierarchy
org.apache.derby.tools.common.*.
</ISSUE>



USING A COMMON CLASS

Code under java/engine, java/drda, java/tools and java/testing should
all import the classes under org.apache.derby.engine.common.

Code under java/client should import the classes under
org.apache.derby.client.common.

Common code unit tests can use either package hierarchy.



DEBUGGING A COMMON CLASS

Stack traces and debug stacks will point to the generated code
under engine/common or client/common depending on whether you
are debugging engine/tools/network code or client code.

<ISSUE>
If the debugger takes a developer to a common file and they see a bug,
the temptation will be to fix it in place and recompile.  This will
not work -- their changes will be lost when they run the build script.

The developer must instead go to the original source file in
org/apache/derby/common and fix it there.  I suspect this is going
to be confusing and annoying.

We could avoid this confusion by removing the generated source file
after compilation, but I think that this would be even more annoying,
as you wouldn't be able to do interactive source-level debugging.  Most
debugging is just stepping through code that you aren't modifying; you
just want to follow the logic.
</ISSUE>



CHECKING IN A COMMON CLASS

You check in the original source, NOT the generated source.

<ISSUE>
This is another potential source of errors.  A newbie developer
could check in the generated source, and we would have to clean
it out
</ISSUE>


RUNTIME WITH MIXED VERSIONS

If you have a 10.2 client and a 10.3 embedded driver in the same
VM, there is no conflict.  The 10.2 client code uses the 10.2
client/common classes and the 10.3 embedded/engine code uses the
engine/common classes



Daniel John Debrunner wrote:

>Kathey Marsden wrote:
>
>
>  
>
>>I liked your idea of adding just the needed classes to the  existing
>>jars.  The trick  is to find a way to get Derby to always load the class
>>from the same jar file first, then no versioning is needed.  
>>Suddenly, creating  separate package namespaces for the common package
>>in the jars as last step of the jar build doesn't seem so weird to me.
>>    
>>
>
>I've been thinking about suggesting that as well, e.g. the same code
>would be generated into two source java files in two (etc.) packages:
>
>org.apache.derby.engine.common
>org.apache.derby.client.common
>
>As I said in an earlier e-mail, you can share at many levels, a lot
>depends on why you are sharing? Is it to reduce development effort,
>reduce static/runtime footprint, add confusion for the user?
>
>  
>
>>Obfuscators rename packages all the time and are widely accepted.
>>    
>>
>
>And we already have generated code, so we handle that currently, ie. the
>java files from the parsers.
>
>  
>
>>It could even happen as a special  releasejar target so it wouldn't
>>confuse day to day development.
>>    
>>
>
>I don't agree with this, a single build process is much better, it
>nmeans the normal development testing is in line with the released builds.
>
>
>Dan.
>
>  
>

[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Knut Anders Hatlen]

David, I think this is a very good proposal! Much better than the
class-version scheme proposed earlier in my opinion. Read my comments
to your questions below.

"David W. Van Couvering" <David.Vancouvering@...> writes:

> I thought Kathey/Dan's idea of generating copies of the common code
> into two separate directories was interesting and solved a lot of problems,
> and I thought it would be worthwhile to walk through this in a bit more
> detail.
>
> I took a look at all the relevant use cases I could think of and describe
> the steps involved and what the user experience will be like.  Through
> this effort I did find a couple of possible issues that may make us want
> to think twice about this approach.  Perhaps others can think of ways around
> these issues.  I have labelled issues uncovered with the tag <ISSUE> in
> the text below, rather than try and summarize them here.
>
>
> CREATING A COMMON CLASS

[snip]

> <ISSUE>
> QUESTION: is there a need for mixed versions between the tools and
> engine code?  If so we will need to generate a third package hierarchy
> org.apache.derby.tools.common.*.
> </ISSUE>

Don't know, but when one first has a framework for common code this
should be relatively easy, don't you think? One more issue will arise
if the shared code is copied into three or more locations: What if
only two of the components use a file? Should that file only be copied
to those two components or to all the components? (Probably not a big
issue. I don't care if a jar gets a couple of KB bigger, but maybe
someone does.)

[snip]

> DEBUGGING A COMMON CLASS
>
> Stack traces and debug stacks will point to the generated code
> under engine/common or client/common depending on whether you
> are debugging engine/tools/network code or client code.
>
> <ISSUE>
> If the debugger takes a developer to a common file and they see a bug,
> the temptation will be to fix it in place and recompile.  This will
> not work -- their changes will be lost when they run the build script.
>
> The developer must instead go to the original source file in
> org/apache/derby/common and fix it there.  I suspect this is going
> to be confusing and annoying.
>
> We could avoid this confusion by removing the generated source file
> after compilation, but I think that this would be even more annoying,
> as you wouldn't be able to do interactive source-level debugging.  Most
> debugging is just stepping through code that you aren't modifying; you
> just want to follow the logic.
> </ISSUE>

We already have a similar issue in the existing Derby code. The Java
source files for the SQL parser are generated with JavaCC. When you
debug Derby you might find a bug there and fix it the wrong place, as
you said. However, the changes are not lost when you run the build
script because you have to run "ant clobber" or something before you
can regenerate the parser source.

Possible solutions for this issue in the common-code case:

  1. Write a header in each generated file saying that it is generated
     from another source and should not be modified.

  2. Don't delete the copies in an ordinary build, only when running
     "ant clobber". This would be annoying for developers working on
     the common code since they would have to run "ant clobber" for
     every small change they would test.

  3. I haven't used ant much, but couldn't ant check whether the main
     source file is newer than the copy? Then one would only overwrite
     the generated files when the main source is modified. This could
     also be extended with creating a timestamp file after each
     successful build, and ant could check whether any of the
     generated files were modified after the last build and possibly
     print a warning.

> CHECKING IN A COMMON CLASS
>
> You check in the original source, NOT the generated source.
>
> <ISSUE>
> This is another potential source of errors.  A newbie developer
> could check in the generated source, and we would have to clean
> it out
> </ISSUE>

Well, a newbie developer couldn't check in the generated source, it
would have to be a newbie _committer_. Who could that be? ;)

Seriously, I don't think this is going to be a big problem. I'm sure
subversion has an ignore property or something that could be used to
solve this.

--
Knut Anders

[Dyre Tjeldvoll]

Knut Anders Hatlen <Knut.Hatlen@...> writes:

> David, I think this is a very good proposal! Much better than the
> class-version scheme proposed earlier in my opinion. Read my comments
> to your questions below.
>
> "David W. Van Couvering" <David.Vancouvering@...> writes:
>
>> I thought Kathey/Dan's idea of generating copies of the common code
>> into two separate directories was interesting and solved a lot of problems,
>> and I thought it would be worthwhile to walk through this in a bit more
>> detail.
>>
>> I took a look at all the relevant use cases I could think of and describe
>> the steps involved and what the user experience will be like.  Through
>> this effort I did find a couple of possible issues that may make us want
>> to think twice about this approach.  Perhaps others can think of ways around
>> these issues.  I have labelled issues uncovered with the tag <ISSUE> in
>> the text below, rather than try and summarize them here.
>>
>>
>> CREATING A COMMON CLASS
>
> [snip]
>
>> <ISSUE>
>> QUESTION: is there a need for mixed versions between the tools and
>> engine code?  If so we will need to generate a third package hierarchy
>> org.apache.derby.tools.common.*.
>> </ISSUE>
>
> Don't know, but when one first has a framework for common code this
> should be relatively easy, don't you think? One more issue will arise
> if the shared code is copied into three or more locations: What if
> only two of the components use a file? Should that file only be copied
> to those two components or to all the components? (Probably not a big
> issue. I don't care if a jar gets a couple of KB bigger, but maybe
> someone does.)
>

<OT>
You realize, of course, that what you guys are proposing is a "home-grown"
version of what REAL programming languages call *templates*,
don't you...? :-D (Sorry, I could not help it)
</OT>

--
dt

[Andrew McIntyre-2]

On Sep 9, 2005, at 1:16 AM, Knut Anders Hatlen wrote:

  3. I haven't used ant much, but couldn't ant check whether the main

     source file is newer than the copy? 

Yes, Ant's <uptodate> task is available for doing exactly that.

http://ant.apache.org/manual/CoreTasks/uptodate.html

andrew

[Daniel John Debrunner]

Knut Anders Hatlen wrote:

> David, I think this is a very good proposal! Much better than the
> class-version scheme proposed earlier in my opinion. Read my comments
> to your questions below.
>
> "David W. Van Couvering" <David.Vancouvering@...> writes:

[snip]

>><ISSUE>
>>QUESTION: is there a need for mixed versions between the tools and
>>engine code?  If so we will need to generate a third package hierarchy
>>org.apache.derby.tools.common.*.
>></ISSUE>
>
>
> Don't know, but when one first has a framework for common code this
> should be relatively easy, don't you think? One more issue will arise
> if the shared code is copied into three or more locations: What if
> only two of the components use a file? Should that file only be copied
> to those two components or to all the components? (Probably not a big
> issue. I don't care if a jar gets a couple of KB bigger, but maybe
> someone does.)

Agreed, and the jar files are built through dependency checking so if a
class is in the engine common but not used by the engine, then it will
not be part of derby.jar. Though it may be better to have selective
copying. Might confuse people to see (say) drda code in the engine
common area.


[snip]

I agree with Knut's other comments on the issues David raised.

Dan.

[Rick Hillegas-2]

I would like to add a few comments to this very useful discussion:

(1)  The question was raised: why do we want to share code in the first
place? For me, the big issue is the brittleness of code cloning. Over
time, cloned code diverges--I don't know of any bullet-proof process for
preventing this. Over time, the assumptions in the cloned code will
diverge. If you are lucky, this will surface as exceptions. If you're
not lucky, this will surface silently in very tricky bugs. This issue
just gets bigger as the Derby developer community grows and is
particularly important for our under-tested network layer.

(2) A possible solution was raised: instead of cloning the code in the
source tree, clone it during the build process. I have written similar
kinds of ant-based code pre-processors and can give some advice if we
adopt this approach. Done right, it's a pleasant afternoon's work.
However, I have reservations about this approach. It seems to me that it
subverts the meaning of "common" code. To me "common" means "behaves the
same way." It's not just a matter of preserving APIs that the compiler
can check. It comes down to actually doing the same thing internally. As
an example, consider a pair of encoding/decoding methods. You want the
internal encoding/decoding logic to be the same everywhere. I don't
think that build-time cloning saves us from the problem of having to do
a run-time compatibility check.

(3) Concerning the packaging of common code: I'm a big fan of
duplicating the classes in the client and server jars. I would like to
keep the "out of box" experience simple for users who are new to Derby
or even new to Java. Our "0 administration" story is arguably our most
compelling differentiator.

(4) Concerning the specifics of David's proposal: I think that
checkCommonCompatibility() would be easier to use if the second argument
were firstCompatibleVersion rather than firstIncompatibleVersion. It
seemed to me that the secondIncompatibleVersion number should be less
than the firstIncompatibleVersionNumber and I always get muddled when 2 < 1.

Cheers,
-Rick

[Kathey Marsden]

David W. Van Couvering wrote:

> BUILDING THE COMMON PACKAGE
>
> The build script for the org/apache/derby/common directories
> will not compile the class.  Instead, it invokes a build-time
> tool that will perform the following actions for each class
> in the org/apache/derby/common hierarchy:
>

Thinking of this in the same light as an obfuscator, I sort of thought
of the rename/copy  coming much later and only for the jar build. So in
your example when the developer added
java/common/org/apache/derby/common/logging/Logging.java, she would just
import org.apache.derby.common.logging.Logging where needed.  Instead of
looking like a regular class it is a regular class.   The default ant
(all) target it would just get built as  you would expect.   The IDE's
would behave and there are no  extra confusing copies.

The jar build (buildjars) target is where the change happens.

PREFERRED OPTION
The jars are built normally and then afterwards some utility runs on the
binaries to rename the package,
For example the utility changes  org.apache.derby.common.*  ->
org.apache.derby.client.common.* in derbyclient.jar. I have no idea how
hard this is to make such a uitility, but was once told that such
utilities exist. Perhaps someone with more Java bytecode knowledge than
I could  tell us how hard.

ALTERNATE OPTION
In the jar build an alternate source code tree is generated in the jar
build location, which
1) Generates  java/common/org/apache/derby/client/common/*,
java/common/org/apache/derby/engine/common/*,
java/common/org/apache/derby/tools/common/*  java  files.
2) Fixes up the imports for the other java files.
3) Builds the new source tree and  makes the jars.

ISSUES -
- The jar build takes more disk space, and becomes more mysterious,
especially for the ALTERNATE OPTION,
- User reported stack traces will show classes that only exist for the
jar build, which might be a bit confusing to developers, but README
files in the generated directories in the source tree could mitigate this.

Kathey

[Daniel John Debrunner]

Rick Hillegas wrote:

> I would like to add a few comments to this very useful discussion:

[snip]

> (2) A possible solution was raised: instead of cloning the code in the
> source tree, clone it during the build process. I have written similar
> kinds of ant-based code pre-processors and can give some advice if we
> adopt this approach. Done right, it's a pleasant afternoon's work.
> However, I have reservations about this approach. It seems to me that it
> subverts the meaning of "common" code. To me "common" means "behaves the
> same way." It's not just a matter of preserving APIs that the compiler
> can check. It comes down to actually doing the same thing internally. As
> an example, consider a pair of encoding/decoding methods. You want the
> internal encoding/decoding logic to be the same everywhere. I don't
> think that build-time cloning saves us from the problem of having to do
> a run-time compatibility check.

Not sure what point you are making here, how does copying the code into
different packages change its behaviour?

With what ever approach, if the client is at version X and the
engine/network server at version Y then by definition the common code
will be different. Could be the same api, but maybe bug fixes in Y that
don't exist in X.

Dan.

[Daniel John Debrunner]

Kathey Marsden wrote:

> David W. Van Couvering wrote:
>
>
>>BUILDING THE COMMON PACKAGE
>>
>>The build script for the org/apache/derby/common directories
>>will not compile the class.  Instead, it invokes a build-time
>>tool that will perform the following actions for each class
>>in the org/apache/derby/common hierarchy:
>>
>
>
> Thinking of this in the same light as an obfuscator, I sort of thought
> of the rename/copy  coming much later and only for the jar build.

[rest snipped]

I would be very tempted to veto any approach that had the jars
containing different code to that built in the classes directory and
usually(?) used for testing by developers. I believe it increases the
chance for problems being introduced in this process but not caught.

Dan.

[David Van Couvering]

Hi, Rick.

Rick Hillegas wrote:

> I would like to add a few comments to this very useful discussion:
>
> (1)  The question was raised: why do we want to share code in the
> first place? For me, the big issue is the brittleness of code cloning.
> Over time, cloned code diverges--I don't know of any bullet-proof
> process for preventing this. Over time, the assumptions in the cloned
> code will diverge. If you are lucky, this will surface as exceptions.
> If you're not lucky, this will surface silently in very tricky bugs.
> This issue just gets bigger as the Derby developer community grows and
> is particularly important for our under-tested network layer.

yes, agreed

>
> (2) A possible solution was raised: instead of cloning the code in the
> source tree, clone it during the build process. I have written similar
> kinds of ant-based code pre-processors and can give some advice if we
> adopt this approach. Done right, it's a pleasant afternoon's work.
> However, I have reservations about this approach. It seems to me that
> it subverts the meaning of "common" code. To me "common" means
> "behaves the same way." It's not just a matter of preserving APIs that
> the compiler can check. It comes down to actually doing the same thing
> internally. As an example, consider a pair of encoding/decoding
> methods. You want the internal encoding/decoding logic to be the same
> everywhere. I don't think that build-time cloning saves us from the
> problem of having to do a run-time compatibility check.

I'm a bit confused.  The code *is and always will be* exactly the same.  
Were you thinking we'd clone and then diverge?  It really is common code
-- it's not just preserving APIs.  The cloned code can not be modified,
it can only be modified from the original source.  I am concerned I'm
missing your point :)

>
> (3) Concerning the packaging of common code: I'm a big fan of
> duplicating the classes in the client and server jars. I would like to
> keep the "out of box" experience simple for users who are new to Derby
> or even new to Java. Our "0 administration" story is arguably our most
> compelling differentiator.

Yes, this message appears to be loud and clear -- duplicate the classes
in client and server jars.  This becomes even more the right thing to do
if we adopt code cloning.

>
> (4) Concerning the specifics of David's proposal: I think that
> checkCommonCompatibility() would be easier to use if the second
> argument were firstCompatibleVersion rather than
> firstIncompatibleVersion. It seemed to me that the
> secondIncompatibleVersion number should be less than the
> firstIncompatibleVersionNumber and I always get muddled when 2 < 1.
>
Good point.  If we find ourselves going back to that approach (my sense
is the consensus is moving towards cloning) then I think this is a good
adjustment.

Thanks,

David

> Cheers,
> -Rick


[david.vancouvering.vcf]

---

begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:david.vancouvering@...
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard

---

[Andrew McIntyre-2]

On 9/9/05, Daniel John Debrunner <djd@...> wrote:

>
> > Thinking of this in the same light as an obfuscator, I sort of thought
> > of the rename/copy  coming much later and only for the jar build.
>
> [rest snipped]
>
> I would be very tempted to veto any approach that had the jars
> containing different code to that built in the classes directory and
> usually(?) used for testing by developers. I believe it increases the
> chance for problems being introduced in this process but not caught.

Does this mean you would object to both of Kathey's proposed options?
Obfuscator-type modification of the binaries, or ant-based
preprocessing of the source? It wasn't totally clear from this mail if
that was the case.

andrew

[Daniel John Debrunner]

Andrew McIntyre wrote:

> On 9/9/05, Daniel John Debrunner <djd@...> wrote:
>
>>>Thinking of this in the same light as an obfuscator, I sort of thought
>>>of the rename/copy  coming much later and only for the jar build.
>>
>>[rest snipped]
>>
>>I would be very tempted to veto any approach that had the jars
>>containing different code to that built in the classes directory and
>>usually(?) used for testing by developers. I believe it increases the
>>chance for problems being introduced in this process but not caught.
>
>
> Does this mean you would object to both of Kathey's proposed options?
> Obfuscator-type modification of the binaries, or ant-based
> preprocessing of the source? It wasn't totally clear from this mail if
> that was the case.

Both, any solution where the content of the jar files does not match the
content of a non-jar build.

Dan.

[Jeremy Boynes]

Daniel John Debrunner wrote:

> Andrew McIntyre wrote:
>>>
>>>I would be very tempted to veto any approach that had the jars
>>>containing different code to that built in the classes directory and
>>>usually(?) used for testing by developers. I believe it increases the
>>>chance for problems being introduced in this process but not caught.
>>
>>
>>Does this mean you would object to both of Kathey's proposed options?
>>Obfuscator-type modification of the binaries, or ant-based
>>preprocessing of the source? It wasn't totally clear from this mail if
>>that was the case.
>
>
> Both, any solution where the content of the jar files does not match the
> content of a non-jar build.
>

I'm with Dan on this - one of the advantages we have is openness and
post-compilation modification of the binary is quite the opposite.

--
Jeremy