Nabble - OpenJDK Expert Group Observer List

JSR 277 spec lead changes

2008-10-10T15:33:45Z

Hi JSR 277 experts,

This is to let you know that today is my last day at Sun. It has been a
great pleasure to work with all of you, and I really appreciate the
contributions all of you have made in this JSR.

Alex Buckley will take over the responsibility for JSR 277, and serve as
the sole spec lead. I have been working with Alex to make sure the
transition is smooth. If you have any questions or concerns about the
JSR, please contact alex.buckley@... directly.

Going forwards, I'm moving to Research In Motion. I'm reachable on
Linkedin. Hopefully, many of you will stay in touch and our paths will
cross again.

- Stanley

Module change detection...

2008-08-12T16:22:56Z

There are a number of class loading optimizations (including HotSpot's Class Data Sharing) that are hard to make generic due to the lack of a standard way to validate cached class state. We have a unique opportunity to fix this in 277...

I propose that we add:

1. An annotation to record a digest over the module's contents, which would then be visible from ModuleInfo.
2. A convenience api to return the digest, e.g.:

public interface ModuleContent {
...
public byte[] getDigest();
}

The jam tool would compute and add the annotation, and the default implementation(s) of ModuleContent would then just return the value from the annotation.

Other implementations would be free to compute the value at install time (or even runtime); development environments could "generate" a value using a simple counter.

Unlike the version number, this value can be used to ensure that any change to the module's contents can be detected. Both HotSpot and other JVMs can then introduce optimizations like Class Data Sharing that depend on being able to validate the cache file.

It may similarly be useful to have ModuleInfo be able to (lazily) compute a digest for itself, and/or have a digest on ModuleDefinition that encompasses both the metadata (ModuleInfo) and the contents.

// Bryan

Re: Deployment module export...

2008-07-21T20:28:28Z

Hi Bryan,

Bryan Atsatt wrote:
> I believe that ModuleArchiveInfo.getFileName() would only get you a
> reference to the metadata file, not the content, and it is allowed to be
> null as currently defined.

This method will give you a reference to the actual content. For
example, in the local repository case, this will return a filename of
the actual file in the repository interchange directory. That said, this
method is allowed to return null because the repository implementations
may not always make the content available as a physical file.

> Since Repository implementations should be free to store in whatever
> format they choose, assuming that a deployment module can be
> re-constructed from the various apis seems like a very weak alternative
> to an explicit, well specified "export" method.

Fair enough. I will look into adding an API to do this in the next
revision of the API specification draft.

- Stanley

Re: Deployment module export...

2008-07-14T10:34:39Z

I believe that ModuleArchiveInfo.getFileName() would only get you a
reference to the metadata file, not the content, and it is allowed to be
null as currently defined.

Since Repository implementations should be free to store in whatever
format they choose, assuming that a deployment module can be
re-constructed from the various apis seems like a very weak alternative
to an explicit, well specified "export" method.

// Bryan

Adrian Brock wrote:

> Couldn't somebody already do this with
> something like:
>
> public void export(Repository repository)
> {
> URL repoURL = repository.getSourceLocation();
> for (ModuleArchiveInfo archive : repository.list())
> {
> URL url = createRemotableURL(repoURL, archive.getFileName());
> process(url);
> }
> }
>
> Or is the archive's "fileName" only meant to be a logical
> identifier?
>
> On Tue, 2008-07-08 at 10:14 -0700, Bryan Atsatt wrote:
>
>> I believe we need a mechanism to extract a deployment module from a
>> Repository. This would enable:
>>
>> 1. Provisioning systems to copy from a remote repository to a local
>> instance.
>> 2. Migration of all content from one repository to another.
>> 3. Backup of repository content regardless of implementation.
>> 4. Tooling to gather dependent deployment modules into a larger
>> deployment unit.
>>
>> Given that Repository.install() currently takes a URL parameter, it
>> would be nicely symmetrical if Repository could produce a URL (for a
>> given ModuleDefinition) that could be passed directly to install(). For
>> Repositories that act as a front end for a remote store, returning such
>> a URL may be a little extra work, but the underlying mechanisms are
>> already required. For local Repositories, if we recommend that
>> implementations keep the original deployment module as a separate file,
>> then a simple "file:" url can be returned. (Modulo the usual unspecified
>> valid lifetime issue inherent in most URLs.)
>>
>> Thoughts?
>>
>> // Bryan
>>
>>

Re: Deployment module export...

2008-07-10T00:47:57Z

Couldn't somebody already do this with
something like:

public void export(Repository repository)
{
URL repoURL = repository.getSourceLocation();
for (ModuleArchiveInfo archive : repository.list())
{
URL url = createRemotableURL(repoURL, archive.getFileName());
process(url);
}
}

Or is the archive's "fileName" only meant to be a logical
identifier?

On Tue, 2008-07-08 at 10:14 -0700, Bryan Atsatt wrote:

> I believe we need a mechanism to extract a deployment module from a
> Repository. This would enable:
>
> 1. Provisioning systems to copy from a remote repository to a local
> instance.
> 2. Migration of all content from one repository to another.
> 3. Backup of repository content regardless of implementation.
> 4. Tooling to gather dependent deployment modules into a larger
> deployment unit.
>
> Given that Repository.install() currently takes a URL parameter, it
> would be nicely symmetrical if Repository could produce a URL (for a
> given ModuleDefinition) that could be passed directly to install(). For
> Repositories that act as a front end for a remote store, returning such
> a URL may be a little extra work, but the underlying mechanisms are
> already required. For local Repositories, if we recommend that
> implementations keep the original deployment module as a separate file,
> then a simple "file:" url can be returned. (Modulo the usual unspecified
> valid lifetime issue inherent in most URLs.)
>
> Thoughts?
>
> // Bryan
>

--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

Deployment module export...

2008-07-08T10:14:10Z

I believe we need a mechanism to extract a deployment module from a
Repository. This would enable:

1. Provisioning systems to copy from a remote repository to a local
instance.
2. Migration of all content from one repository to another.
3. Backup of repository content regardless of implementation.
4. Tooling to gather dependent deployment modules into a larger
deployment unit.

Given that Repository.install() currently takes a URL parameter, it
would be nicely symmetrical if Repository could produce a URL (for a
given ModuleDefinition) that could be passed directly to install(). For
Repositories that act as a front end for a remote store, returning such
a URL may be a little extra work, but the underlying mechanisms are
already required. For local Repositories, if we recommend that
implementations keep the original deployment module as a separate file,
then a simple "file:" url can be returned. (Modulo the usual unspecified
valid lifetime issue inherent in most URLs.)

Thoughts?

// Bryan

Re: ImportDependency

2008-06-26T15:24:05Z

These are reasonable suggestions. According to the code conventions,
class names should be nouns:

http://java.sun.com/docs/codeconv/html/CodeConventions.doc8.html

As Adrian and Bryan suggested, I will remove the getType() method from
ImportDependency. A module system may need to recognize dependencies in
a module definition from other module system for interoperability (e.g.
recognizes module re-export), so I think we will need to define common
dependency types like ModuleDependency and PackageDependency in the API.

Regarding whether we should use interfaces vs abstract classes, Joe
Darcy has a very good write-up on this topic recently:

http://blogs.sun.com/darcy/entry/api_design_interfaces_versus_abstract

Since these import dependency types are created and used mostly by the
module systems and each module system may have its own set of
implementation for these types, defining them as interfaces is probably
a more sensible thing to do. I will also move the factory methods from
ImportDependency into Modules for now.

I will make these changes in the next revision of the draft API
specification. Let me know if there are other feedbacks on this.

- Stanley

Adrian Brock wrote:

> I agree, getType() looks wrong to me
> (not very object orientated ;-)
>
> But I'm not sure that the spec should define import module and package
> as explicit subclasses or that Import[Dependency] should be a
> concrete class?
>
> I'd prefer to see interfaces. The factory methods would
> just construct implementations of those interfaces.
>
> Somebody else would be free to implement them in a different way,
> e.g. as a wrapper/adapter to the equivalent manifest information of a
> 291 implementation.
>
> On Wed, 2008-06-11 at 13:11 -0700, Bryan Atsatt wrote:
>> The design of this class seems broken to me with the addition of two
>> "types". Why not:
>>
>> 1. Make ImportDependency abstract, removing the getType() and newX methods.
>> 2. Make ImportPackage and ImportModule subclasses, each with their own
>> newX method.
>>
>> (I also strongly suggest that we rename ImportDependency to simply
>> "Import" :^)
>>
>> This model also obviously enables new subtypes, an extensibility point
>> that may come in handy in the future.
>>
>> // Bryan

Re: Repository WAS Re: Goals was: Re: Supporting OSGi Bundles in the Java Module System

2008-06-17T16:51:36Z

Adrian Brock wrote:

> The question is how easy is it to do all this overriding of
> repository implementations to change behaviour and
> not loose some aspect of the behaviour.
>
> e.g. To programmatically create a module like on the other thread.
>
> The tools of a repository are not defined by the spec,
> and there is no repository.install(ModuleContent)
> instead I would create a (possibly dummy) URL backed
> by a jam file stream (or whatever format the repository understands).
>
> But does that really do what I want? If I'm mapping a legacy
> file to a module definition at runtime, I probably don't
> want to install it in the JDK repository implementation
> since it will copy it to some disk location.
>
> The next time I reboot my runtime, I would have to delete it
> and recreate it otherwise it might be stale.
>
> This is all so I can enable use the tools that come with the
> JDK repository implementation and possibly other behaviour
> like Bryan's resolving Module imports to Package imports.

As you mentioned in another thread, "all we really want to do is to PUSH
a ModuleDefinition/Content to the jsr 277 module system such that we can
get back a Module/Classloader with the imports resolved." I don't think
you really meant to change the behavior of the existing repositories
(e.g. bootstrap, etc.), but to make it easy to create a new repository
by extending from a stock implementation and override some of its
behavior. If I misunderstand you, please let me know.

The repository API in the current draft requires many methods to be
overridden in order to build a fully functional repository
implementation. The use case you mentioned is to build a simple
repository at runtime that all it does is to return ModuleDefinitions;
it doesn't has a real physical repository under the cover, and it
probably doesn't support install/uninstall and all the other methods. In
this case, I agreed that the API probably requires too much work for the
implementors, and I will look into this issue to see if some
simplifications can be made in the API.

- Stanley

Re: ModuleDefinition construction WAS Re: Goals was: Re: Supporting OSGi Bundles in the Java Module System

2008-06-13T17:12:36Z

Adrian Brock wrote:

> I see a number of possible use cases here
> (I'll continue with appserver example):
>
> 1) Fully JSR 277
>
> The appserver and JavaEE specs are updated to fully understand
> JSR277/294. In this case, the JavaEE deployment process would
> probably make use of a JSR 277 repository.
>
> 2) Partial JSR 277
>
> The appserver knows about JSR277/294 but the JavaEE specs
> haven't been updated (or they are legacy deployments).
>
> In this case the appserver could try to adapt the deployments
> to conform to the ModuleDefinition/Content.
>
> In this case, there probably wouldn't be a real repository
> implementation, instead the "repository" would be dummy.
> It would just be serving as an adapter from the old deployment packaging
> and metadata to the new one at runtime.
>
> 3) Mixed
>
> Similar to 2, but the opposite.
> The appserver knows about JSR277/294 and wants to support
> .jam files as well as .jar files in JavaEE deployments.
> i.e. .jam files would be unpacked from a .jar EAR deployment
> and then added to the module system runtime.
>
> Conclusion:
>
> I expect there are other possible uses.
>
> The question is whether Repository is the best api
> for cases (2) and (3). i.e. we implement a dummy
> repository with a dummy "URL" for the source
> or whether there should be a more abstract parent
> class (I called it ModuleSource) that doesn't
> assume a physical repository behind the scenes.
>
> At least for cases (2) and (3) all we really want
> to do is to PUSH a ModuleDefinition/Content to the
> jsr 277 module system such that we can get back
> a Module/Classloader with the imports resolved.

In your example, if you want to create ModuleDefinitions from the .jar
files, there are two ways to achieve it:

1. a. Create a ModuleContent based on the content of the .jar file.
b. Generate the JAM's metadata based on information in the .jar file.
c. Construct a JAM's ModuleDefinition with the ModuleContent using
Modules.newJamModuleDefinition(). The ModuleDefinition would be
associated with the JAM module system.

2. a. Create a ModuleContent based on the content of the .jar file.
b. Create your own ModuleDefinition with the ModuleContent and use
the JAM module system as the associated module system. As long as
your ModuleDefinition exposes the appropriate metadata information
reflectively, the JAM module system would treat it like other
JAM's ModuleDefinitions.

In either case, the JAM module system could be used to instantiate and
initialize Module/ClassLoader from the ModuleDefinition.

Creating a repository implementation is a different story. In the
current draft API, you do need to override a number of methods in
Repository in order to have a functional repository implementation. I
still think Repository is the api for module systems to find
ModuleDefinitions. The question is how we could change the API to make
it much easier for developers to create custom repository implementation.

Perhaps we should provide more useful default implementation in the
Repository's methods so developers only need to override very small
number of methods. Or perhaps what we really need is a repository
builder which allows developers to provide a set of ModuleDefinitions
and get back a basic repository implementation to use.

I'll look into the draft API to see if further simplifications can be made.

- Stanley

Re: Refactorability proposal...

2008-06-13T13:10:43Z

Yes, this could be an issue. If JavaEE:4.0.0 merely imports and
re-exports the specific packages, the imports can just be copied out. If
it actually contains them, and they themselves declare their own
versions, the jam tool could emit imports for those specific versions,
but it cannot generate ranges.

This brings us back to the "contract" idea: a named/versioned collection
of import constraints. The indirection afforded by this could certainly
solve this problem.

And we do have it, sort of: a module that contains no classes or
resources itself, only imports and re-exports. In the current model,
this is a bit of a hack because the module will actually be instantiated
with its own loader, which is a waste of cycles/resources.

If, OTOH, we special case such "contract" ("indirect"?) modules such
that they are not instantiated (or at least don't have a loader) but act
only as a bag of import constraints that can be "borrowed" by an
importer, that seems like a reasonable solution.

This could be simplified by doing the borrowing at build time, as in my
proposal, at the cost of some loss of flexibility.

// Bryan

Adrian Brock wrote:

> Another issue would be how you handle the generated package
> constraints.
>
> e.g. Suppose you a module JavaEE:4.0.0 that
> contains packages javax.jms:1.1.0, javax.resource:1.5.0, etc.
>
> I then declare a module constraint
> JavaEE:[4.0.0, 5.0.0)
> i.e. I work with JavaEE4 but not JavaEE5
>
> What do the package constraints resolve to for jms and jca
> if we don't have a JavaEE5 module in the repository? :-)
>
> JavaEE is not a very a good example since package versions
> wouldn't change across a major release, but they could
> in other cases or more general module version constraints.
>
> On Tue, 2008-06-10 at 18:27 -0700, Bryan Atsatt wrote:
>
>> (FYI: I've discussed this with Stanley but wanted to ensure it was
>> visible to everyone.)
>>
>> Proposal
>>
>> 1. Eliminate runtime use of import-by-module in the JAM system:
>>
>> a. Support the (very convenient) import-by-module at the source level.
>> b. jam tool transforms import-by-module to a list of
>> import-by-package statements.
>>
>> 2. Add APIs to fully support import-by-package in the JAM system:
>>
>> a. Support Version annotation in package-info.java (or in
>> module-info.java). If a
>> package does not declare a version, it "inherits" that of the
>> enclosing module.
>> b. Add ImportPackage annotation with version constraints.
>>
>> 3. Add APIs to fully support import-by-package in the abstract framework:
>>
>> a. Add methods to Query to produce import-by-package nodes.
>> b. Replace Query.getIndexableNames() with fully generic variants (I
>> proposed
>> a solution here previously which I will re-post).
>>
>> Rationale
>>
>> Module refactoring is inevitable, particularly during the transition
>> from the current, effectively flat class space to a fine-grained space
>> provided by module systems. We have significant experience with this
>> issue at Oracle (with the transition to our own module system), and OSGi
>> best-practices for conversion include starting with everything in one
>> bundle and then separating out pieces as experience is gained.
>>
>> A very common pattern, in our experience, is for developers to start
>> with many extra jars in their initial module (a mini version of
>> class-path hell). As that module is put into wider use, someone
>> discovers that package X is also contained in their module, and that
>> duplication either leads to runtime conflicts (very bad), or just plain
>> footprint bloat. The obvious answer is to put package X in a separate
>> module, and have everyone share it via imports.
>>
>> But... not so fast. If there are consumers of that module who import it
>> by module name alone, then pulling X out of it will cause those
>> importers to break. And if it is possible for your module to have been
>> imported by *anyone* by name alone, then you are stuck: either you break
>> them or you live with the incorrect granularity (which just isn't an
>> option in the conflict scenarios). Not a happy choice.
>>
>> Originally, I had proposed to do away with import-by-module altogether,
>> both to avoid this problem and to eliminate the conceptual disconnect.
>> Your code does not today contain import statements that name *jars*, it
>> names packages and/or specific classes in those packages. Why invent a
>> new system that takes such a large step backwards?
>>
>> The answer is simply convenience. Imagine a module that contains 100
>> packages and it is obvious that writing a single import statement is far
>> easier than discovering and explicitly writing all the package imports.
>> Yes, IDEs will likely mostly eliminate this issue, but it still makes
>> sense to be able to do this by hand.
>>
>> This proposal is an attempt to maintain the convenience while adding the
>> crucial ability to safely refactor: step 1b is the central idea.
>>
>> // Bryan
>>

Re: Refactorability proposal...

2008-06-13T12:22:52Z

Yeah, that was my original proposal (seems like years ago now :^), but
it means that you can never escape from your mistakes.

As a newbie, you create module X containing your code in package X. But
you have a dependency on package Y, so, following the traditional
keep-adding-to-the-classpath-till-it-works model, you throw in package
Y. But, argh, Y depends on Z, so you throw that in as well. Being a
newbie, you export everything to keep it simple.

Sometime later you realize it was a mistake to export Y and Z: your code
didn't actually depend on Z, nor did it expose Y. And the version of Y
that you want to use now is available in a shiny new module.

So you refactor to remove packages Y and Z and import Y instead.

But for compatibility, you are forced to also:

1. Import Z.
2. Re-export both Y and Z.

Ugly. Far worse though: it binds your importers both to the version of Y
that you depend on (but don't expose in your code), and the version of Z
that you don't depend on at all!

And all of this is a direct consequence of import-by-module. With
import-by-package, this issue just... vanishes.

// Bryan

Adrian Brock wrote:

> On Thu, 2008-06-12 at 17:13 -0700, Bryan Atsatt wrote:
>
>> Really, I just want to make it possible for a module developer to be in
>> a position to "know" that splitting one module into multiple modules
>> will not break existing importers. I'm quite open to other solutions, as
>> long as they are reasonable.
>>
>
> But can't that also be achieved with the old module re-exporting the
> split out module for backwards compatibility purposes?
>

Re: Refactorability proposal...

2008-06-13T03:10:06Z

On Thu, 2008-06-12 at 10:53 -0700, Bryan Atsatt wrote:
> The proposal is to resolve module-->packages at *build* time, not
> deploy. This way, the mapping is guaranteed to be valid.

Whether it is done at build time or deploy time,
my criticisms still apply.

Its still based upon what was in the repository at that time
and may not reflect what is in the repository at runtime.

Generating the package version constraints from a module constraint
is non-trivial in general.

> And yes, this does make the sysadmin job a bit more difficult, but
> that should be relatively easy to address with tooling.

The old, "A tool will solve the complexity problem" opt out. ;-)
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

Re: Refactorability proposal...

2008-06-13T03:05:02Z

On Thu, 2008-06-12 at 17:13 -0700, Bryan Atsatt wrote:
> Really, I just want to make it possible for a module developer to be in
> a position to "know" that splitting one module into multiple modules
> will not break existing importers. I'm quite open to other solutions, as
> long as they are reasonable.

But can't that also be achieved with the old module re-exporting the
split out module for backwards compatibility purposes?
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

Re: Refactorability proposal...

2008-06-12T17:13:39Z

Stanley M. Ho wrote:

> Hi Bryan,
>
> I agreed that refactorability is indeed an important issue we need to
> address. That said, there are couple concerns I have related to step
> 1b. I'm not arguing against import-by-package at runtime; only against
> the transformation of import-by-module to import-by-package at
> compile-time.
>
> A compile/build-time transformation from import-by-module to
> import-by-package could fundamentally be done in one of two ways.
>
> First, it could involve analysis of the module's method signatures.
> The issue is that the analysis may not be accurate all the time (e.g.
> code that uses reflection, Class.forName(), etc.). The programmer will
> then be required do extra trial-and-error steps to fix up the
> dependencies in the metadata, and this is not acceptable.
>
> Second, it could simply expand import-by-module as the set of exported
> packages in the imported module, then there is a problem when
> migrating from one version of an imported module to a newer one and
> the set of packages have changed. Suppose a core module in JDK 7
> exports package P and the one in JDK 8 exports packages P and Q. If a
> module imports JDK 7+ and is compiled against JDK 7, step 1b will
> transform the module to import only the packages that are known in the
> core module in JDK 7. When this module runs in JDK 8 or later, it will
> not see the new packages. This is problematic because many modules
> will compile against an older version of the JDK and want to call new
> API in newer JDK through reflection; in general, this situation
> applies to using other libraries as well. This will also cause problem
> in applications that involve weaving at runtime and the exact required
> packages are not known at compile-time.

This is why OSGi has dynamic-import. And we need it as well. The fact
that a forName/loadClass() happens to work *without* an explicit import
is nothing but a ticking time-bomb.

>
> There are some general problems with expanding modules to packages at
> compile/build-time, regardless of how the expansion is achieved.
>
> First, there are modules (e.g. classpath module) that cannot logically
> have exported packages which are known at compile-time and at runtime,
> thus it would be impossible to perform the transformation in this
> case. (Technically, we can go through every JAR in the classpath in
> every launch to find out the exports of the classpath module at
> runtime, but this has huge startup performance impact.)

This seems like a very special case to me. First, a "classpath" module
is, by definition, a very strange beast in that it is fully dynamic: any
importer can be broken at any time by a simple command-line change. I
can certainly understand a classpath module existing to support legacy
code, but I have a hard time imagining creating a new module that would
express a dependency on such a thing.

Even so, I concede that there are likely special cases where import by
"something other than package name" at runtime is useful.

Really, I just want to make it possible for a module developer to be in
a position to "know" that splitting one module into multiple modules
will not break existing importers. I'm quite open to other solutions, as
long as they are reasonable.
>
> Second, it implies that people export packages but import modules.
> This will be hard to explain to beginners.
So explain that you both export and import packages, but we provide
syntactic sugar for both for convenience. (I'd even by happy to
eliminate that convenience on the import, to simplify things further, as
you know.)

This is a silly argument IMO (sorry). In modern IDEs, when you type in
an unqualified class name, the system pops up and asks you which one you
mean. Without it, you have to (gasp) type a bunch of import statements
at the top of your file. Why didn't James make this easier by allowing
us to import a jar?
>
> Last, but perhaps the most important, it means that unresolveable
> packages at runtime will generate error messages which the programmer
> must manually map back to imported modules and the transformation
> process implemented by a tool. The fundamental aim of a programming
> system should be to reduce the semantic gap between what a programmer
> sees in code and what a programmer sees at execution; the mapping from
> packages back to modules increases the semantic gap.
I agree there is a semantic gap, but not the one you cite.

Either refactoring is supported, or it isn't. If it isn't, this
particular error should not occur. If it is supported, but an
import-by-module exists, it will be even worse than you suggest: there
won't be *any* failure at resolution time.

Say I import "module X", which contains package X and also happens to
contain packages Y and Z which I "silently" depend on. I compile and run
fine, but in some future X is split into X, Y and Z modules, and, for
some reason module Z is not installed. With my proposal, resolution for
my module will fail ("package Z not found") iff there is no installed
module containing Z.

Without my proposal (or some other similar solution), resolution will
succeed when it shouldn't, and a NoClassDefFoundError will occur at some
random point in execution naming a class in a package that was never
explicitly imported. How is that better?
>
> I accept that pushing module imports through to runtime makes it hard
> to "rewrite" a program (i.e. to refactor module members) but it makes
> it easy to "read" the program (in the sense of a small semantic gap
> between what imports are read/written and what imports are executed).
> A core Java principle has always been that reading is more important
> than writing, so we do not believe increased ease of refactoring on
> relatively rare occasions is worth complicating every single reading
> of a dependency.
Well there's a distinct difference between our perceptions: "relatively
rare occasions" strikes me as extremely misleading. Though I have
significant, direct, and exactly analogous experience that tells me the
frequency will be higher than you seem to expect, I'm perfectly happy to
concede that it may be lower. But unless you believe that it will be
zero, we need a tenable solution.
> I still think the refactorability problem you brought up is very
> important to address.
Well at least we can agree on that :^). Any ideas that are less ugly
than mine?

// Bryan

Re: Refactorability proposal...

2008-06-12T11:02:08Z

Hi Bryan,

I agreed that refactorability is indeed an important issue we need to
address. That said, there are couple concerns I have related to step 1b.
I'm not arguing against import-by-package at runtime; only against the
transformation of import-by-module to import-by-package at compile-time.

A compile/build-time transformation from import-by-module to
import-by-package could fundamentally be done in one of two ways.

First, it could involve analysis of the module's method signatures. The
issue is that the analysis may not be accurate all the time (e.g. code
that uses reflection, Class.forName(), etc.). The programmer will then
be required do extra trial-and-error steps to fix up the dependencies in
the metadata, and this is not acceptable.

Second, it could simply expand import-by-module as the set of exported
packages in the imported module, then there is a problem when migrating
from one version of an imported module to a newer one and the set of
packages have changed. Suppose a core module in JDK 7 exports package P
and the one in JDK 8 exports packages P and Q. If a module imports JDK
7+ and is compiled against JDK 7, step 1b will transform the module to
import only the packages that are known in the core module in JDK 7.
When this module runs in JDK 8 or later, it will not see the new
packages. This is problematic because many modules will compile against
an older version of the JDK and want to call new API in newer JDK
through reflection; in general, this situation applies to using other
libraries as well. This will also cause problem in applications that
involve weaving at runtime and the exact required packages are not known
at compile-time.

There are some general problems with expanding modules to packages at
compile/build-time, regardless of how the expansion is achieved.

First, there are modules (e.g. classpath module) that cannot logically
have exported packages which are known at compile-time and at runtime,
thus it would be impossible to perform the transformation in this case.
(Technically, we can go through every JAR in the classpath in every
launch to find out the exports of the classpath module at runtime, but
this has huge startup performance impact.)

Second, it implies that people export packages but import modules. This
will be hard to explain to beginners.

Last, but perhaps the most important, it means that unresolveable
packages at runtime will generate error messages which the programmer
must manually map back to imported modules and the transformation
process implemented by a tool. The fundamental aim of a programming
system should be to reduce the semantic gap between what a programmer
sees in code and what a programmer sees at execution; the mapping from
packages back to modules increases the semantic gap.

I accept that pushing module imports through to runtime makes it hard to
"rewrite" a program (i.e. to refactor module members) but it makes it
easy to "read" the program (in the sense of a small semantic gap between
what imports are read/written and what imports are executed). A core
Java principle has always been that reading is more important than
writing, so we do not believe increased ease of refactoring on
relatively rare occasions is worth complicating every single reading of
a dependency.

I still think the refactorability problem you brought up is very
important to address. That said, the proposed transformation seems to
have many issues as an acceptable general solution.

- Stanley

Re: Refactorability proposal...

2008-06-12T10:53:11Z

The proposal is to resolve module-->packages at *build* time, not
deploy. This way, the mapping is guaranteed to be valid.

And yes, this does make the sysadmin job a bit more difficult, but that
should be relatively easy to address with tooling.

The main point is that relying on import-by-module at runtime guarantees
that refactoring is problematic. My proposal seems like a reasonable way
to deal with this, but I wanted to start this discussion to explore
others as well.

Any thoughts on how to solve this problem?

// Bryan

Adrian Brock wrote:

> Refactoring:
>
> Aren't there cases where resolving a module import
> into its packages during deployment into the repository
> will break with some refactorings?
>
> I'm not talking about the more stable module refactorings
> where modules are broken into other modules (which
> you seem to be addressing).
>
> This is more development time practices where somebody
> is refactoring module contents. i.e. the importing
> module could end up with a "stale" view of what it
> should be importing.
>
> A not very good example would be my module B imports a module A
> where A exports com.acme.a.interfaces and
> com.acme.a.implementation
> I don't use com.acme.a.implementation explicitly so it
> shouldn't really be one of my constraints.
>
> Somebody then refactors module B such that the
> implementation is now in com.acme.a.impl
> but my module wants to import the now non-existant
> (or at least stale) com.acme.a.implementation.
>
> Overrides in the repository:
>
> I know we avoid these discussions on the list since it
> is really a tooling issue.
>
> But it is a lot easier for a sysadmin to change
> a single version constraint for a module import
> than it is to have to figure out
> what package versions they need to change on the
> package imports and change each one individually.
>
> Basically, the convenience of import-by-module
> extends beyond the development/source code phase.
>
> OSGi/291
>
> There's still going to be a kind of module import
> somewhere to handle OSGi's bundle import constraint.
> i.e. OSGi bundle B issues a request to import bundle A
> which is really a 277 module.
>
> On Tue, 2008-06-10 at 18:27 -0700, Bryan Atsatt wrote:
>
>> (FYI: I've discussed this with Stanley but wanted to ensure it was
>> visible to everyone.)
>>
>> Proposal
>>
>> 1. Eliminate runtime use of import-by-module in the JAM system:
>>
>> a. Support the (very convenient) import-by-module at the source level.
>> b. jam tool transforms import-by-module to a list of
>> import-by-package statements.
>>
>> 2. Add APIs to fully support import-by-package in the JAM system:
>>
>> a. Support Version annotation in package-info.java (or in
>> module-info.java). If a
>> package does not declare a version, it "inherits" that of the
>> enclosing module.
>> b. Add ImportPackage annotation with version constraints.
>>
>> 3. Add APIs to fully support import-by-package in the abstract framework:
>>
>> a. Add methods to Query to produce import-by-package nodes.
>> b. Replace Query.getIndexableNames() with fully generic variants (I
>> proposed
>> a solution here previously which I will re-post).
>>
>> Rationale
>>
>> Module refactoring is inevitable, particularly during the transition
>> from the current, effectively flat class space to a fine-grained space
>> provided by module systems. We have significant experience with this
>> issue at Oracle (with the transition to our own module system), and OSGi
>> best-practices for conversion include starting with everything in one
>> bundle and then separating out pieces as experience is gained.
>>
>> A very common pattern, in our experience, is for developers to start
>> with many extra jars in their initial module (a mini version of
>> class-path hell). As that module is put into wider use, someone
>> discovers that package X is also contained in their module, and that
>> duplication either leads to runtime conflicts (very bad), or just plain
>> footprint bloat. The obvious answer is to put package X in a separate
>> module, and have everyone share it via imports.
>>
>> But... not so fast. If there are consumers of that module who import it
>> by module name alone, then pulling X out of it will cause those
>> importers to break. And if it is possible for your module to have been
>> imported by *anyone* by name alone, then you are stuck: either you break
>> them or you live with the incorrect granularity (which just isn't an
>> option in the conflict scenarios). Not a happy choice.
>>
>> Originally, I had proposed to do away with import-by-module altogether,
>> both to avoid this problem and to eliminate the conceptual disconnect.
>> Your code does not today contain import statements that name *jars*, it
>> names packages and/or specific classes in those packages. Why invent a
>> new system that takes such a large step backwards?
>>
>> The answer is simply convenience. Imagine a module that contains 100
>> packages and it is obvious that writing a single import statement is far
>> easier than discovering and explicitly writing all the package imports.
>> Yes, IDEs will likely mostly eliminate this issue, but it still makes
>> sense to be able to do this by hand.
>>
>> This proposal is an attempt to maintain the convenience while adding the
>> crucial ability to safely refactor: step 1b is the central idea.
>>
>> // Bryan
>>

Repository WAS Re: Goals was: Re: Supporting OSGi Bundles in the Java Module System

2008-06-12T06:55:47Z

On Wed, 2008-06-11 at 22:31 -0700, Stanley M. Ho wrote:

> >> 3) QoS - what tools are available for the repository
>
> The Java Module System implementation will come with a standard tool to
> manage module archives in the repository, e.g. install/uninstall, etc.
> Is there any particular aspect of the tools support that you are
> concerned with but currently not possible in the draft API?
>
> >> 4) Location - file system/url based, etc.
>
> This is handled by the ModuleContent abstraction described above.
>
> > * Developing a full repository just to define a different archive
> > format is too much (e.g. legacy javaee deployments)
>
> I would like to know what you think about the draft API to see if it is
> still too much to develop a repository.

The question is how easy is it to do all this overriding of
repository implementations to change behaviour and
not loose some aspect of the behaviour.

e.g. To programmatically create a module like on the other thread.

The tools of a repository are not defined by the spec,
and there is no repository.install(ModuleContent)
instead I would create a (possibly dummy) URL backed
by a jam file stream (or whatever format the repository understands).

But does that really do what I want? If I'm mapping a legacy
file to a module definition at runtime, I probably don't
want to install it in the JDK repository implementation
since it will copy it to some disk location.

The next time I reboot my runtime, I would have to delete it
and recreate it otherwise it might be stale.

This is all so I can enable use the tools that come with the
JDK repository implementation and possibly other behaviour
like Bryan's resolving Module imports to Package imports.
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

Delegation model WAS Re: Goals was: Re: Supporting OSGi Bundles in the Java Module System

2008-06-12T06:28:37Z

On Wed, 2008-06-11 at 22:31 -0700, Stanley M. Ho wrote:
> >> 2) Model - parent/child or peer
>
> I think you meant the module system inteorp model, not the repository
> delegation model. Right?
>

I don't see the difference? Hierarchical or Peer delegation
policies between repositories are orthogonal choices to which
module system backs that repository.

It's precisely this orthogonality that causes me to think
that we can't just say "create a new repository implementation"
to change a particular rule/policy.

There are ~ N^2 choices of overrides to what the default
repositories do.

e.g. (probably not a good example since the osgi
module system would use OBR?)

1) A simple file repository containing jam files
2) The same containing osgi bundles
3) A remote repository containing jam files
4) A remote repository contains osgi bundles

And those are just two of the many
orthogonal choices 2^2 = 4
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

ModuleDefinition construction WAS Re: Goals was: Re: Supporting OSGi Bundles in the Java Module System

2008-06-12T06:21:02Z

On Wed, 2008-06-11 at 22:31 -0700, Stanley M. Ho wrote:

> Hi Adrian,
>
> Adrian Brock wrote:
> > My critisms of using the JSR277 repositories to do the integration
> > included:
>
> I want to better understand your concerns around developing
> repositories, since I believe some of your concerns have already been
> addressed in the draft API.
>

The comments I included were from discussions (a year ago)
before the current api changes, so some parts maybe out-of-date.

> > * There are already lots of orthogonal reasons for swapping
> > repository implementations
> >
> > On Mon, 2007-02-26 at 12:26 +0100, Adrian wrote:
> >> So besides the peer/hierarchy argument there
> >> is also the problem that the repository is dealing
> >> with too many concerns.
> >>
> >> 1) ModuleDefinition construction
>
> The draft API provides a Modules.newJamModuleDefinition() method to
> construct a ModuleDefinition from JAM's metadata and ModuleContent. The
> ModuleContent abstraction enables a module archive to be stored in the
> repository in any form under the cover. This should make it easier for a
> repository implementation to construct ModuleDefinitions from JAM files,
> or from other archive formats (as long as the metadata information of a
> module can be expressed as JAM's metadata, and the content of the module
> can be exposed through the ModuleContent abstraction).
>
> In your original use case in JavaEE, the appserver wants to use the
> module system for "plain old module" wiring but also wants to define its
> own "modules" for wars, ears, etc., and some of which looks nothing like
> jars/jams in structure. Do you think the draft API and the ModuleContent
> abstraction are sufficient to address the ModuleDefinition construction
> issue you were concerned with?

I see a number of possible use cases here
(I'll continue with appserver example):

1) Fully JSR 277

The appserver and JavaEE specs are updated to fully understand
JSR277/294. In this case, the JavaEE deployment process would
probably make use of a JSR 277 repository.

2) Partial JSR 277

The appserver knows about JSR277/294 but the JavaEE specs
haven't been updated (or they are legacy deployments).

In this case the appserver could try to adapt the deployments
to conform to the ModuleDefinition/Content.

In this case, there probably wouldn't be a real repository
implementation, instead the "repository" would be dummy.
It would just be serving as an adapter from the old deployment packaging
and metadata to the new one at runtime.

3) Mixed

Similar to 2, but the opposite.
The appserver knows about JSR277/294 and wants to support
.jam files as well as .jar files in JavaEE deployments.
i.e. .jam files would be unpacked from a .jar EAR deployment
and then added to the module system runtime.

Conclusion:

I expect there are other possible uses.

The question is whether Repository is the best api
for cases (2) and (3). i.e. we implement a dummy
repository with a dummy "URL" for the source
or whether there should be a more abstract parent
class (I called it ModuleSource) that doesn't
assume a physical repository behind the scenes.

At least for cases (2) and (3) all we really want
to do is to PUSH a ModuleDefinition/Content to the
jsr 277 module system such that we can get back
a Module/Classloader with the imports resolved.
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

Re: ImportDependency

2008-06-12T05:52:33Z

I agree, getType() looks wrong to me
(not very object orientated ;-)

But I'm not sure that the spec should define import module and package
as explicit subclasses or that Import[Dependency] should be a
concrete class?

I'd prefer to see interfaces. The factory methods would
just construct implementations of those interfaces.

Somebody else would be free to implement them in a different way,
e.g. as a wrapper/adapter to the equivalent manifest information of a
291 implementation.

On Wed, 2008-06-11 at 13:11 -0700, Bryan Atsatt wrote:

> The design of this class seems broken to me with the addition of two
> "types". Why not:
>
> 1. Make ImportDependency abstract, removing the getType() and newX methods.
> 2. Make ImportPackage and ImportModule subclasses, each with their own
> newX method.
>
> (I also strongly suggest that we rename ImportDependency to simply
> "Import" :^)
>
> This model also obviously enables new subtypes, an extensibility point
> that may come in handy in the future.
>
> // Bryan

--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

Re: Refactorability proposal...

2008-06-12T05:01:26Z

Another issue would be how you handle the generated package
constraints.

e.g. Suppose you a module JavaEE:4.0.0 that
contains packages javax.jms:1.1.0, javax.resource:1.5.0, etc.

I then declare a module constraint
JavaEE:[4.0.0, 5.0.0)
i.e. I work with JavaEE4 but not JavaEE5

What do the package constraints resolve to for jms and jca
if we don't have a JavaEE5 module in the repository? :-)

JavaEE is not a very a good example since package versions
wouldn't change across a major release, but they could
in other cases or more general module version constraints.

On Tue, 2008-06-10 at 18:27 -0700, Bryan Atsatt wrote:

> (FYI: I've discussed this with Stanley but wanted to ensure it was
> visible to everyone.)
>
> Proposal
>
> 1. Eliminate runtime use of import-by-module in the JAM system:
>
> a. Support the (very convenient) import-by-module at the source level.
> b. jam tool transforms import-by-module to a list of
> import-by-package statements.
>
> 2. Add APIs to fully support import-by-package in the JAM system:
>
> a. Support Version annotation in package-info.java (or in
> module-info.java). If a
> package does not declare a version, it "inherits" that of the
> enclosing module.
> b. Add ImportPackage annotation with version constraints.
>
> 3. Add APIs to fully support import-by-package in the abstract framework:
>
> a. Add methods to Query to produce import-by-package nodes.
> b. Replace Query.getIndexableNames() with fully generic variants (I
> proposed
> a solution here previously which I will re-post).
>
> Rationale
>
> Module refactoring is inevitable, particularly during the transition
> from the current, effectively flat class space to a fine-grained space
> provided by module systems. We have significant experience with this
> issue at Oracle (with the transition to our own module system), and OSGi
> best-practices for conversion include starting with everything in one
> bundle and then separating out pieces as experience is gained.
>
> A very common pattern, in our experience, is for developers to start
> with many extra jars in their initial module (a mini version of
> class-path hell). As that module is put into wider use, someone
> discovers that package X is also contained in their module, and that
> duplication either leads to runtime conflicts (very bad), or just plain
> footprint bloat. The obvious answer is to put package X in a separate
> module, and have everyone share it via imports.
>
> But... not so fast. If there are consumers of that module who import it
> by module name alone, then pulling X out of it will cause those
> importers to break. And if it is possible for your module to have been
> imported by *anyone* by name alone, then you are stuck: either you break
> them or you live with the incorrect granularity (which just isn't an
> option in the conflict scenarios). Not a happy choice.
>
> Originally, I had proposed to do away with import-by-module altogether,
> both to avoid this problem and to eliminate the conceptual disconnect.
> Your code does not today contain import statements that name *jars*, it
> names packages and/or specific classes in those packages. Why invent a
> new system that takes such a large step backwards?
>
> The answer is simply convenience. Imagine a module that contains 100
> packages and it is obvious that writing a single import statement is far
> easier than discovering and explicitly writing all the package imports.
> Yes, IDEs will likely mostly eliminate this issue, but it still makes
> sense to be able to do this by hand.
>
> This proposal is an attempt to maintain the convenience while adding the
> crucial ability to safely refactor: step 1b is the central idea.
>
> // Bryan

--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

Re: Refactorability proposal...

2008-06-12T04:50:41Z

Refactoring:

Aren't there cases where resolving a module import
into its packages during deployment into the repository
will break with some refactorings?

I'm not talking about the more stable module refactorings
where modules are broken into other modules (which
you seem to be addressing).

This is more development time practices where somebody
is refactoring module contents. i.e. the importing
module could end up with a "stale" view of what it
should be importing.

A not very good example would be my module B imports a module A
where A exports com.acme.a.interfaces and
com.acme.a.implementation
I don't use com.acme.a.implementation explicitly so it
shouldn't really be one of my constraints.

Somebody then refactors module B such that the
implementation is now in com.acme.a.impl
but my module wants to import the now non-existant
(or at least stale) com.acme.a.implementation.

Overrides in the repository:

I know we avoid these discussions on the list since it
is really a tooling issue.

But it is a lot easier for a sysadmin to change
a single version constraint for a module import
than it is to have to figure out
what package versions they need to change on the
package imports and change each one individually.

Basically, the convenience of import-by-module
extends beyond the development/source code phase.

OSGi/291

There's still going to be a kind of module import
somewhere to handle OSGi's bundle import constraint.
i.e. OSGi bundle B issues a request to import bundle A
which is really a 277 module.

On Tue, 2008-06-10 at 18:27 -0700, Bryan Atsatt wrote:

--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

Re: Goals was: Re: Supporting OSGi Bundles in the Java Module System

2008-06-11T22:31:07Z

Hi Adrian,

Adrian Brock wrote:
> My critisms of using the JSR277 repositories to do the integration
> included:

I want to better understand your concerns around developing
repositories, since I believe some of your concerns have already been
addressed in the draft API.

> * There are already lots of orthogonal reasons for swapping
> repository implementations
>
> On Mon, 2007-02-26 at 12:26 +0100, Adrian wrote:
>> So besides the peer/hierarchy argument there
>> is also the problem that the repository is dealing
>> with too many concerns.
>>
>> 1) ModuleDefinition construction

The draft API provides a Modules.newJamModuleDefinition() method to
construct a ModuleDefinition from JAM's metadata and ModuleContent. The
ModuleContent abstraction enables a module archive to be stored in the
repository in any form under the cover. This should make it easier for a
repository implementation to construct ModuleDefinitions from JAM files,
or from other archive formats (as long as the metadata information of a
module can be expressed as JAM's metadata, and the content of the module
can be exposed through the ModuleContent abstraction).

In your original use case in JavaEE, the appserver wants to use the
module system for "plain old module" wiring but also wants to define its
own "modules" for wars, ears, etc., and some of which looks nothing like
jars/jams in structure. Do you think the draft API and the ModuleContent
abstraction are sufficient to address the ModuleDefinition construction
issue you were concerned with?

>> 2) Model - parent/child or peer

I think you meant the module system inteorp model, not the repository
delegation model. Right?

>> 3) QoS - what tools are available for the repository

The Java Module System implementation will come with a standard tool to
manage module archives in the repository, e.g. install/uninstall, etc.
Is there any particular aspect of the tools support that you are
concerned with but currently not possible in the draft API?

>> 4) Location - file system/url based, etc.

This is handled by the ModuleContent abstraction described above.

> * Developing a full repository just to define a different archive
> format is too much (e.g. legacy javaee deployments)

I would like to know what you think about the draft API to see if it is
still too much to develop a repository.

- Stanley

ImportDependency

2008-06-11T13:11:53Z

The design of this class seems broken to me with the addition of two
"types". Why not:

1. Make ImportDependency abstract, removing the getType() and newX methods.
2. Make ImportPackage and ImportModule subclasses, each with their own
newX method.

(I also strongly suggest that we rename ImportDependency to simply
"Import" :^)

This model also obviously enables new subtypes, an extensibility point
that may come in handy in the future.

// Bryan

Re: Exported Classes and OSGi Re: Supporting OSGi Bundles in the Java Module System

2008-06-11T11:37:56Z

Adrian Brock wrote:
> So a 291 module that hasn't been updated to reflect changes from 294
> would just expose package exports and the importing 277 module
> would know how to deal with that?

A 291 module which exports packages is fundamentally exporting public
types, which will continue to work across module boundaries. If a JAM
module imports a 291 module, it'll know how to deal with this if
getExportedClasses() is not supported in the imported ModuleDefinition.

> The 291 module classes would be like legacy classes that
> don't belong to a module in the 294 sense.
> i.e. Class.getModuleInfo() would return null

Types which are not in a named module are in the unnamed module; the
changes in the Java Language Specification (JLS) will say this. null is
probably as good a value as any for representing this.

- Stanley

Re: Exported Classes and OSGi Re: Supporting OSGi Bundles in the Java Module System

2008-06-11T05:08:57Z

On Tue, 2008-06-10 at 17:10 -0700, Stanley M. Ho wrote:

> > ModuleDefinition still refers to exported and member classes.
> > Although this is probably more of an issue for 294/291 interoperation,
> > I don't think 291 has such notions?
>
> With modules in the language, only public types are accessible outside
> their development modules. A deployment module system could leverage
> this by considering public types (or a subset thereof) as 'exported',
> and the granularity of exports can therefore be types or packages. The
> JAM module system integrates tightly with development modules and it
> will expose type-level exports, but other module systems can continue to
> expose package-level exports. Thus, ModuleDefinition should support both
> granularity levels. If a module system does not support type-level
> exports (e.g. 291), the implementation of
> ModuleDefinition.getExportedClasses()/getMemberClasses() could simply
> throw UnsupportedOperationException.
>

So a 291 module that hasn't been updated to reflect changes from 294
would just expose package exports and the importing 277 module
would know how to deal with that?

The 291 module classes would be like legacy classes that
don't belong to a module in the 294 sense.
i.e. Class.getModuleInfo() would return null

> - Stanley
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

Refactorability proposal...

2008-06-10T18:27:28Z

(FYI: I've discussed this with Stanley but wanted to ensure it was
visible to everyone.)

Proposal

1. Eliminate runtime use of import-by-module in the JAM system:

a. Support the (very convenient) import-by-module at the source level.
b. jam tool transforms import-by-module to a list of
import-by-package statements.

2. Add APIs to fully support import-by-package in the JAM system:

a. Support Version annotation in package-info.java (or in
module-info.java). If a
package does not declare a version, it "inherits" that of the
enclosing module.
b. Add ImportPackage annotation with version constraints.

3. Add APIs to fully support import-by-package in the abstract framework:

a. Add methods to Query to produce import-by-package nodes.
b. Replace Query.getIndexableNames() with fully generic variants (I
proposed
a solution here previously which I will re-post).

Rationale

Module refactoring is inevitable, particularly during the transition
from the current, effectively flat class space to a fine-grained space
provided by module systems. We have significant experience with this
issue at Oracle (with the transition to our own module system), and OSGi
best-practices for conversion include starting with everything in one
bundle and then separating out pieces as experience is gained.

A very common pattern, in our experience, is for developers to start
with many extra jars in their initial module (a mini version of
class-path hell). As that module is put into wider use, someone
discovers that package X is also contained in their module, and that
duplication either leads to runtime conflicts (very bad), or just plain
footprint bloat. The obvious answer is to put package X in a separate
module, and have everyone share it via imports.

But... not so fast. If there are consumers of that module who import it
by module name alone, then pulling X out of it will cause those
importers to break. And if it is possible for your module to have been
imported by *anyone* by name alone, then you are stuck: either you break
them or you live with the incorrect granularity (which just isn't an
option in the conflict scenarios). Not a happy choice.

Originally, I had proposed to do away with import-by-module altogether,
both to avoid this problem and to eliminate the conceptual disconnect.
Your code does not today contain import statements that name *jars*, it
names packages and/or specific classes in those packages. Why invent a
new system that takes such a large step backwards?

The answer is simply convenience. Imagine a module that contains 100
packages and it is obvious that writing a single import statement is far
easier than discovering and explicitly writing all the package imports.
Yes, IDEs will likely mostly eliminate this issue, but it still makes
sense to be able to do this by hand.

This proposal is an attempt to maintain the convenience while adding the
crucial ability to safely refactor: step 1b is the central idea.

// Bryan

Re: Goals was: Re: Supporting OSGi Bundles in the Java Module System

2008-06-10T17:53:47Z

Hi Adrian,

Adrian Brock wrote:

> On Mon, 2008-04-28 at 20:16 -0700, Stanley M. Ho wrote:
>> 2. Requirements
>> 1. It shall be possible for an OSGi container to implement the
>> Java Module System.
>> 2. It shall be possible for a JAM module to express an import
>> dependency on any Module Definition in any Java Module System.
>
> These goals appear to have changed from the ones we agreed?
> I am misreading it or is this document only expressing one side of
> the equation?

This document only addresses how to use OSGi bundles in the Java Module
System (i.e. the original goal #1).

The Java Module System already provides reflective API that allows the
OSGi framework to make use of JAM modules (i.e. the original goal #2) if
the framework implementation chooses to provide the support. That said,
defining the semantic and syntax of how an OSGi bundle should import a
JAM module in the OSGi framework is actually outside the scope of 277,
thus it is not covered in this document.

I'm going to be offline very soon. I will follow up the remaining issues
you raised tomorrow.

- Stanley

Re: Exported Classes and OSGi Re: Supporting OSGi Bundles in the Java Module System

2008-06-10T17:10:36Z

Hi Adrian,

Adrian Brock wrote:

> On Mon, 2008-04-28 at 20:16 -0700, Stanley M. Ho wrote:
>> 3.2 ModuleDefinition class
>> Two new methods are added in the ModuleDefinition class to return the
>> exported packages and member packages respectively. The export and
>> member definitions contained in the OSGi metadata are in package
>> granularity. In addition, a new PackageDefinition class is added to
>> allow an OSGi bundle to expose the metadata for an exported package.
>> This is required to meet the requirements (1) and (2).
>> ModuleDefinition class:
>> public abstract Set<PackageDefinition> getExportedPackageDefinitions();
>> public abstract Set<PackageDefinition> getMemberPackageDefinitions();
>>
>> PackageDefinition class:
>> public abstract class PackageDefinition {
>> public String getName();
>> public Version getVersion();
>> public Set<String> getAttributeNames();
>> public String getAttribute(String name);
>> }
>
> ModuleDefinition still refers to exported and member classes.
> Although this is probably more of an issue for 294/291 interoperation,
> I don't think 291 has such notions?

With modules in the language, only public types are accessible outside
their development modules. A deployment module system could leverage
this by considering public types (or a subset thereof) as 'exported',
and the granularity of exports can therefore be types or packages. The
JAM module system integrates tightly with development modules and it
will expose type-level exports, but other module systems can continue to
expose package-level exports. Thus, ModuleDefinition should support both
granularity levels. If a module system does not support type-level
exports (e.g. 291), the implementation of
ModuleDefinition.getExportedClasses()/getMemberClasses() could simply
throw UnsupportedOperationException.

- Stanley

Re: Supporting OSGi Bundles in the Java Module System

2008-06-10T17:00:38Z

Hi Michal,

Yes, we're working on an implementation. Alex and I are currently
working on the EDR2 spec. After the spec is ready for the EG to discuss
and review, we expect to make the implementation available to the EG.

- Stanley

Michal Cierniak wrote:

> Hi Stanley,
>
> Thank you for providing this proposal. I think that it looks quite
> reasonable. Once we have an implementation (are you working on one?),
> we should have a look at details like: what exactly the behavior for
> circular dependencies should be (I don't think there is a problem --
> it's just one of those things where it's possible to overlook details
> until you start to implement something). But I think that the overall
> proposal is a pretty good approach to this very important problem of
> interop.
>
> I'm curious what the OSGi experts think.
>
> Michal

Re: Supporting OSGi Bundles in the Java Module System

2008-06-10T05:21:04Z

> I'm curious what the OSGi experts think. > > Michal >
I am still waiting for EDR2 to become available. There are too many references to new EDR2 content to properly evaluate the somewhat thin OSGi interop proposal.

I have many issues with the interop proposal, but I can't be sure understand the interop proposal until I have EDR2 in hand.
--

BJ Hargrave
Senior Technical Staff Member, IBM
OSGi Fellow and CTO of the OSGi Alliance
hargrave@...

office: +1 386 848 1781
mobile: +1 386 848 3788

Exported Classes and OSGi Re: Supporting OSGi Bundles in the Java Module System

2008-06-10T04:57:13Z

On Mon, 2008-04-28 at 20:16 -0700, Stanley M. Ho wrote:

> 3.2 ModuleDefinition class
> Two new methods are added in the ModuleDefinition class to return the
> exported packages and member packages respectively. The export and
> member definitions contained in the OSGi metadata are in package
> granularity. In addition, a new PackageDefinition class is added to
> allow an OSGi bundle to expose the metadata for an exported package.
> This is required to meet the requirements (1) and (2).
> ModuleDefinition class:
> public abstract Set<PackageDefinition> getExportedPackageDefinitions();
> public abstract Set<PackageDefinition> getMemberPackageDefinitions();
>
> PackageDefinition class:
> public abstract class PackageDefinition {
> public String getName();
> public Version getVersion();
> public Set<String> getAttributeNames();
> public String getAttribute(String name);
> }

ModuleDefinition still refers to exported and member classes.
Although this is probably more of an issue for 294/291 interoperation,
I don't think 291 has such notions?
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

Re: Supporting OSGi Bundles in the Java Module System

2008-06-10T04:52:25Z

On Mon, 2008-04-28 at 20:16 -0700, Stanley M. Ho wrote:

> 3.1.2 Constraint Checking
> A new method ModuleSystem.getModules is added to allow a ModuleSystem
> implementation to instantiate Module instances for multiple
> ModuleDefinitions in the same resolution and also to enforce
> constraints specified in these Modules.
> ModuleSystem class:
> /**
> * Returns the list of Module instances for the imports
> * in the same resolution for the specified importer.
> * The returned Module instances are instantiated and initialized
> * using the algorithm specific to this ModuleSystem.
> *
> * This method is called by a ModuleSystem when initializing
> * a Module instance (importer) that imports Modules (imports)
> * from this ModuleSystem.
> */
> public List<Module> getModules(ModuleDefinition importer,
> List<ModuleDefinition> imports)
> throws ModuleInitializationException;
>
> OSGi allows to put constraints on the importer through the metadata
> for an exported package in another bundle. The OSGi module system can
> enforce the constraints on the importer in the implementation of this
> getModules method. The importer can be a ModuleDefinition from other
> ModuleSystem.
>
> For example, the Apache Xerces XML parser bundle in Example 1 of
> Section 1 exports the org.apache.xerces.parsers package whose export
> definition has a "use" constraint. This export definition puts a
> constraint on the importer of the org.apache.xerces.parsers package to
> use the org.apache.commons.logging package wired to the
> org.apache.xerces.parsers package in the same resolution. The 1.0
> version of the Wombat application imports the
> org.apache.xerces.parsers bundle and the org.apache.derby bundle. The
> org.apache.xerces.parsers package will get wired to the version 1.0.4
> logging package and the constraint is satisfied and thus it can be
> wired successfully.
>
> Let's say a new version of the Wombat application (say version 1.2) is
> updated and it depends on an additional Apache Commons Logging utility
> which is also an OSGi bundle.
>
> //
> // com/wombat/app/module-info.java
> //
> @Version("1.2")
> @ImportModules({
> @ImportModule(name="org.apache.xerces.parsers", version="2.6.6+")
> @ImportModule(name="org.apache.derby", version="10.0+")
> @ImportModule(name="org.apache.commons.logging", version="2.0+")
> })
> module com.wombat.app;
>
> The bundle manifest header for the Apache Commons Logging utility is:
> org.apache.commons.logging:
> Bundle-SymbolicName: org.apache.commons.logging
> Bundle-Version: 2.0
> Export-Package: org.apache.commons.logging; version=2.0
>
> The 1.2 version of the Wombat application imports the
> org.apache.commons.logging bundle that violates the constraint if the
> org.apache.xerces.parsers package is wired to the version 1.0.4
> logging package but the Wombat application requires the version 2.0
> logging package. The version 1.2 of the Wombat application should fail
> to initialize due to this constraint. In other words, the getModules()
> method of the OSGi module system should throw a
> ModuleInitializationException when the OSGi module system determines
> that the constraint is violated.

I assume the list of "importers" is intended to take the place
of Glynn's ResolutionContext?

On Mon, 2007-02-26 at 09:58 +0000, Glyn Normington wrote:
> Yes, if we want to support first class interoperation. Without
> something
> like a resolution context, I don't see how we can maximise sharing of
> dependencies between module systems. If we don't do that, we'll get
> all
> the usual symptoms of inconsistent class spaces, like class cast
> exceptions, broken loading constraints, etc.

Assuming that is its purpose, I think the api is incomplete?

>From Glynn's original examples what is needed is to not to just look at
direct imports from the same module system.
Instead, to maximise sharing and consistency it needs to know about
all other resolutions in a given top level resolution request.

I made a similar point on Glynn's original prototype:

On Mon, 2007-02-26 at 12:26 +0100, Adrian wrote:

> On Mon, 2007-02-26 at 09:58 +0000, Glyn Normington wrote:
> > So the default behaviour in AbstractWireFactory.create calls
> > createLocal first so that the local name space takes precedence over
> > those of other module systems. The rationale is that the current
> > module system is like a local scope and we don't want other module
> > systems coming along and subverting the value associated with a given
> > module name that is defined locally.
> >
>
> But equally, we want two module systems using the same module
> where possible.
>
> e.g. If we are configured to have both a 277/291 repository
> and both define log4j:1.2.8 then the default mechanism
> should choose one of them, NOT each module system
> prefers its own.

I made another point (but I can't find the original e-mail :-)
where I thought it probably wasn't enough to just pass
the modules in the current resolution process (i.e.
those just being resolved).

Instead, when resolving a module, there should be some
"ClassLoading Space" where the resolution can determine
all other modules that are already related due to
imports/exports (including transient ones).
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

Goals was: Re: Supporting OSGi Bundles in the Java Module System

2008-06-10T04:28:12Z

On Mon, 2008-04-28 at 20:16 -0700, Stanley M. Ho wrote:
> 2. Requirements
> 1. It shall be possible for an OSGi container to implement the
> Java Module System.
> 2. It shall be possible for a JAM module to express an import
> dependency on any Module Definition in any Java Module System.

These goals appear to have changed from the ones we agreed?
I am misreading it or is this document only expressing one side of
the equation?

On Fri, 2007-02-16 at 13:08 -0800, Stanley M. Ho wrote:

> Goals:
>
> 1. It shall be possible for JSR-277 modules to make use of modules from
> other module systems (e.g. JSR-291) (i.e. if other module systems expose
> their modules as JSR-277 modules through a repository implementation.)
>
> 2. It shall be possible for other module systems (e.g. JSR-291) to be
> implemented on top of the JSR-277 APIs such that the modules from other
> module system can make use of JSR-277 modules.
>

I originally read (2) as a rewording of Glynn's original
definition where the goal is to create a more peer orientated
delegation.

> On Fri, 2007-01-12 at 09:49 +0000, Glyn Normington wrote:
> > Michal Cierniak <cierniak@...> wrote on 09/01/2007 20:20:12:
> > Co[u]ld you write a one-paragraph outline of what "first class
> > interoperation with JSR 291" would mean?
>
> First class interoperation with JSR 291 means that JSR 291
> implementations
> should be implementable on top of Java 7 APIs such that JSR 291
> bundles can
> make effective use of JSR 277 modules and vice versa.
>
> (I guess JSR 291 modules residing in JSR 277 repositories would be
> helpful
> too, especially if anyone needs to create a hybrid application
> containing
> modules of both sorts.)

My critisms of using the JSR277 repositories to do the integration
included:

* There are already lots of orthogonal reasons for swapping
repository implementations

On Mon, 2007-02-26 at 12:26 +0100, Adrian wrote:

> So besides the peer/hierarchy argument there
> is also the problem that the repository is dealing
> with too many concerns.
>
> 1) ModuleDefinition construction
> 2) Model - parent/child or peer
> 3) QoS - what tools are available for the repository
> 4) Location - file system/url based, etc.
> 5) Delegation - links to standard repositories
> etc.

* Developing a full repository just to define a different archive
format is too much (e.g. legacy javaee deployments)

On Wed, 2007-02-28 at 17:12 +0100, Adrian wrote:
On Mon, 2007-02-26 at 18:05 -0800, Stanley M. Ho wrote:

> > If I understand you correctly, your primary concern is around
> requiring
> > other module systems which support 277 modules that are stored
> > differently (e.g. war, ear, etc.) to implement the classloading and
> > other functionalities from scratch. Is this accurate?
>
> Not just that, anybody that wants to integrate their own
> packaging format and construct module definition (wrappers)
> should be able to plugin without dictating
> or developing a full repository.
>
> The examples so far discussed are:
> OSGi modules
> JavaEE deployment packages

* The concern about hierarchical integration

On Wed, 2007-02-28 at 17:18 +0100, Adrian wrote:

> On Mon, 2007-02-26 at 18:05 -0800, Stanley M. Ho wrote:
> > Hi Adrian,
> >
> > > The hierarchical assumption of repositories makes it difficult
> > > to plugin peer modules, unless we are going to define
> > > a composite repository for this purpose.
> >
> > Can you give me an example of peer modules that you are concerned with?
> >
>
> It is really peer module systems.
>
> e.g. My repository is made up of modules from
>
> 1) a local JSR77 repository
> 2) an OSGi repository handled by my OSGi "module system"
> 3) Some JavaEE deployments handled by the appserver's "module system"
> 4) a link to a repository on the internet
>
> I want my JavaEE modules to be able to use OSGi modules
> and vice versa OSGi modules should be able to use my JavaEE modules
> (e.g. an appclient or a resource adapter)
>
> This requires a peer integration at the classloader and security
> level.

--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

Re: Supporting OSGi Bundles in the Java Module System

2008-06-09T23:39:11Z

Hi Stanley,

Thank you for providing this proposal. I think that it looks quite
reasonable. Once we have an implementation (are you working on one?),
we should have a look at details like: what exactly the behavior for
circular dependencies should be (I don't think there is a problem --
it's just one of those things where it's possible to overlook details
until you start to implement something). But I think that the overall
proposal is a pretty good approach to this very important problem of
interop.

I'm curious what the OSGi experts think.

Michal

On Mon, Apr 28, 2008 at 8:16 PM, Stanley M. Ho <Stanley.Ho@...> wrote:
> Dear 277 experts,
>
> The first attachment is a draft spec for supporting OSGi bundles in the Java
> Module System. This is based on the past EG discussion in particular the
> proposals and inputs from Glyn, Richard, and Bryan. This is a work in
> progress and is expected to evolve based on further inputs from this EG.
>