Goals was: Re: Supporting OSGi Bundles in the Java Module System

[Adrian Brock-4]

On Mon, 2008-04-28 at 20:16 -0700, Stanley M. Ho wrote:
> 2. Requirements
>      1. It shall be possible for an OSGi container to implement the
>         Java Module System.
>      2. It shall be possible for a JAM module to express an import
>         dependency on any Module Definition in any Java Module System.

These goals appear to have changed from the ones we agreed?
I am misreading it or is this document only expressing one side of
the equation?

On Fri, 2007-02-16 at 13:08 -0800, Stanley M. Ho wrote:

> Goals:
>
> 1. It shall be possible for JSR-277 modules to make use of modules from
> other module systems (e.g. JSR-291) (i.e. if other module systems expose
> their modules as JSR-277 modules through a repository implementation.)
>
> 2. It shall be possible for other module systems (e.g. JSR-291) to be
> implemented on top of the JSR-277 APIs such that the modules from other
> module system can make use of JSR-277 modules.
>

I originally read (2) as a rewording of Glynn's original
definition where the goal is to create a more peer orientated
delegation.

> On Fri, 2007-01-12 at 09:49 +0000, Glyn Normington wrote:
> > Michal Cierniak <cierniak@...> wrote on 09/01/2007 20:20:12:
> > Co[u]ld you write a one-paragraph outline of what "first class
> > interoperation with JSR 291" would mean?
>
> First class interoperation with JSR 291 means that JSR 291
> implementations
> should be implementable on top of Java 7 APIs such that JSR 291
> bundles can
> make effective use of JSR 277 modules and vice versa.
>
> (I guess JSR 291 modules residing in JSR 277 repositories would be
> helpful
> too, especially if anyone needs to create a hybrid application
> containing
> modules of both sorts.)

My critisms of using the JSR277 repositories to do the integration
included:

* There are already lots of orthogonal reasons for swapping
repository implementations

On Mon, 2007-02-26 at 12:26 +0100, Adrian wrote:

> So besides the peer/hierarchy argument there
> is also the problem that the repository is dealing
> with too many concerns.
>
> 1) ModuleDefinition construction
> 2) Model - parent/child or peer
> 3) QoS - what tools are available for the repository
> 4) Location - file system/url based, etc.
> 5) Delegation - links to standard repositories
> etc.

* Developing a full repository just to define a different archive
format is too much (e.g. legacy javaee deployments)

On Wed, 2007-02-28 at 17:12 +0100, Adrian wrote:
On Mon, 2007-02-26 at 18:05 -0800, Stanley M. Ho wrote:

> > If I understand you correctly, your primary concern is around
> requiring
> > other module systems which support 277 modules that are stored
> > differently (e.g. war, ear, etc.) to implement the classloading and
> > other functionalities from scratch. Is this accurate?
>
> Not just that, anybody that wants to integrate their own
> packaging format and construct module definition (wrappers)
> should be able to plugin without dictating
> or developing a full repository.
>
> The examples so far discussed are:
> OSGi modules
> JavaEE deployment packages

* The concern about hierarchical integration

On Wed, 2007-02-28 at 17:18 +0100, Adrian wrote:

> On Mon, 2007-02-26 at 18:05 -0800, Stanley M. Ho wrote:
> > Hi Adrian,
> >
> > > The hierarchical assumption of repositories makes it difficult
> > > to plugin peer modules, unless we are going to define
> > > a composite repository for this purpose.
> >
> > Can you give me an example of peer modules that you are concerned with?
> >
>
> It is really peer module systems.
>
> e.g. My repository is made up of modules from
>
> 1) a local JSR77 repository
> 2) an OSGi repository handled by my OSGi "module system"
> 3) Some JavaEE deployments handled by the appserver's "module system"
> 4) a link to a repository on the internet
>
> I want my JavaEE modules to be able to use OSGi modules
> and vice versa OSGi modules should be able to use my JavaEE modules
> (e.g. an appclient or a resource adapter)
>
> This requires a peer integration at the classloader and security
> level.

--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

[Stanley M. Ho]

Hi Adrian,

Adrian Brock wrote:

> On Mon, 2008-04-28 at 20:16 -0700, Stanley M. Ho wrote:
>> 2. Requirements
>>      1. It shall be possible for an OSGi container to implement the
>>         Java Module System.
>>      2. It shall be possible for a JAM module to express an import
>>         dependency on any Module Definition in any Java Module System.
>
> These goals appear to have changed from the ones we agreed?
> I am misreading it or is this document only expressing one side of
> the equation?

This document only addresses how to use OSGi bundles in the Java Module
System (i.e. the original goal #1).

The Java Module System already provides reflective API that allows the
OSGi framework to make use of JAM modules (i.e. the original goal #2) if
the framework implementation chooses to provide the support. That said,
defining the semantic and syntax of how an OSGi bundle should import a
JAM module in the OSGi framework is actually outside the scope of 277,
thus it is not covered in this document.

I'm going to be offline very soon. I will follow up the remaining issues
you raised tomorrow.

- Stanley

[Stanley M. Ho]

Hi Adrian,

Adrian Brock wrote:
> My critisms of using the JSR277 repositories to do the integration
> included:

I want to better understand your concerns around developing
repositories, since I believe some of your concerns have already been
addressed in the draft API.

> * There are already lots of orthogonal reasons for swapping
> repository implementations
>
> On Mon, 2007-02-26 at 12:26 +0100, Adrian wrote:
>> So besides the peer/hierarchy argument there
>> is also the problem that the repository is dealing
>> with too many concerns.
>>
>> 1) ModuleDefinition construction

The draft API provides a Modules.newJamModuleDefinition() method to
construct a ModuleDefinition from JAM's metadata and ModuleContent. The
ModuleContent abstraction enables a module archive to be stored in the
repository in any form under the cover. This should make it easier for a
repository implementation to construct ModuleDefinitions from JAM files,
or from other archive formats (as long as the metadata information of a
module can be expressed as JAM's metadata, and the content of the module
can be exposed through the ModuleContent abstraction).

In your original use case in JavaEE, the appserver wants to use the
module system for "plain old module" wiring but also wants to define its
own "modules" for wars, ears, etc., and some of which looks nothing like
jars/jams in structure. Do you think the draft API and the ModuleContent
abstraction are sufficient to address the ModuleDefinition construction
issue you were concerned with?

>> 2) Model - parent/child or peer

I think you meant the module system inteorp model, not the repository
delegation model. Right?


>> 3) QoS - what tools are available for the repository

The Java Module System implementation will come with a standard tool to
manage module archives in the repository, e.g. install/uninstall, etc.
Is there any particular aspect of the tools support that you are
concerned with but currently not possible in the draft API?

>> 4) Location - file system/url based, etc.

This is handled by the ModuleContent abstraction described above.

> * Developing a full repository just to define a different archive
> format is too much (e.g. legacy javaee deployments)

I would like to know what you think about the draft API to see if it is
still too much to develop a repository.

- Stanley

[Adrian Brock-4]

On Wed, 2008-06-11 at 22:31 -0700, Stanley M. Ho wrote:

> Hi Adrian,
>
> Adrian Brock wrote:
> > My critisms of using the JSR277 repositories to do the integration
> > included:
>
> I want to better understand your concerns around developing
> repositories, since I believe some of your concerns have already been
> addressed in the draft API.
>

The comments I included were from discussions (a year ago)
before the current api changes, so some parts maybe out-of-date.

> > * There are already lots of orthogonal reasons for swapping
> > repository implementations
> >
> > On Mon, 2007-02-26 at 12:26 +0100, Adrian wrote:
> >> So besides the peer/hierarchy argument there
> >> is also the problem that the repository is dealing
> >> with too many concerns.
> >>
> >> 1) ModuleDefinition construction
>
> The draft API provides a Modules.newJamModuleDefinition() method to
> construct a ModuleDefinition from JAM's metadata and ModuleContent. The
> ModuleContent abstraction enables a module archive to be stored in the
> repository in any form under the cover. This should make it easier for a
> repository implementation to construct ModuleDefinitions from JAM files,
> or from other archive formats (as long as the metadata information of a
> module can be expressed as JAM's metadata, and the content of the module
> can be exposed through the ModuleContent abstraction).
>
> In your original use case in JavaEE, the appserver wants to use the
> module system for "plain old module" wiring but also wants to define its
> own "modules" for wars, ears, etc., and some of which looks nothing like
> jars/jams in structure. Do you think the draft API and the ModuleContent
> abstraction are sufficient to address the ModuleDefinition construction
> issue you were concerned with?

I see a number of possible use cases here
(I'll continue with appserver example):

1) Fully JSR 277

The appserver and JavaEE specs are updated to fully understand
JSR277/294. In this case, the JavaEE deployment process would
probably make use of a JSR 277 repository.

2) Partial JSR 277

The appserver knows about JSR277/294 but the JavaEE specs
haven't been updated (or they are legacy deployments).

In this case the appserver could try to adapt the deployments
to conform to the ModuleDefinition/Content.

In this case, there probably wouldn't be a real repository
implementation, instead the "repository" would be dummy.
It would just be serving as an adapter from the old deployment packaging
and metadata to the new one at runtime.

3) Mixed

Similar to 2, but the opposite.
The appserver knows about JSR277/294 and wants to support
.jam files as well as .jar files in JavaEE deployments.
i.e. .jam files would be unpacked from a .jar EAR deployment
and then added to the module system runtime.

Conclusion:

I expect there are other possible uses.

The question is whether Repository is the best api
for cases (2) and (3). i.e. we implement a dummy
repository with a dummy "URL" for the source
or whether there should be a more abstract parent
class (I called it ModuleSource) that doesn't
assume a physical repository behind the scenes.

At least for cases (2) and (3) all we really want
to do is to PUSH a ModuleDefinition/Content to the
jsr 277 module system such that we can get back
a Module/Classloader with the imports resolved.
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

[Adrian Brock-4]

On Wed, 2008-06-11 at 22:31 -0700, Stanley M. Ho wrote:
> >> 2) Model - parent/child or peer
>
> I think you meant the module system inteorp model, not the repository
> delegation model. Right?
>

I don't see the difference? Hierarchical or Peer delegation
policies between repositories are orthogonal choices to which
module system backs that repository.

It's precisely this orthogonality that causes me to think
that we can't just say "create a new repository implementation"
to change a particular rule/policy.

There are ~ N^2 choices of overrides to what the default
repositories do.

e.g. (probably not a good example since the osgi
module system would use OBR?)

1) A simple file repository containing jam files
2) The same containing osgi bundles
3) A remote repository containing jam files
4) A remote repository contains osgi bundles

And those are just two of the many
orthogonal choices 2^2 = 4
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

[Adrian Brock-4]

On Wed, 2008-06-11 at 22:31 -0700, Stanley M. Ho wrote:

> >> 3) QoS - what tools are available for the repository
>
> The Java Module System implementation will come with a standard tool to
> manage module archives in the repository, e.g. install/uninstall, etc.
> Is there any particular aspect of the tools support that you are
> concerned with but currently not possible in the draft API?
>
> >> 4) Location - file system/url based, etc.
>
> This is handled by the ModuleContent abstraction described above.
>
> > * Developing a full repository just to define a different archive
> > format is too much (e.g. legacy javaee deployments)
>
> I would like to know what you think about the draft API to see if it is
> still too much to develop a repository.

The question is how easy is it to do all this overriding of
repository implementations to change behaviour and
not loose some aspect of the behaviour.

e.g. To programmatically create a module like on the other thread.

The tools of a repository are not defined by the spec,
and there is no repository.install(ModuleContent)
instead I would create a (possibly dummy) URL backed
by a jam file stream (or whatever format the repository understands).

But does that really do what I want? If I'm mapping a legacy
file to a module definition at runtime, I probably don't
want to install it in the JDK  repository implementation
since it will copy it to some disk location.

The next time I reboot my runtime, I would have to delete it
and recreate it otherwise it might be stale.

This is all so I can enable use the tools that come with the
JDK repository implementation and possibly other behaviour
like Bryan's resolving Module imports to Package imports.
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx

[Stanley M. Ho]

Adrian Brock wrote:

> I see a number of possible use cases here
> (I'll continue with appserver example):
>
> 1) Fully JSR 277
>
> The appserver and JavaEE specs are updated to fully understand
> JSR277/294. In this case, the JavaEE deployment process would
> probably make use of a JSR 277 repository.
>
> 2) Partial JSR 277
>
> The appserver knows about JSR277/294 but the JavaEE specs
> haven't been updated (or they are legacy deployments).
>
> In this case the appserver could try to adapt the deployments
> to conform to the ModuleDefinition/Content.
>
> In this case, there probably wouldn't be a real repository
> implementation, instead the "repository" would be dummy.
> It would just be serving as an adapter from the old deployment packaging
> and metadata to the new one at runtime.
>
> 3) Mixed
>
> Similar to 2, but the opposite.
> The appserver knows about JSR277/294 and wants to support
> .jam files as well as .jar files in JavaEE deployments.
> i.e. .jam files would be unpacked from a .jar EAR deployment
> and then added to the module system runtime.
>
> Conclusion:
>
> I expect there are other possible uses.
>
> The question is whether Repository is the best api
> for cases (2) and (3). i.e. we implement a dummy
> repository with a dummy "URL" for the source
> or whether there should be a more abstract parent
> class (I called it ModuleSource) that doesn't
> assume a physical repository behind the scenes.
>
> At least for cases (2) and (3) all we really want
> to do is to PUSH a ModuleDefinition/Content to the
> jsr 277 module system such that we can get back
> a Module/Classloader with the imports resolved.

In your example, if you want to create ModuleDefinitions from the .jar
files, there are two ways to achieve it:

1. a. Create a ModuleContent based on the content of the .jar file.
    b. Generate the JAM's metadata based on information in the .jar file.
    c. Construct a JAM's ModuleDefinition with the ModuleContent using
       Modules.newJamModuleDefinition(). The ModuleDefinition would be
       associated with the JAM module system.

2. a. Create a ModuleContent based on the content of the .jar file.
    b. Create your own ModuleDefinition with the ModuleContent and use
       the JAM module system as the associated module system. As long as
       your ModuleDefinition exposes the appropriate metadata information
       reflectively, the JAM module system would treat it like other
       JAM's ModuleDefinitions.

In either case, the JAM module system could be used to instantiate and
initialize Module/ClassLoader from the ModuleDefinition.


Creating a repository implementation is a different story. In the
current draft API, you do need to override a number of methods in
Repository in order to have a functional repository implementation. I
still think Repository is the api for module systems to find
ModuleDefinitions. The question is how we could change the API to make
it much easier for developers to create custom repository implementation.

Perhaps we should provide more useful default implementation in the
Repository's methods so developers only need to override very small
number of methods. Or perhaps what we really need is a repository
builder which allows developers to provide a set of ModuleDefinitions
and get back a basic repository implementation to use.

I'll look into the draft API to see if further simplifications can be made.

- Stanley

[Stanley M. Ho]

Adrian Brock wrote:

> The question is how easy is it to do all this overriding of
> repository implementations to change behaviour and
> not loose some aspect of the behaviour.
>
> e.g. To programmatically create a module like on the other thread.
>
> The tools of a repository are not defined by the spec,
> and there is no repository.install(ModuleContent)
> instead I would create a (possibly dummy) URL backed
> by a jam file stream (or whatever format the repository understands).
>
> But does that really do what I want? If I'm mapping a legacy
> file to a module definition at runtime, I probably don't
> want to install it in the JDK  repository implementation
> since it will copy it to some disk location.
>
> The next time I reboot my runtime, I would have to delete it
> and recreate it otherwise it might be stale.
>
> This is all so I can enable use the tools that come with the
> JDK repository implementation and possibly other behaviour
> like Bryan's resolving Module imports to Package imports.

As you mentioned in another thread, "all we really want to do is to PUSH
a ModuleDefinition/Content to the jsr 277 module system such that we can
get back a Module/Classloader with the imports resolved." I don't think
you really meant to change the behavior of the existing repositories
(e.g. bootstrap, etc.), but to make it easy to create a new repository
by extending from a stock implementation and override some of its
behavior. If I misunderstand you, please let me know.

The repository API in the current draft requires many methods to be
overridden in order to build a fully functional repository
implementation. The use case you mentioned is to build a simple
repository at runtime that all it does is to return ModuleDefinitions;
it doesn't has a real physical repository under the cover, and it
probably doesn't support install/uninstall and all the other methods. In
this case, I agreed that the API probably requires too much work for the
implementors, and I will look into this issue to see if some
simplifications can be made in the API.

- Stanley