Proposal for managing gem dependencies in a Java + JRuby application

[Alon Salant]

Hey all,

I’ve seen a few posts on this mailing list and in the JRuby JIRA looking for gem management strategies that don’t rely on a local installation of JRuby and locally installed gems in a deployed Java + JRuby application.

The application model I am looking at is a Java application built by Maven that embeds JRuby scripts that have gem dependencies. I'd like to support a deployment model that doesn't require a local install of JRuby and the required gems.

The primary issue appears to be that Ruby and RubyGems are heavily file system dependent so are not friendly to having resources loaded from the classpath.

jruby-complete takes care of providing the JRuby and the core Ruby dependencies with a strategy where core Ruby libraries are extracted at runtime to the file system from the jar.

I have been experimenting with a similar strategy for managing gem dependencies and have a working implementation that I’d like to float out here for comment.

It goes something like this:

Package [name]-[version].gem files in the Java application classpath.
Either at application startup or on demand (by modifying Kernel::gem) find the desired gem if it is not already installed by scanning the root of the classpath, ala Spring’s PathMatchingResourcePatternResolver (http://static.springframework.org/spring/docs/2.0.x/api/org/springframework/core/io/support/PathMatchingResourcePatternResolver.html).
Extract and install the gem into an application-specific GEM_HOME using Gem::Installer.
Done.

This enables packaging and deploying gems as jars (the .gem file in the root of a jar) and managing these dependencies with Maven, including transitive dependencies among gems.

Goldspike (Rails on Tomcat) uses a similar strategy by bundling expanded gems in WEB-INF and setting GEM_HOME to that location.

I have a version of this working that first extracts the .gem file to file system from the classpath by reading from a classpath resource InputStream. I noticed that Gem::Format has a method for reading from an IO stream (Gem::Format.from_io). It would be great to skip the intermediate extraction step by using this method. I ran into trouble figuring out how to use the InputStream from the classloader as an IO object in Ruby. I tried creating an org.jruby.RubyIO object from the InputStream but get an error when calling Gem::Format.from_io with the created object:

  /lib/ruby/site_ruby/1.8/rubygems/package.rb:411:in `new': Illegal seek (Errno::ESPIPE)


Comments on the overall strategy? If desired by others I’ll definitely share the end result.

Suggestions on the IO issue?

Thanks,

Alon

[Charles Oliver Nutter-2]

I am interested in hearing more! What do you think about the  
feasibility of making gems run entirely from within the archive? There  
are two possible approaches: make JRuby's imply of File work against  
URL based resources (hard); or make RubyGems itself be archive-aware.

Either way I am interested on seeing what you have!

- Charlie

On Dec 4, 2007, at 16:47, Alon Salant <alon@...> wrote:

>
> Hey all,
>
> I’ve seen a few posts on this mailing list and in the JRuby JIRA loo
> king for
> gem management strategies that don’t rely on a local installation of
>  JRuby
> and locally installed gems in a deployed Java + JRuby application.
>
> The application model I am looking at is a Java application built by  
> Maven
> that embeds JRuby scripts that have gem dependencies. I'd like to  
> support a
> deployment model that doesn't require a local install of JRuby and the
> required gems.
>
> The primary issue appears to be that Ruby and RubyGems are heavily  
> file
> system dependent so are not friendly to having resources loaded from  
> the
> classpath.
>
> jruby-complete takes care of providing the JRuby and the core Ruby
> dependencies with a strategy where core Ruby libraries are extracted  
> at
> runtime to the file system from the jar.
>
> I have been experimenting with a similar strategy for managing gem
> dependencies and have a working implementation that I’d like to floa
> t out
> here for comment.
>
> It goes something like this:
>
> Package [name]-[version].gem files in the Java application classpath.
> Either at application startup or on demand (by modifying  
> Kernel::gem) find
> the desired gem if it is not already installed by scanning the root  
> of the
> classpath, ala Spring’s PathMatchingResourcePatternResolver
> (http://static.springframework.org/spring/docs/2.0.x/api/org/springframework/core/io/support/PathMatchingResourcePatternResolver.html 
> ).
> Extract and install the gem into an application-specific GEM_HOME  
> using
> Gem::Installer.
> Done.
>
> This enables packaging and deploying gems as jars (the .gem file in  
> the root
> of a jar) and managing these dependencies with Maven, including  
> transitive
> dependencies among gems.
>
> Goldspike (Rails on Tomcat) uses a similar strategy by bundling  
> expanded
> gems in WEB-INF and setting GEM_HOME to that location.
>
> I have a version of this working that first extracts the .gem file  
> to file
> system from the classpath by reading from a classpath resource  
> InputStream.
> I noticed that Gem::Format has a method for reading from an IO stream
> (Gem::Format.from_io). It would be great to skip the intermediate  
> extraction
> step by using this method. I ran into trouble figuring out how to  
> use the
> InputStream from the classloader as an IO object in Ruby. I tried  
> creating
> an org.jruby.RubyIO object from the InputStream but get an error when
> calling Gem::Format.from_io with the created object:
>
>  /lib/ruby/site_ruby/1.8/rubygems/package.rb:411:in `new': Illegal  
> seek
> (Errno::ESPIPE)
>
>
> Comments on the overall strategy? If desired by others I’ll definite
> ly share
> the end result.
>
> Suggestions on the IO issue?
>
> Thanks,
>
> Alon
>
>
> --
> View this message in context: http://www.nabble.com/Proposal-for-managing-gem-dependencies-in-a-Java-%2B-JRuby-application-tf4946209.html#a14161350
> Sent from the JRuby - Dev mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list please visit:
>
>    http://xircles.codehaus.org/manage_email
>

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Nick Sieger-2]

On 12/4/07, Alon Salant <alon@...> wrote:

>
> Hey all,
>
> I've seen a few posts on this mailing list and in the JRuby JIRA looking for
> gem management strategies that don't rely on a local installation of JRuby
> and locally installed gems in a deployed Java + JRuby application.
>
> The application model I am looking at is a Java application built by Maven
> that embeds JRuby scripts that have gem dependencies. I'd like to support a
> deployment model that doesn't require a local install of JRuby and the
> required gems.
>
> The primary issue appears to be that Ruby and RubyGems are heavily file
> system dependent so are not friendly to having resources loaded from the
> classpath.
>
> jruby-complete takes care of providing the JRuby and the core Ruby
> dependencies with a strategy where core Ruby libraries are extracted at
> runtime to the file system from the jar.

This is primarily because of the
rubygems-needs-to-run-from-the-filesystem issue. It's either done the
first time a gem install is attempted with the complete jar, or when
requested.

> I have been experimenting with a similar strategy for managing gem
> dependencies and have a working implementation that I'd like to float out
> here for comment.
>
> It goes something like this:
>
> Package [name]-[version].gem files in the Java application classpath.
> Either at application startup or on demand (by modifying Kernel::gem) find
> the desired gem if it is not already installed by scanning the root of the
> classpath, ala Spring's PathMatchingResourcePatternResolver
> (http://static.springframework.org/spring/docs/2.0.x/api/org/springframework/core/io/support/PathMatchingResourcePatternResolver.html).
> Extract and install the gem into an application-specific GEM_HOME using
> Gem::Installer.
> Done.

This alone seems like it could be worthwhile, you simply will need to
patch Kernel::gem to do the extraction and the gem install (see
src/jruby/commands.rb and src/jruby/extract).

> This enables packaging and deploying gems as jars (the .gem file in the root
> of a jar) and managing these dependencies with Maven, including transitive
> dependencies among gems.

Well, not sure about Maven for managing gems, Rubygems seems to do
that just fine. As long as your solution doesn't require Maven, it
should be fine. (You of course are welcome to use whatever tools you
want for yourself.)

>
> Goldspike (Rails on Tomcat) uses a similar strategy by bundling expanded
> gems in WEB-INF and setting GEM_HOME to that location.
>
> I have a version of this working that first extracts the .gem file to file
> system from the classpath by reading from a classpath resource InputStream.
> I noticed that Gem::Format has a method for reading from an IO stream
> (Gem::Format.from_io). It would be great to skip the intermediate extraction
> step by using this method. I ran into trouble figuring out how to use the
> InputStream from the classloader as an IO object in Ruby. I tried creating
> an org.jruby.RubyIO object from the InputStream but get an error when
> calling Gem::Format.from_io with the created object:
>
>   /lib/ruby/site_ruby/1.8/rubygems/package.rb:411:in `new': Illegal seek
> (Errno::ESPIPE)

Looks like Rubygems wants to be able to seek through the package, but
when creating RubyIO from an InputStream, it makes the resulting IO
unseekable since we don't know whether the underlying stream supports
it.

Unfortunately we don't make it easy to register a seekable stream. A
patch to RubyIO might be necessary in order to support this.

Feel free to open a JIRA (http://jira.codehaus.org/browse/JRUBY) and
attach patches there.

Cheers,
/Nick

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Alon Salant]

Nick Sieger-2 wrote:

> Package [name]-[version].gem files in the Java application classpath.
> Either at application startup or on demand (by modifying Kernel::gem) find
> the desired gem if it is not already installed by scanning the root of the
> classpath, ala Spring's PathMatchingResourcePatternResolver
> (http://static.springframework.org/spring/docs/2.0.x/api/org/springframework/core/io/support/PathMatchingResourcePatternResolver.html).
> Extract and install the gem into an application-specific GEM_HOME using
> Gem::Installer.
> Done.

This alone seems like it could be worthwhile, you simply will need to
patch Kernel::gem to do the extraction and the gem install (see
src/jruby/commands.rb and src/jruby/extract).

Exactly. I'm thinking of something like:

require 'jrubygems'
gem 'Foo'

jrubygems.rb would require rubygems and patch Kernel::gem to find and install the gem from the classpath if it is not already installed.

Nick Sieger-2 wrote:

> This enables packaging and deploying gems as jars (the .gem file in the root
> of a jar) and managing these dependencies with Maven, including transitive
> dependencies among gems.

Well, not sure about Maven for managing gems, Rubygems seems to do
that just fine. As long as your solution doesn't require Maven, it
should be fine. (You of course are welcome to use whatever tools you
want for yourself.)

Agreed. This would be valuable for applications that are primarily Java using JRuby where the build and packaging is happening in Java using Maven. The .gem files need some way to get on the classpath and for Java applications this would work well. Accompanying Maven tools could fetch .gem files from RubyForge and package/install/deploy them as jars for use in this kind of application.

Nick Sieger-2 wrote:

> I ran into trouble figuring out how to use the
> InputStream from the classloader as an IO object in Ruby. I tried creating
> an org.jruby.RubyIO object from the InputStream but get an error when
> calling Gem::Format.from_io with the created object:
>
>   /lib/ruby/site_ruby/1.8/rubygems/package.rb:411:in `new': Illegal seek
> (Errno::ESPIPE)

Looks like Rubygems wants to be able to seek through the package, but
when creating RubyIO from an InputStream, it makes the resulting IO
unseekable since we don't know whether the underlying stream supports
it.

Unfortunately we don't make it easy to register a seekable stream. A
patch to RubyIO might be necessary in order to support this.

Feel free to open a JIRA (http://jira.codehaus.org/browse/JRUBY) and
attach patches there.

Is there a hard way to register a seekable stream given a Java InputStream?

Alon

[Alon Salant]

Charles Oliver Nutter-2 wrote:

I am interested in hearing more! What do you think about the  
feasibility of making gems run entirely from within the archive? There  
are two possible approaches: make JRuby's imply of File work against  
URL based resources (hard); or make RubyGems itself be archive-aware.

Either way I am interested on seeing what you have!

I'm taking the more direct approach of making RubyGems be archive-aware only at bootstrap time to find the .gem and unpack it to the filesystem. This makes the solution pretty straight forward. Of course this approach will only work for gems that don't require local compilation and such. So pure Ruby or Ruby + included jar gems should work fine.

I'll share soon. I'm honestly still getting my head wrapped around how much should be written in Java and used from Ruby v. just written in Ruby using Java APIs. What I have now could be simplified by reducing the Java support classes I have written.

Alon

[Alon Salant]

I've gotten this to a point that is working well for me.

https://svn.carbonfive.com/public/carbonfive/jruby/jrubygems/trunk

JRubyGems extends RubyGems to support loading gem dependencies from the Java classpath.

There is a java component built with Maven 2 and a Ruby component structured as a gem. The gem was bootsrapped with newgem so there's a lot of template files in there. lib/ and tests/ have the meat, particularly test/test_jrubygems.rb. The artifact of the Java component is built and in ruby/lib so you don't have to build it to check this out.

Here's what I've done:

1. Gem.activate searches the Java classpath for the .gem file if it is not already installed.
2. If found, extract the .gem to the filesystem and install it with Gem:Installer.
3. If the gem has additional dependencies, steps 1 & 2 will be run for each dependency before the gem is installed.

That's about it.

A few comments:

In order to get to the meat of the job, I used Spring's PathMatchingResourcePatternResolver to do the fancy classpath searching stuff. So this implementation depends on spring-core for now (and therefore commons-logging) but this is temporary. Implementing the classpath searching myself is coming soon.

The gem installation location is determined using the original gem methods. This means that if JRuby is running from jruby-complete extracted to USER_HOME/.jruby, the gems will be installed in there too. You can set ENV['GEM_HOME'] to set another location.

.gem files must be in the classpath under the path prefix /gems. These .gem files can be in jars or directories in the classpath. They can't be in the root because are issues searching for resources in the root of a classpath that has both file system directories and jars. This originates from a limitation in the JDK's ClassLoader.getResources method which only returns file system locations for a passed-in empty String (indicating potential roots to search).

If we can figure out how to use Gem::Format.from_io to read from a InputStream from a classpath resource URL, there are opportunities for much slicker implementations that skip the initial step of extracting the .gem file from the classpath.

There are a couple ways to package this library for use. It doesn't make sense to package it as a gem because of the obvious bootstrapping issues. I think the thing to do is to include the jrubygems.rb script in the root of the jrubygems jar file. If the jar is in your application's classpath you would simply use it with:

require 'jrubygems'
gem 'activerecord-jdbc-adapter'

If this implementation turns out to be useful to the JRuby core, it could be packaged with jruby-complete and required internally to extend RubyGems' behavior for JRuby applications. This seems most desirable since it would add classpath loading support for JRuby applications but not change the behavior of regular Ruby applications at all.

I'm looking forward to feedback.

Alon

[Thomas E Enebo]

This sounds great to me.  Once you get rid of Spring as a dependency I
would expect it to be downright attractive.  If we automatically
included this as part of jruby-complete I would be mildly worried
about performance of loading since it would require a classpath
search.  In the past we refactored load/require to make sure we were
not doing that searching first since it was really slow and linear to
the amount of classes/jars it needed to search through.

Making from_io work would be even better of course.

-Tom

On Dec 7, 2007 12:23 PM, Alon Salant <alon@...> wrote:

>
> I've gotten this to a point that is working well for me.
>
> https://svn.carbonfive.com/public/carbonfive/jruby/jrubygems/trunk
>
> JRubyGems extends RubyGems to support loading gem dependencies from the Java
> classpath.
>
> There is a java component built with Maven 2 and a Ruby component structured
> as a gem. The gem was bootsrapped with newgem so there's a lot of template
> files in there. lib/ and tests/ have the meat, particularly
> test/test_jrubygems.rb. The artifact of the Java component is built and in
> ruby/lib so you don't have to build it to check this out.
>
> Here's what I've done:
>
> 1. Gem.activate searches the Java classpath for the .gem file if it is not
> already installed.
> 2. If found, extract the .gem to the filesystem and install it with
> Gem:Installer.
> 3. If the gem has additional dependencies, steps 1 & 2 will be run for each
> dependency before the gem is installed.
>
> That's about it.
>
> A few comments:
>
> In order to get to the meat of the job, I used Spring's
> PathMatchingResourcePatternResolver to do the fancy classpath searching
> stuff. So this implementation depends on spring-core for now (and therefore
> commons-logging) but this is temporary. Implementing the classpath searching
> myself is coming soon.
>
> The gem installation location is determined using the original gem methods.
> This means that if JRuby is running from jruby-complete extracted to
> USER_HOME/.jruby, the gems will be installed in there too. You can set
> ENV['GEM_HOME'] to set another location.
>
> .gem files must be in the classpath under the path prefix /gems. These .gem
> files can be in jars or directories in the classpath. They can't be in the
> root because are issues searching for resources in the root of a classpath
> that has both file system directories and jars. This originates from a
> limitation in the JDK's ClassLoader.getResources method which only returns
> file system locations for a passed-in empty String (indicating potential
> roots to search).
>
> If we can figure out how to use Gem::Format.from_io to read from a
> InputStream from a classpath resource URL, there are opportunities for much
> slicker implementations that skip the initial step of extracting the .gem
> file from the classpath.
>
> There are a couple ways to package this library for use. It doesn't make
> sense to package it as a gem because of the obvious bootstrapping issues. I
> think the thing to do is to include the jrubygems.rb script in the root of
> the jrubygems jar file. If the jar is in your application's classpath you
> would simply use it with:
>
> require 'jrubygems'
> gem 'activerecord-jdbc-adapter'
>
> If this implementation turns out to be useful to the JRuby core, it could be
> packaged with jruby-complete and required internally to extend RubyGems'
> behavior for JRuby applications. This seems most desirable since it would
> add classpath loading support for JRuby applications but not change the
> behavior of regular Ruby applications at all.
>
> I'm looking forward to feedback.
>
> Alon
> --
> View this message in context: http://www.nabble.com/Proposal-for-managing-gem-dependencies-in-a-Java-%2B-JRuby-application-tf4946209.html#a14217742
>
> Sent from the JRuby - Dev mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>

--
Blog: http://www.bloglines.com/blog/ThomasEEnebo
Email: enebo@... , tom.enebo@...

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email

[Alon Salant]

Thomas E Enebo wrote:

If we automatically
included this as part of jruby-complete I would be mildly worried
about performance of loading since it would require a classpath
search.

Note that this would only happen if the gem was not yet loaded or installed on the file system.

Alon

[Alon Salant]

Thomas E Enebo wrote:

This sounds great to me.  Once you get rid of Spring as a dependency I
would expect it to be downright attractive.

Spring dependency is now gone.

Also, Maven builds the jar with jrubygems.rb in the root of the jar so the jar is all that is required to use this extension to RubyGems.

Alon

[Stephen Bannasch-3]

At 5:00 PM -0800 12/7/07, Alon Salant wrote:

>Thomas E Enebo wrote:
>>
>> This sounds great to me.  Once you get rid of Spring as a dependency I
>> would expect it to be downright attractive.
>>
>
>Spring dependency is now gone.
>
>Also, Maven builds the jar with jrubygems.rb in the root of the jar so the
>jar is all that is required to use this extension to RubyGems.
>
>Alon

Alon, this sounds great.

You might want to send a note to the rubygems-developers@... list describing what you are working on.

---------------------------------------------------------------------
To unsubscribe from this list please visit:

    http://xircles.codehaus.org/manage_email