Distribute python lib used in a java project

[Thomas Mortagne-2]

Hi,

I'm wondering what is the cleanest way distribute a war with a servlet
which is using some python lib. I want everything to work without
having to put the .py files in some standard python folder in the
system where the war is installed.

Here is my general more precise use case: I'm in a servlet java
project (XWiki) and i want to use Pygments to do some highlight on a
String and insert the result.

For now I'm packaging Pygments .py files in a jar and at runtime i'm
searching the full path to the Pygments .py files home (so a path
looking like jar:file:/path/to/pygments.jar!/Lib/) and I put it in
"python.home" with a System.setProperty("python.home",
pygmentsHomePath); before creating the PythonInterpreter. I'm using
Jython standalone jar to have all the standard python module already
automatically loaded.

It's working well but It looks a little hackish to me and i'm
wondering if depending of the application server security
configuration it might not work.

i was kind of hoping that Jython would scan all jars in the classpath
to find the .py files automatically but it seems i did not found the
right thing to enable or configure. The best would be to compile
Pygments in .class in a clean jar I guess but i could not find if
there was already a way to use the coming compiler ("clamp" if i
remember well).

Thanks,
--
Thomas Mortagne

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Jython-users mailing list
Jython-users@...
https://lists.sourceforge.net/lists/listinfo/jython-users

[Charlie Groves]

On Wed, Sep 2, 2009 at 10:21 AM, Thomas
Mortagne<thomas.mortagne@...> wrote:
> i was kind of hoping that Jython would scan all jars in the classpath
> to find the .py files automatically but it seems i did not found the
> right thing to enable or configure.

That should be enabled by default.  As of 2.5.0, Jython adds
__pyclasspath__/ to sys.path, and that serves as a PEP 302 import hook
to tell it to look for .py and $py.class files on the classpath.  As
long as you have a jar containing those .py files in its root on the
classpath, Jython should load them.  If that isn't the case, filing a
bug at bugs.jython.org would be appreciated.

> The best would be to compile
> Pygments in .class in a clean jar I guess but i could not find if
> there was already a way to use the coming compiler ("clamp" if i
> remember well).

As long as it's pure Python code like Pygments, nothing beyond
standard Jython is necessary.  clamp makes it to where Python classes
can be called directly by Java, which isn't necessary for this.

For compiling python modules, look at the compileall.py module in Lib.
 It can be invoked by Jython as a script, and it compiles a whole
directory tree of Python files.  You can then jar up the resulting
$py.class, and the same __pyclasspath__ hook will handle importing
them.

Charlie

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Jython-users mailing list
Jython-users@...
https://lists.sourceforge.net/lists/listinfo/jython-users

[Thomas Mortagne]

Hi,

On Fri, Sep 4, 2009 at 08:44, Charlie Groves<charlie.groves@...> wrote:

> On Wed, Sep 2, 2009 at 10:21 AM, Thomas
> Mortagne<thomas.mortagne@...> wrote:
>> i was kind of hoping that Jython would scan all jars in the classpath
>> to find the .py files automatically but it seems i did not found the
>> right thing to enable or configure.
>
> That should be enabled by default.  As of 2.5.0, Jython adds
> __pyclasspath__/ to sys.path, and that serves as a PEP 302 import hook
> to tell it to look for .py and $py.class files on the classpath.  As
> long as you have a jar containing those .py files in its root on the
> classpath, Jython should load them.  If that isn't the case, filing a
> bug at bugs.jython.org would be appreciated.

Ok for simple standalone .py files but something like Pygments has sub modules

- pygments
-- filters
-- formatters
-- lexers
-- styles

And i can't put all the .py files in the root since each folder
contains a __init__.py files (sorry for maybe dumb questions I'm
really not a Python expert).

>
>> The best would be to compile
>> Pygments in .class in a clean jar I guess but i could not find if
>> there was already a way to use the coming compiler ("clamp" if i
>> remember well).
>
> As long as it's pure Python code like Pygments, nothing beyond
> standard Jython is necessary.  clamp makes it to where Python classes
> can be called directly by Java, which isn't necessary for this.
>
> For compiling python modules, look at the compileall.py module in Lib.
>  It can be invoked by Jython as a script, and it compiles a whole
> directory tree of Python files.  You can then jar up the resulting
> $py.class, and the same __pyclasspath__ hook will handle importing
> them.

Ok great, exactly what i was searching for :)

But again what about the sub folders ?

>
> Charlie
>


Thanks,
--
Thomas Mortagne

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Jython-users mailing list
Jython-users@...
https://lists.sourceforge.net/lists/listinfo/jython-users

[Marcos Sánchez Provencio-3]

Hi there

(I use jython as extension language in my web project at work)

So, if I have understood, it would be enough just to build a
jython-libs.jar with the contents of Lib and just add one file to my web
app?

Why is not this jar included in the standard dist? Or is it? I seems
this is the best way to deal with the inclusion of the standard Lib...

El jue, 03-09-2009 a las 23:44 -0700, Charlie Groves escribió:
>
> That should be enabled by default.  As of 2.5.0, Jython adds
> __pyclasspath__/ to sys.path, and that serves as a PEP 302 import hook
> to tell it to look for .py and $py.class files on the classpath.  As
> long as you have a jar containing those .py files in its root on the
> classpath, Jython should load them.  If that isn't the case, filing a
> bug at bugs.jython.org would be appreciated.
>


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Jython-users mailing list
Jython-users@...
https://lists.sourceforge.net/lists/listinfo/jython-users

[Charlie Groves]

On Fri, Sep 4, 2009 at 12:24 AM, Thomas
Mortagne<thomas.mortagne@...> wrote:

> On Fri, Sep 4, 2009 at 08:44, Charlie Groves<charlie.groves@...> wrote:
>>
>> That should be enabled by default.  As of 2.5.0, Jython adds
>> __pyclasspath__/ to sys.path, and that serves as a PEP 302 import hook
>> to tell it to look for .py and $py.class files on the classpath.  As
>> long as you have a jar containing those .py files in its root on the
>> classpath, Jython should load them.  If that isn't the case, filing a
>> bug at bugs.jython.org would be appreciated.
>
> Ok for simple standalone .py files but something like Pygments has sub modules
>
> - pygments
> -- filters
> -- formatters
> -- lexers
> -- styles
>
> And i can't put all the .py files in the root since each folder
> contains a __init__.py files (sorry for maybe dumb questions I'm
> really not a Python expert).

I'm sorry, I wasn't clear about what I said to put at the root.  Put
that whole pygments directory structure in the jar file.  If you're in
the directory with pygments in it, "jar cf pygments.jar pygments"
would do the trick.

The same holds for the compiled version.  compileall.py will put the
$py.class files in the proper place in that directory structure.  If
you make the jar after that, the compiled versions will be used in
preference to the source versions.

Charlie

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Jython-users mailing list
Jython-users@...
https://lists.sourceforge.net/lists/listinfo/jython-users

[Charlie Groves]

2009/9/4 Marcos Sánchez Provencio <msanchez@...>:
> So, if I have understood, it would be enough just to build a
> jython-libs.jar with the contents of Lib and just add one file to my web
> app?

Yes.

> Why is not this jar included in the standard dist? Or is it? I seems
> this is the best way to deal with the inclusion of the standard Lib...

It isn't included, but there's no good reason for that.    The import
from classpath feature was addded relatively late in the development
of 2.5, so we didn't want to switch everything over at that point.

I agree that it'd be the easiest way to get the Lib files onto the
classpath, so we should look at it for 2.5.2 since we're adding
features there.

Charlie

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Jython-users mailing list
Jython-users@...
https://lists.sourceforge.net/lists/listinfo/jython-users

[Thomas Mortagne]

On Fri, Sep 4, 2009 at 09:41, Charlie Groves<charlie.groves@...> wrote:

> On Fri, Sep 4, 2009 at 12:24 AM, Thomas
> Mortagne<thomas.mortagne@...> wrote:
>> On Fri, Sep 4, 2009 at 08:44, Charlie Groves<charlie.groves@...> wrote:
>>>
>>> That should be enabled by default.  As of 2.5.0, Jython adds
>>> __pyclasspath__/ to sys.path, and that serves as a PEP 302 import hook
>>> to tell it to look for .py and $py.class files on the classpath.  As
>>> long as you have a jar containing those .py files in its root on the
>>> classpath, Jython should load them.  If that isn't the case, filing a
>>> bug at bugs.jython.org would be appreciated.
>>
>> Ok for simple standalone .py files but something like Pygments has sub modules
>>
>> - pygments
>> -- filters
>> -- formatters
>> -- lexers
>> -- styles
>>
>> And i can't put all the .py files in the root since each folder
>> contains a __init__.py files (sorry for maybe dumb questions I'm
>> really not a Python expert).
>
> I'm sorry, I wasn't clear about what I said to put at the root.  Put
> that whole pygments directory structure in the jar file.  If you're in
> the directory with pygments in it, "jar cf pygments.jar pygments"
> would do the trick.
>
> The same holds for the compiled version.  compileall.py will put the
> $py.class files in the proper place in that directory structure.  If
> you make the jar after that, the compiled versions will be used in
> preference to the source versions.

Ok thanks a lot, i understand all this better now, at last i will have
something clean ;)

>
> Charlie
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Jython-users mailing list
> Jython-users@...
> https://lists.sourceforge.net/lists/listinfo/jython-users
>

--
Thomas Mortagne

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Jython-users mailing list
Jython-users@...
https://lists.sourceforge.net/lists/listinfo/jython-users

[David Handy-3]

When you install Jython, you have two options, the "regular" installation, which has the Lib directory separately in the filesystem, and the "stand-alone" installation, which includes the Lib directory in jython.jar and is completely self-contained.

What would be the advantage of creating a jython-libs.jar as described below over just using the stand-alone jython.jar that is already available?

David H

> -----Original Message-----
> From: Charlie Groves [mailto:charlie.groves@...]
> Sent: Friday, September 04, 2009 3:47 AM
> To: Marcos Sánchez Provencio
> Cc: jython-users@...
> Subject: Re: [Jython-users] Distribute python lib used in a java
> project
>
> 2009/9/4 Marcos Sánchez Provencio <msanchez@...>:
> > So, if I have understood, it would be enough just to build a
> > jython-libs.jar with the contents of Lib and just add one file to my
> web
> > app?
>
> Yes.
>
> > Why is not this jar included in the standard dist? Or is it? I seems
> > this is the best way to deal with the inclusion of the standard Lib...
>
> It isn't included, but there's no good reason for that.    The import
> from classpath feature was addded relatively late in the development
> of 2.5, so we didn't want to switch everything over at that point.
>
> I agree that it'd be the easiest way to get the Lib files onto the
> classpath, so we should look at it for 2.5.2 since we're adding
> features there.
>
> Charlie
>
> -----------------------------------------------------------------------
> -------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008
> 30-Day
> trial. Simplify your report design, integration and deployment - and
> focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Jython-users mailing list
> Jython-users@...
> https://lists.sourceforge.net/lists/listinfo/jython-users

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Jython-users mailing list
Jython-users@...
https://lists.sourceforge.net/lists/listinfo/jython-users

[Charlie Groves]

2009/9/4 David Handy <David.Handy@...>:
> When you install Jython, you have two options, the "regular" installation, which has the Lib directory separately in the filesystem, and the "stand-alone" installation, which includes the Lib directory in jython.jar and is completely self-contained.
>
> What would be the advantage of creating a jython-libs.jar as described below over just using the stand-alone jython.jar that is already available?

The code that finds it in Lib works by finding jython.jar itself.  It
finds the path to the jar, and adds that path + /Lib to sys.path
directly.

That works find when you're using a normal classpath, and jython can
introspect it to find the path to its own jar.  When a more
complicated classloading scheme is being used like in a servlet
engine, jython can't find the jar and therefore can't find the .py
files.  By using the classpath directly, it doesn't care where the jar
is.

It also doesn't care what jar the files are in, as long as they're on
the classpath.  There's no reason they couldn't be put in jython.jar
when building standalone mode, so this doesn't necessitate a separate
jython-libs.jar or something like that.

Charlie

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Jython-users mailing list
Jython-users@...
https://lists.sourceforge.net/lists/listinfo/jython-users