"pre-compiling" Python scripts

[Jyrki Saarinen-3]

Hello all,

as being a Python novice I have a small question:
When profiling my server side application, I can see that most of the
CPU time is spent in interpreting the embedded Python code with
org.python.util.PythonInterpreter.exec(String).

There's a flavour of exec() that takes a PyObject object as a parameter,
is there
a way of "pre-compile" the script into a PyObject that would be executed
with
exec() call? I found CompilerFacade with a static "compile" method, but
I didn't quite
understand how it should be used?

Jyrki


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Jython-users mailing list
Jython-users@...
https://lists.sourceforge.net/lists/listinfo/jython-users

[Charlie Groves]

On Thu, Sep 3, 2009 at 2:09 AM, Jyrki Saarinen<jyrki.saarinen@...> wrote:

> Hello all,
>
> as being a Python novice I have a small question:
> When profiling my server side application, I can see that most of the
> CPU time is spent in interpreting the embedded Python code with
> org.python.util.PythonInterpreter.exec(String).
>
> There's a flavour of exec() that takes a PyObject object as a parameter,
> is there
> a way of "pre-compile" the script into a PyObject that would be executed
> with
> exec() call? I found CompilerFacade with a static "compile" method, but
> I didn't quite
> understand how it should be used?

One of the new features of Jython 2.5.1 is a jsr-223 implementation in
Jython, and some compile methods were added to PythonInterpreter for
that.  If you grab rc1 from jython.org, you can use
PythonInterpreter.compile(String) to produce a PyCode which you can
then pass to exec.

Charlie

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Jython-users mailing list
Jython-users@...
https://lists.sourceforge.net/lists/listinfo/jython-users

[Jyrki Saarinen-3]

Charlie Groves wrote:

> On Thu, Sep 3, 2009 at 2:09 AM, Jyrki Saarinen<jyrki.saarinen@...> wrote:
>  
>> Hello all,
>>
>> as being a Python novice I have a small question:
>> When profiling my server side application, I can see that most of the
>> CPU time is spent in interpreting the embedded Python code with
>> org.python.util.PythonInterpreter.exec(String).
>>
>> There's a flavour of exec() that takes a PyObject object as a parameter,
>> is there
>> a way of "pre-compile" the script into a PyObject that would be executed
>> with
>> exec() call? I found CompilerFacade with a static "compile" method, but
>> I didn't quite
>> understand how it should be used?
>>    
>
> One of the new features of Jython 2.5.1 is a jsr-223 implementation in
> Jython, and some compile methods were added to PythonInterpreter for
> that.  If you grab rc1 from jython.org, you can use
> PythonInterpreter.compile(String) to produce a PyCode which you can
> then pass to exec.
>  

Ok, thanks. Now I'm facing the following exception when using
PythonInterpreter.compile():

Caused by: LookupError: no codec search functions registered: can't find
encoding 'utf-8'

   at org.python.core.PyException.fillInStackTrace(PyException.java:70)
   at java.lang.Throwable.<init>(Throwable.java:181)
   at java.lang.Exception.<init>(Exception.java:29)
   at java.lang.RuntimeException.<init>(RuntimeException.java:32)
   at org.python.core.PyException.<init>(PyException.java:46)
   at org.python.core.PyException.<init>(PyException.java:43)
   at org.python.core.PyException.<init>(PyException.java:61)
   at org.python.core.codecs.lookup(codecs.java:80)
   at org.python.core.codecs.getEncoder(codecs.java:209)
   at org.python.core.codecs.encode(codecs.java:187)
   at org.python.core.PyString.str_encode(PyString.java:2429)
   at org.python.core.PyString.encode(PyString.java:2420)
<antlr stack traces omitted>
   at org.python.antlr.BaseParser.parseModule(BaseParser.java:107)
   at org.python.core.CompileMode$3.dispatch(CompileMode.java:22)
   at org.python.core.ParserFacade.parse(ParserFacade.java:156)
   at
org.python.core.ParserFacade.parseExpressionOrModule(ParserFacade.java:130)
   at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:221)
   at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:218)
   at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:212)

I can see what causes the problem, but how should it be fixed?

Jyrki

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Jython-users mailing list
Jython-users@...
https://lists.sourceforge.net/lists/listinfo/jython-users

[falbriard]

Dear Jyrki,

 I´ve had a similar issue some weeks ago when I added a PythonInterpreter call under LINUX. Below the code snippet that solved the issue for my case.
Setting the python.home variable might fix the look-up path for finding the Lib/encodings  folder and setting the cachedir solves eventual permission errors under Linux.  

// Setup of the PythonInterpreter properties  
// define Jython home directory to find the registry file, use an unprotected cachedir area (chmod -R 777)
Properties props = new Properties();
props.setProperty("python.home", "/opt/jython");    
props.setProperty("python.cachedir", "/opt/tmp/cachedir");  
       
// Pass arguments using a Java String array  
String[] args = {"0",""};

// setup interpreter
PythonInterpreter.initialize(System.getProperties(),props,args );
final PythonInterpreter interpreter = new PythonInterpreter();


Another experimental solution I´ve found in a Python FAQ is to modify the utf-8 encoding setup file named  /LIB/encodings/utf-8.py, This allows to clear an invalid Unicode  character in the source file, like for example clearing a copyright symbol.  Check the hotfix in the blue section below:  


""" Python 'utf-8' Codec

Written by Marc-Andre Lemburg (mal@...).

(c) Copyright CNRI, All Rights Reserved. NO WARRANTY.

"""
import codecs

### Codec APIs

encode = codecs.utf_8_encode

#def decode(input, errors='strict'):
#    return codecs.utf_8_decode(input, errors, True)

def decode(input, errors='strict'):
    try:
      return codecs.utf_8_decode(input, errors, True)
    except:
      # FIXME: clear problematic text
      return codecs.utf_8_decode("", errors, True)

class IncrementalEncoder(codecs.IncrementalEncoder):
    def encode(self, input, final=False):
        return codecs.utf_8_encode(input, self.errors)[0]

class IncrementalDecoder(codecs.BufferedIncrementalDecoder):
    _buffer_decode = codecs.utf_8_decode

class StreamWriter(codecs.StreamWriter):
    encode = codecs.utf_8_encode

class StreamReader(codecs.StreamReader):
    decode = codecs.utf_8_decode

### encodings module API

def getregentry():
    return codecs.CodecInfo(
        name='utf-8',
        encode=encode,
        decode=decode,
        incrementalencoder=IncrementalEncoder,
        incrementaldecoder=IncrementalDecoder,
        streamreader=StreamReader,
        streamwriter=StreamWriter,
    )

Regards,
Claude


Claude Falbriard
Developer
AMS Hortolândia / SP - Brazil
phone:    [phone]
cell:         [phone]
e-mail:    claudef@...

From:	Jyrki Saarinen <jyrki.saarinen@...>
To:	jython-users@...
Date:	09/10/2009 07:07 AM
Subject:	Re: [Jython-users] "pre-compiling" Python scripts

---

Charlie Groves wrote:
> On Thu, Sep 3, 2009 at 2:09 AM, Jyrki Saarinen<jyrki.saarinen@...> wrote:
>  
>> Hello all,
>>
>> as being a Python novice I have a small question:
>> When profiling my server side application, I can see that most of the
>> CPU time is spent in interpreting the embedded Python code with
>> org.python.util.PythonInterpreter.exec(String).
>>
>> There's a flavour of exec() that takes a PyObject object as a parameter,
>> is there
>> a way of "pre-compile" the script into a PyObject that would be executed
>> with
>> exec() call? I found CompilerFacade with a static "compile" method, but
>> I didn't quite
>> understand how it should be used?
>>    
>
> One of the new features of Jython 2.5.1 is a jsr-223 implementation in
> Jython, and some compile methods were added to PythonInterpreter for
> that.  If you grab rc1 from jython.org, you can use
> PythonInterpreter.compile(String) to produce a PyCode which you can
> then pass to exec.
>  

Ok, thanks. Now I'm facing the following exception when using
PythonInterpreter.compile():

Caused by: LookupError: no codec search functions registered: can't find
encoding 'utf-8'

  at org.python.core.PyException.fillInStackTrace(PyException.java:70)
  at java.lang.Throwable.<init>(Throwable.java:181)
  at java.lang.Exception.<init>(Exception.java:29)
  at java.lang.RuntimeException.<init>(RuntimeException.java:32)
  at org.python.core.PyException.<init>(PyException.java:46)
  at org.python.core.PyException.<init>(PyException.java:43)
  at org.python.core.PyException.<init>(PyException.java:61)
  at org.python.core.codecs.lookup(codecs.java:80)
  at org.python.core.codecs.getEncoder(codecs.java:209)
  at org.python.core.codecs.encode(codecs.java:187)
  at org.python.core.PyString.str_encode(PyString.java:2429)
  at org.python.core.PyString.encode(PyString.java:2420)
<antlr stack traces omitted>
  at org.python.antlr.BaseParser.parseModule(BaseParser.java:107)
  at org.python.core.CompileMode$3.dispatch(CompileMode.java:22)
  at org.python.core.ParserFacade.parse(ParserFacade.java:156)
  at
org.python.core.ParserFacade.parseExpressionOrModule(ParserFacade.java:130)
  at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:221)
  at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:218)
  at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:212)

I can see what causes the problem, but how should it be fixed?

Jyrki

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Jython-users mailing list
Jython-users@...
https://lists.sourceforge.net/lists/listinfo/jython-users




------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Jython-users mailing list
Jython-users@...
https://lists.sourceforge.net/lists/listinfo/jython-users

[Charlie Groves]

On Thu, Sep 10, 2009 at 3:08 AM, Jyrki Saarinen
<jyrki.saarinen@...> wrote:
> Ok, thanks. Now I'm facing the following exception when using
> PythonInterpreter.compile():
>
> Caused by: LookupError: no codec search functions registered: can't find
> encoding 'utf-8'

I believe this should be fixed on trunk now.  Jython wasn't using its
builtin codecs as a fallback if the registry was empty.  The fix will
be in 2.5.1rc2 which should be out in the next few days.

Charlie

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Jython-users mailing list
Jython-users@...
https://lists.sourceforge.net/lists/listinfo/jython-users