Slow templates in py2exe

[Aahz]

Cheetah 2.0.1, Python 2.6, CherryPy 3.1.1

I'm having a weird problem where Cheetah templates on Windows XP (and
probably Vista, but haven't tested yet) are more than a hundred times
slower inside an app built with py2exe.  This slowdown is not observed
with either raw or compiled templates run from outside the build, and Mac
built with py2app also works fine.

Any clues?  Ideas for places to look?

Yes, literally more than a hundred times slower; a page that takes less
than a second to display outside the build can take several minutes
inside the .EXE, with CPU pegged at 100%.  I have verified with logging
statements that it's TMPL.respond() that's the problem.  The problem is
linear on the size of the generated output (i.e. a for loop with more
iterations takes longer).
--
Aahz (aahz@...)           <*>         http://www.pythoncraft.com/

"I won't accept a model of the universe in which free will, omniscient
gods, and atheism are simultaneously true."  --M

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

[R. Tyler Ballance-5]

On Thu, 17 Sep 2009, Aahz wrote:

> Cheetah 2.0.1, Python 2.6, CherryPy 3.1.1

Old Cheetah is Old ;)

(current is v2.2.2)

> I'm having a weird problem where Cheetah templates on Windows XP (and
> probably Vista, but haven't tested yet) are more than a hundred times
> slower inside an app built with py2exe.  This slowdown is not observed
> with either raw or compiled templates run from outside the build, and Mac
> built with py2app also works fine.
>
> Any clues?  Ideas for places to look?

My initial instinct is that py2exe is pulling in the Python version of
the NameMapper, which is indeed slower than the C version. The
NameMapper is what is responsible for mapping $myVar into the actual
entity that it represents.

Do you have the compiled version of NameMapper installed? Do you know if
py2exe is pulling it in?

>
> Yes, literally more than a hundred times slower; a page that takes less
> than a second to display outside the build can take several minutes
> inside the .EXE, with CPU pegged at 100%.  I have verified with logging
> statements that it's TMPL.respond() that's the problem.  The problem is
> linear on the size of the generated output (i.e. a for loop with more
> iterations takes longer).


Cheers

-R. Tyler Ballance
Slide, Inc.


------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

[Aahz]

On Thu, Sep 17, 2009, R. Tyler Ballance wrote:
> On Thu, 17 Sep 2009, Aahz wrote:
>>
>> Cheetah 2.0.1, Python 2.6, CherryPy 3.1.1
>
> Old Cheetah is Old ;)
> (current is v2.2.2)

Yes, I know -- but particularly after your response, I kinda doubt that
version is the issue.  ;-)

>> I'm having a weird problem where Cheetah templates on Windows XP (and
>> probably Vista, but haven't tested yet) are more than a hundred times
>> slower inside an app built with py2exe.  This slowdown is not observed
>> with either raw or compiled templates run from outside the build, and Mac
>> built with py2app also works fine.
>
> My initial instinct is that py2exe is pulling in the Python version of
> the NameMapper, which is indeed slower than the C version. The
> NameMapper is what is responsible for mapping $myVar into the actual
> entity that it represents.
>
> Do you have the compiled version of NameMapper installed? Do you know if
> py2exe is pulling it in?

Curioser and curioser, as Alice said.  AFAICT, there isn't any
_namemapper.pyd anywhere on the system.  (There's a _namemapper.so on my
Mac.)  There doesn't seem to be an official pyd for 2.6, should I use
http://feisley.com/python/cheetah/pyd2.2.1/py26/_namemapper.pyd
?

What are the chances that it will work with Cheetah 2.0.1?  (I'm close
to a production release and would really rather not change Cheetah
versions right now.)  Alternatively, how difficult is it likely to be to
upgrade?

Any clue why Cheetah is so fast outside the .EXE without any
_namemapper.pyd around?
--
Aahz (aahz@...)           <*>         http://www.pythoncraft.com/

"I won't accept a model of the universe in which free will, omniscient
gods, and atheism are simultaneously true."  --M

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

[R. Tyler Ballance-5]

On Thu, 17 Sep 2009, Aahz wrote:

> > My initial instinct is that py2exe is pulling in the Python version of
> > the NameMapper, which is indeed slower than the C version. The
> > NameMapper is what is responsible for mapping $myVar into the actual
> > entity that it represents.
> >
> > Do you have the compiled version of NameMapper installed? Do you know if
> > py2exe is pulling it in?
>
> Curioser and curioser, as Alice said.  AFAICT, there isn't any
> _namemapper.pyd anywhere on the system.  (There's a _namemapper.so on my
> Mac.)  There doesn't seem to be an official pyd for 2.6, should I use
> http://feisley.com/python/cheetah/pyd2.2.1/py26/_namemapper.pyd
> ?

Perhaps if you asked James nicely enough he could post a build of 2.0.1
on Windows, the downside of this of course is that prior to v2.2.2
Cheetah was less-than-friendly in terms of building on Windows (which is
why we actually had _namemapper.pyd files checked in.)


> What are the chances that it will work with Cheetah 2.0.1?  (I'm close
> to a production release and would really rather not change Cheetah
> versions right now.)  Alternatively, how difficult is it likely to be to
> upgrade?

The upgrade isn't too terribly difficult provided you're not already
handling unicode objects in some hackish fashion (we had this issue at
Slide, some "automagic code" encoding/decoding back and forth into utf-8
encoded str objects).

Might be worth a try depending on your timeline.

> Any clue why Cheetah is so fast outside the .EXE without any
> _namemapper.pyd around?

The very nature of the NameMapper's function means it's going to call a
lot of functions and iterate quite a bit through a few lists, IIRC the
C version just cuts down a lot of overhead related to Python dynamic
lookup and dispatch. I'm probably wrong about that though :)


Cheers

-R. Tyler Ballance
Slide, Inc.

p.s. mind me asking what you're shipping with Cheetah on Windows? :D



------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss

[Aahz]

On Fri, Sep 18, 2009, R. Tyler Ballance wrote:

> On Thu, 17 Sep 2009, Aahz wrote:
>>> My initial instinct is that py2exe is pulling in the Python version of
>>> the NameMapper, which is indeed slower than the C version. The
>>> NameMapper is what is responsible for mapping $myVar into the actual
>>> entity that it represents.
>>>
>>> Do you have the compiled version of NameMapper installed? Do you know if
>>> py2exe is pulling it in?
>>
>> Curioser and curioser, as Alice said.  AFAICT, there isn't any
>> _namemapper.pyd anywhere on the system.  (There's a _namemapper.so on my
>> Mac.)  There doesn't seem to be an official pyd for 2.6, should I use
>> http://feisley.com/python/cheetah/pyd2.2.1/py26/_namemapper.pyd
>> ?
>
> Perhaps if you asked James nicely enough he could post a build of 2.0.1
> on Windows, the downside of this of course is that prior to v2.2.2
> Cheetah was less-than-friendly in terms of building on Windows (which is
> why we actually had _namemapper.pyd files checked in.)

<shrug>  The 2.2.1 .pyd seems to work fine with 2.0.1 (I already tested
that before you responded because I was in a hurry); if we run into
problems, we'll probably just upgrade Cheetah.

(Just to be explicit, installing the .pyd did fix the performance
problems.)

>> Any clue why Cheetah is so fast outside the .EXE without any
>> _namemapper.pyd around?
>
> The very nature of the NameMapper's function means it's going to call a
> lot of functions and iterate quite a bit through a few lists, IIRC the
> C version just cuts down a lot of overhead related to Python dynamic
> lookup and dispatch. I'm probably wrong about that though :)

Yeah, I know, I just can't figure out how it could possibly work so fast
using the .py outside the .EXE.  It's not worth chasing down, though.
(I did verify by doing a manual import that there isn't any
_namemapper.pyd around)

> p.s. mind me asking what you're shipping with Cheetah on Windows? :D

http://www.egnyte.com/ has a "local cloud client" that uses Cheetah on
CherryPy to run the UI.
--
Aahz (aahz@...)           <*>         http://www.pythoncraft.com/

"I won't accept a model of the universe in which free will, omniscient
gods, and atheism are simultaneously true."  --M

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
Cheetahtemplate-discuss mailing list
Cheetahtemplate-discuss@...
https://lists.sourceforge.net/lists/listinfo/cheetahtemplate-discuss