Grok, python 2.6 and unladen-swallow

[Sebastian Ware]

Having looked at some of the performance data of unladen-swallow

   http://code.google.com/p/unladen-swallow/wiki/Release2009Q1

It would seem to bring huge performance benefits to Grok apps,  
especially when using Zodb-style backends. When is it reasonable to  
expect Grok to support python 2.6?

Mvh Sebastian
_______________________________________________
Grok-dev mailing list
Grok-dev@...
https://mail.zope.org/mailman/listinfo/grok-dev

[Jan-Wijbrand Kolman-3]

Sebastian Ware wrote:
> Having looked at some of the performance data of unladen-swallow
>
>    http://code.google.com/p/unladen-swallow/wiki/Release2009Q1
>
> It would seem to bring huge performance benefits to Grok apps,  
> especially when using Zodb-style backends. When is it reasonable to  
> expect Grok to support python 2.6?
>
> Mvh Sebastian

Soon-ish. As soon as 1.0 is final, we can start to integrating the ZTK
into Grok "trunk". As far as I understand ZTK's aim is to run on
python2.6. At least the buildbot[1] tests for that.

So, feature releases after 1.0 see official 2.6 support, depending on
the amount of work we can put in to this work.

regards,
jw


[1] http://dev.thehealthagency.com/buildbot/

_______________________________________________
Grok-dev mailing list
Grok-dev@...
https://mail.zope.org/mailman/listinfo/grok-dev

[Hanno Schlichting-4]

On Mon, Sep 21, 2009 at 12:19 PM, Sebastian Ware <sebastian@...> wrote:
> Having looked at some of the performance data of unladen-swallow
>
>   http://code.google.com/p/unladen-swallow/wiki/Release2009Q1
>
> It would seem to bring huge performance benefits to Grok apps,
> especially when using Zodb-style backends. When is it reasonable to
> expect Grok to support python 2.6?

I tested repoze.bfg, Zope2 and Plone against unladen swallow (at
various points during their Q2 and Q3 development). There was no
noticeable performance improvement in any of those. Even when running
with "-j always" which means everything is JIT-compiled, you only got
a massively increased memory consumption but no real speed-up.

I'm afraid unladen swallow won't be a magic bullet making things
faster. Not unless they actually implement some real performance
optimization strategies for their LLVM code generation.

I tried profile based optimizations as found in gcc 4.4 as well at
some point and couldn't find any real improvements either. It seems
there are no simple optimizations that apply to large Zope based
applications, but it is sheer complexity and amount of code that is
slowing it down in the end. If you want to go faster by a large
margin, you need to do less and not the same stuff slightly faster.

Hanno
_______________________________________________
Grok-dev mailing list
Grok-dev@...
https://mail.zope.org/mailman/listinfo/grok-dev

[Sebastian Ware]

Thanks for the info, I'll keep my expectations low then. I saw the  
impressive improvements in Pickle and was thinking that it would speed  
up Zodb access, but obviously it doesn't have to give any practical  
improvements. I haven't profiled any of my apps in detail so I don't  
even know were potential bottlenecks might be (Grok is fast enough as  
is). A larger memory footprint will obviously have a negative impact,  
so I hope they will address that during their next iterations.

Mvh Sebastian

On 21 sep 2009, at 13.45, Hanno Schlichting wrote:

> On Mon, Sep 21, 2009 at 12:19 PM, Sebastian Ware <sebastian@...
> > wrote:
>> Having looked at some of the performance data of unladen-swallow
>>
>>   http://code.google.com/p/unladen-swallow/wiki/Release2009Q1
>>
>> It would seem to bring huge performance benefits to Grok apps,
>> especially when using Zodb-style backends. When is it reasonable to
>> expect Grok to support python 2.6?
>
> I tested repoze.bfg, Zope2 and Plone against unladen swallow (at
> various points during their Q2 and Q3 development). There was no
> noticeable performance improvement in any of those. Even when running
> with "-j always" which means everything is JIT-compiled, you only got
> a massively increased memory consumption but no real speed-up.
>
> I'm afraid unladen swallow won't be a magic bullet making things
> faster. Not unless they actually implement some real performance
> optimization strategies for their LLVM code generation.
>
> I tried profile based optimizations as found in gcc 4.4 as well at
> some point and couldn't find any real improvements either. It seems
> there are no simple optimizations that apply to large Zope based
> applications, but it is sheer complexity and amount of code that is
> slowing it down in the end. If you want to go faster by a large
> margin, you need to do less and not the same stuff slightly faster.
>
> Hanno

_______________________________________________
Grok-dev mailing list
Grok-dev@...
https://mail.zope.org/mailman/listinfo/grok-dev

[Martijn Faassen-2]

Sebastian Ware wrote:
> Thanks for the info, I'll keep my expectations low then. I saw the  
> impressive improvements in Pickle and was thinking that it would speed  
> up Zodb access, but obviously it doesn't have to give any practical  
> improvements.

The ZODB uses cPickle as far as I know, so speeding up pickle wouldn't
help here.

Regards,

Martijn

_______________________________________________
Grok-dev mailing list
Grok-dev@...
https://mail.zope.org/mailman/listinfo/grok-dev

[Hanno Schlichting-4]

On Mon, Sep 21, 2009 at 6:43 PM, Martijn Faassen <faassen@...> wrote:
> Sebastian Ware wrote:
>> Thanks for the info, I'll keep my expectations low then. I saw the
>> impressive improvements in Pickle and was thinking that it would speed
>> up Zodb access, but obviously it doesn't have to give any practical
>> improvements.
>
> The ZODB uses cPickle as far as I know, so speeding up pickle wouldn't
> help here.

Actually all of the unladen swallow optimizations are for cPickle, so
they are relevant. Their use-case was memcached and not the ZODB, but
in principle we still benefit from them ;-)

I think what happens in Zope applications quite often, is that most of
the objects are already in the connection cache and so the number of
unpickled objects tend to be small. The only times where I have seen
the ZODB to be noticeably slow is once you introduce large network
latency. The pickle/unpickle overhead doesn't seem to matter much at
all.

Hanno
_______________________________________________
Grok-dev mailing list
Grok-dev@...
https://mail.zope.org/mailman/listinfo/grok-dev

[Martijn Faassen-2]

Hanno Schlichting wrote:

> On Mon, Sep 21, 2009 at 6:43 PM, Martijn Faassen <faassen@...> wrote:
>> Sebastian Ware wrote:
>>> Thanks for the info, I'll keep my expectations low then. I saw the
>>> impressive improvements in Pickle and was thinking that it would speed
>>> up Zodb access, but obviously it doesn't have to give any practical
>>> improvements.
>> The ZODB uses cPickle as far as I know, so speeding up pickle wouldn't
>> help here.
>
> Actually all of the unladen swallow optimizations are for cPickle, so
> they are relevant. Their use-case was memcached and not the ZODB, but
> in principle we still benefit from them ;-)

Ah, cool, so in theory this could help.

> I think what happens in Zope applications quite often, is that most of
> the objects are already in the connection cache and so the number of
> unpickled objects tend to be small. The only times where I have seen
> the ZODB to be noticeably slow is once you introduce large network
> latency. The pickle/unpickle overhead doesn't seem to matter much at
> all.

Understood.

As usual, making apps go faster is hard :)

Anyway, hopefully we'll get unladen swallow support in Grok for free
once we move to Python 2.6 and someone experiments with it. We'll then
see whether it matters for them.

Regards,

Martijn

_______________________________________________
Grok-dev mailing list
Grok-dev@...
https://mail.zope.org/mailman/listinfo/grok-dev

[Hanno Schlichting-4]

On Mon, Sep 21, 2009 at 9:07 PM, Martijn Faassen <faassen@...> wrote:
> Anyway, hopefully we'll get unladen swallow support in Grok for free
> once we move to Python 2.6 and someone experiments with it. We'll then
> see whether it matters for them.

I'd be surprised to see any real issues with unladen swallow and Grok.
The one thing they did make sure is to be virtually 100% compatible
with standard Python 2.6.

I was able to run all of Plone, including its vast stack with just two
small changes. All the ZTK underpinnings including their C extensions
worked without changes.

The two things were:

1. they extend the -O command line switch to have a -O3 option. You
need to specify it to benefit from all their speedups. Already on -O2
or -OO Python removes docstrings. There is some code that computes
docstrings via some "string" + otherclass.__doc__ constructs. Run in
-O2 this fails, as this becomes "string" + None and raises an error.
Also the Zope2 ZPublisher "security check" doesn't work, which
prevents methods without a docstring from being published.

2. RestrictedPython has some notion of "safe_builtins", trying to
proxy all builtins with save versions. Unladen swallow introduces some
new internal builtins, which need to be present to execute any code
via eval or exec later on. As with any normal extension of the
builtins in a major Python version, these new builtins need to be
passed through. They have some rather fancy names starting with "#@",
for example "#@make_function", "#@buildclass" and so on. It's quite
tricky to call these from Python code ;-)

On a general note: I do expect to see much more outcome from the
project. They are just still in the early stages, where they focussed
on getting some quick wins out first and now work on the entire Python
to LLVM project. There's lots of potential, but still quite a bit of
work ahead.

Hanno
_______________________________________________
Grok-dev mailing list
Grok-dev@...
https://mail.zope.org/mailman/listinfo/grok-dev