Zend Loader performance

[Łukasz Bandzarewicz]

Hi all!

During simple profiling I noticed that Zend_Loader could consume 20-25% of execution time.
It's quite obvious, because ZF has to load lots of files, but..
have you ever wondering about compile ZF code? Something similar to phpDoctrine's compile method (http://www.phpdoctrine.org/documentation/manual/0_11?one-page#improving-performance)?

The compile method makes single file of most used components. Additionally it could remove comments and white spaces to reduce size of compiled file. It could reduce slow disk operations (check if file exists, load file, etc.) and increase performance.

What you think, is this feasible?

[Karol Grecki]

Did you test it with opcode cache enabled?

It's true that lots of files are included and compiled, but that's what you have APC for.

Karol

Łukasz Bandzarewicz wrote:

Hi all!

During simple profiling I noticed that Zend_Loader could consume 20-25% of execution time.
It's quite obvious, because ZF has to load lots of files, but..
have you ever wondering about compile ZF code? Something similar to phpDoctrine's compile method (http://www.phpdoctrine.org/documentation/manual/0_11?one-page#improving-performance)?

The compile method makes single file of most used components. Additionally it could remove comments and white spaces to reduce size of compiled file. It could reduce slow disk operations (check if file exists, load file, etc.) and increase performance.

What you think, is this feasible?

[Philip G]

There's arguments on both sides of the fence on this one. I personally like having organization in my code, which is why I like the multi-file method and we can just use APC to speed up code execution. However, not everybody has that option, which runs into a problem. On top of that, you start relying on your cache, not your code efficiency, for speedy code which, imo, is very, very bad.

Now, as far as Zend_Loader, don't use it. It completely negates the ability to use APC. Zend_Loader allows dynamic loading of classes which just happens to prevent APC's ability to determine which files should be cached. Always use require/include for your classes, that way you can use APC and its features. Zend_Loader is convenient, but it's not worth the overhead it gives. The only situations I use Zend_Loader is during factor class creations. That's it.

I like the compile() method you pointed out, Lukasz, but I'm not quite completely sold on it yet. I can see the benefits, but there are questions to ask on both sides of that, too

Philip

On Sun, May 11, 2008 at 8:36 AM, Karol Grecki <kgrecki@...> wrote:

Did you test it with opcode cache enabled?

It's true that lots of files are included and compiled, but that's what you
have APC for.

Karol

Łukasz Bandzarewicz wrote:
>
> Hi all!
>
> During simple profiling I noticed that Zend_Loader could consume 20-25% of
> execution time.
> It's quite obvious, because ZF has to load lots of files, but..
> have you ever wondering about compile ZF code? Something similar to
> phpDoctrine's compile method
> (http://www.phpdoctrine.org/documentation/manual/0_11?one-page#improving-performance)?
>
> The compile method makes single file of most used components. Additionally
> it could remove comments and white spaces to reduce size of compiled file.
> It could reduce slow disk operations (check if file exists, load file,
> etc.) and increase performance.
>
> What you think, is this feasible?
>
>

--

View this message in context: http://www.nabble.com/Zend-Loader-performance-tp17170525p17170545.html

Sent from the Zend Framework mailing list archive at Nabble.com.

--
Philip
gp@...
http://www.gpcentre.net/

[weierophinney]

-- Philip G <gp@...> wrote
(on Sunday, 11 May 2008, 09:36 AM -0500):

> There's arguments on both sides of the fence on this one. I personally like
> having organization in my code, which is why I like the multi-file method and
> we can just use APC to speed up code execution. However, not everybody has that
> option, which runs into a problem. On top of that, you start relying on your
> cache, not your code efficiency, for speedy code which, imo, is very, very bad.
>
> Now, as far as Zend_Loader, don't use it. It completely negates the ability to
> use APC. Zend_Loader allows dynamic loading of classes which just happens to
> prevent APC's ability to determine which files should be cached. Always use
> require/include for your classes, that way you can use APC and its features.
> Zend_Loader is convenient, but it's not worth the overhead it gives. The only
> situations I use Zend_Loader is during factor class creations. That's it.

Philip, this is simply not true in PHP >= 5.2.0. PHP 5.2.0 added a
realpath cache, and Zend_Loader performs very well under both APC and
Zend Platform on those versions -- typically within a few percentage
points of a straight require_once. We will be releasing figures in the
coming months showing real-world benchmarks of this.

The typical source regarding APC and optional requires is a post by
Rasmus that is *SEVERAL* years old and predates the realpath cache. If
you have any up-to-date information that proves me wrong, please post
it, including methodology.

Finally, despite any performance considerations of Zend_Loader and/or
require_once, the fact of the matter is that loading the files for your
application is going to be a small fraction of any site that utilizes
databases or web services. The only reason I can see for optimizing this
by using a compile-to-a-single-file type of solution is when you're
getting so many hits that doing so is going to save you money in terms
of no longer needing additional _servers_ to handle load.

> I like the compile() method you pointed out, Lukasz, but I'm not quite
> completely sold on it yet. I can see the benefits, but there are
> questions to ask on both sides of that, too
>
> Philip
>
>
> On Sun, May 11, 2008 at 8:36 AM, Karol Grecki <kgrecki@...> wrote:
>
>
>     Did you test it with opcode cache enabled?
>
>     It's true that lots of files are included and compiled, but that's what you
>     have APC for.
>
>     Karol
>
>
>      ukasz Bandzarewicz wrote:
>     >
>     > Hi all!
>     >
>     > During simple profiling I noticed that Zend_Loader could consume 20-25%
>     of
>     > execution time.
>     > It's quite obvious, because ZF has to load lots of files, but..
>     > have you ever wondering about compile ZF code? Something similar to
>     > phpDoctrine's compile method
>     > (http://www.phpdoctrine.org/documentation/manual/0_11?one-page#
>     improving-performance)?
>     >
>     > The compile method makes single file of most used components.
>     Additionally
>     > it could remove comments and white spaces to reduce size of compiled
>     file.
>     > It could reduce slow disk operations (check if file exists, load file,
>     > etc.) and increase performance.
>     >
>     > What you think, is this feasible?
>     >
>     >
>
>     --
>     View this message in context: http://www.nabble.com/
>     Zend-Loader-performance-tp17170525p17170545.html
>     Sent from the Zend Framework mailing list archive at Nabble.com.
>
>
>
>
>
> --
> Philip
> gp@...
> http://www.gpcentre.net/

--
Matthew Weier O'Phinney
Software Architect       | matthew@...
Zend - The PHP Company   | http://www.zend.com/

[Philip G]

On Sun, May 11, 2008 at 11:37 AM, Matthew Weier O'Phinney <matthew@...> wrote:

-- Philip G <gp@...> wrote
(on Sunday, 11 May 2008, 09:36 AM -0500):

> There's arguments on both sides of the fence on this one. I personally like
> having organization in my code, which is why I like the multi-file method and
> we can just use APC to speed up code execution. However, not everybody has that
> option, which runs into a problem. On top of that, you start relying on your
> cache, not your code efficiency, for speedy code which, imo, is very, very bad.
>
> Now, as far as Zend_Loader, don't use it. It completely negates the ability to
> use APC. Zend_Loader allows dynamic loading of classes which just happens to
> prevent APC's ability to determine which files should be cached. Always use
> require/include for your classes, that way you can use APC and its features.
> Zend_Loader is convenient, but it's not worth the overhead it gives. The only
> situations I use Zend_Loader is during factor class creations. That's it.

Philip, this is simply not true in PHP >= 5.2.0. PHP 5.2.0 added a
realpath cache, and Zend_Loader performs very well under both APC and
Zend Platform on those versions -- typically within a few percentage
points of a straight require_once. We will be releasing figures in the
coming months showing real-world benchmarks of this.

I'd be very interested in seeing those results.

The typical source regarding APC and optional requires is a post by
Rasmus that is *SEVERAL* years old and predates the realpath cache. If
you have any up-to-date information that proves me wrong, please post
it, including methodology.

Try several months:
http://blog.digitalstruct.com/2007/12/23/zend-framework-performance-zend_loader/
http://blog.digitalstruct.com/2007/12/24/accelerators-revisited/




I'd be quite curious to see if things have changed.
 

--
Matthew Weier O'Phinney
Software Architect       | matthew@...
Zend - The PHP Company   | http://www.zend.com/

--
Philip
gp@...
http://www.gpcentre.net/

[Karol Grecki]

Philip G wrote:

There's arguments on both sides of the fence on this one. I personally like
having organization in my code, which is why I like the multi-file method
and we can just use APC to speed up code execution. However, not everybody
has that option, which runs into a problem. On top of that, you start
relying on your cache, not your code efficiency, for speedy code which, imo,
is very, very bad.

Now, as far as Zend_Loader, don't use it. It completely negates the ability
to use APC. Zend_Loader allows dynamic loading of classes which just happens
to prevent APC's ability to determine which files should be cached. Always
use require/include for your classes, that way you can use APC and its
features. Zend_Loader is convenient, but it's not worth the overhead it
gives. The only situations I use Zend_Loader is during factor class
creations. That's it.

This is absolutely not true as Matthew already mentioned. Enabling APC removes almost all overhead for loading those files. I was profiling my code (for other reasons), I've seen dumps in Kcachegrind, you can do the same.
I guess it wouldn't hurt offering some more options for people that cannot enable it for some reasons (shared hosting etc) but as general rule, APC is there to solve exactly this problem. Also opcode caching is in no way related to writing good, efficient code.

Karol

[weierophinney]

-- Philip G <gp@...> wrote
(on Sunday, 11 May 2008, 12:25 PM -0500):

> On Sun, May 11, 2008 at 11:37 AM, Matthew Weier O'Phinney <matthew@...>
> wrote:
>
>     -- Philip G <gp@...> wrote
>     (on Sunday, 11 May 2008, 09:36 AM -0500):
>     > There's arguments on both sides of the fence on this one. I personally
>     like
>     > having organization in my code, which is why I like the multi-file method
>     and
>     > we can just use APC to speed up code execution. However, not everybody
>     has that
>     > option, which runs into a problem. On top of that, you start relying on
>     your
>     > cache, not your code efficiency, for speedy code which, imo, is very,
>     very bad.
>     >
>     > Now, as far as Zend_Loader, don't use it. It completely negates the
>     ability to
>     > use APC. Zend_Loader allows dynamic loading of classes which just happens
>     to
>     > prevent APC's ability to determine which files should be cached. Always
>     use
>     > require/include for your classes, that way you can use APC and its
>     features.
>     > Zend_Loader is convenient, but it's not worth the overhead it gives. The
>     only
>     > situations I use Zend_Loader is during factor class creations. That's it.
>
>     Philip, this is simply not true in PHP >= 5.2.0. PHP 5.2.0 added a
>     realpath cache, and Zend_Loader performs very well under both APC and
>     Zend Platform on those versions -- typically within a few percentage
>     points of a straight require_once. We will be releasing figures in the
>     coming months showing real-world benchmarks of this.
>
> I'd be very interested in seeing those results.

Ralph and Stas are working on getting something no later than 1.6.0; I
believe they are also going to publish the test suite so that others can
run the tests independently.

>     The typical source regarding APC and optional requires is a post by
>     Rasmus that is *SEVERAL* years old and predates the realpath cache. If
>     you have any up-to-date information that proves me wrong, please post
>     it, including methodology.
>
> Try several months:
> http://blog.digitalstruct.com/2007/12/23/zend-framework-performance-zend_loader
> /
> http://blog.digitalstruct.com/2007/12/24/accelerators-revisited/
>
> I'd be quite curious to see if things have changed.

Those posts were the catalyst for some changes to Zend_Loader that were
released with 1.0.3 and later versions. Before these versions, we were
checking to see if the file being loaded was readable, which introduces
an additional stat call, and thus slows things down); we now go
directly to include_once (which has its own issues, but performance is
not one of them). So, yes, there *have* been changes to Zend_Loader, and
I will contact the owner of that blog to have him re-run the tests with
a more recent version of ZF.

Regardless, you ignored the next paragraph I wrote, which stated this:

    Finally, despite any performance considerations of Zend_Loader and/or
    require_once, the fact of the matter is that loading the files for your
    application is going to be a small fraction of any site that utilizes
    databases or web services.

Let's be performance conscious, but not go overboard on it. Use an
opcode cache, a recent PHP version, and worry about the real performance
bottlenecks -- data access and retrieval.

--
Matthew Weier O'Phinney
Software Architect       | matthew@...
Zend - The PHP Company   | http://www.zend.com/

[Matthew Ratzloff]

Although I was (like you) under the impression that opcode caches couldn't
cache autoloaded classes, I'm more inclined to trust Matthew and Ralph
than a blog post from last December that doesn't have the test suite
available for download.  Five months is a long time in Zend Framework
time; since then, there have been three releases: 1.0.4, 1.5.0, and 1.5.1.

This belief is common among PHP developers, though, so if it isn't the
case I'd like to know about it.  Needless to say, I'm looking forward to
Zend's paper on the subject.

-Matt

On Sun, May 11, 2008 10:25 am, Philip G wrote:

> Try several months:
> http://blog.digitalstruct.com/2007/12/23/zend-framework-performance-zend_loader/
> http://blog.digitalstruct.com/2007/12/24/accelerators-revisited/
>
> I'd be quite curious to see if things have changed.

[Philip G]

I'm glad to hear things have improved. Now, don't get me wrong, I didn't say I was sold on the subject, but I did show some evidence showing the opposite being true. I would love to see an official Zend paper on this.

Autoloading vs direct includes is a large debate, even at work, which I would love to see some hard supporting facts on the subject. As for creating my own test suites? I'm just too busy to be able to give it the proper attention that is required. So, I rely on the breakdown from others. Mike's blog was the last thing I herd on the subject (and was backed up by other testimonials I've read), so I tended to gravitate towards his findings. In a way, Matthew even validated what Mike had discovered.

However, Matthew did said things have improved since then -- great! I'd love to see some evidence.
(Logically, I can completely understand, but logic is just that: logic not fact)

Until the Zend paper is released, all we have to fall back on is Mike's findings, which show the opposite.

Cheers!

Philip

On Sun, May 11, 2008 at 10:28 PM, Matthew Ratzloff <matt@...> wrote:

Although I was (like you) under the impression that opcode caches couldn't
cache autoloaded classes, I'm more inclined to trust Matthew and Ralph
than a blog post from last December that doesn't have the test suite
available for download.  Five months is a long time in Zend Framework
time; since then, there have been three releases: 1.0.4, 1.5.0, and 1.5.1.

This belief is common among PHP developers, though, so if it isn't the
case I'd like to know about it.  Needless to say, I'm looking forward to
Zend's paper on the subject.

-Matt

On Sun, May 11, 2008 10:25 am, Philip G wrote:

> Try several months:
> http://blog.digitalstruct.com/2007/12/23/zend-framework-performance-zend_loader/
> http://blog.digitalstruct.com/2007/12/24/accelerators-revisited/
>
> I'd be quite curious to see if things have changed.

--
Philip
gp@...
http://www.gpcentre.net/

[weierophinney]

-- Philip G <gp@...> wrote
(on Sunday, 11 May 2008, 10:59 PM -0500):

> I'm glad to hear things have improved. Now, don't get me wrong, I
> didn't say I was sold on the subject, but I did show some evidence
> showing the opposite being true. I would love to see an official Zend
> paper on this.
>
> Autoloading vs direct includes is a large debate, even at work, which
> I would love to see some hard supporting facts on the subject. As for
> creating my own test suites? I'm just too busy to be able to give it
> the proper attention that is required. So, I rely on the breakdown
> from others. Mike's blog was the last thing I herd on the subject (and
> was backed up by other testimonials I've read), so I tended to
> gravitate towards his findings. In a way, Matthew even validated what
> Mike had discovered.

I did, but we had to develop our own code to benchmark, as this
developer did not publish his methodology. We developed our code because
we wanted to either verify or invalidate his results. As it was, the
situation was not terribly grave -- but we may have been testing
differently than he had.

One thing you should always keep in mind when you see publications such
as these: do they provide a way for others to independently verify the
results? If they don't, you should be skeptical and take them with a
grain of salt. Why? Because it could be a matter of a poorly configured
environment, OS issues (for instance, Mac is notorious for I/O
throughput issues), or even poor test suite setup. But you can never be
sure, because there is no way to check.

I've posted a comment on the entry asking for this information and also
asking him to re-run the tests with a newer ZF release (the author
provides no means for contact, and, in fact, comments appear to be
moderated); hopefully we can get a better idea of the current state of
affairs from this.

> However, Matthew did said things have improved since then -- great!
> I'd love to see some evidence.  (Logically, I can completely
> understand, but logic is just that: logic not fact)

Right -- apply that same rigorousness when reviewing benchmarks as well.
:-)

> Until the Zend paper is released, all we have to fall back on is Mike's
> findings, which show the opposite.

And, as I noted, even these findings are a bit suspect, as nobody can
independently verify or invalidate them.

> On Sun, May 11, 2008 at 10:28 PM, Matthew Ratzloff <matt@...>
> wrote:
>
>     Although I was (like you) under the impression that opcode caches couldn't
>     cache autoloaded classes, I'm more inclined to trust Matthew and Ralph
>     than a blog post from last December that doesn't have the test suite
>     available for download.  Five months is a long time in Zend Framework
>     time; since then, there have been three releases: 1.0.4, 1.5.0, and 1.5.1.
>
>     This belief is common among PHP developers, though, so if it isn't the
>     case I'd like to know about it.  Needless to say, I'm looking forward to
>     Zend's paper on the subject.
>
>     -Matt
>
>     On Sun, May 11, 2008 10:25 am, Philip G wrote:
>
>     > Try several months:
>     > http://blog.digitalstruct.com/2007/12/23/
>     zend-framework-performance-zend_loader/
>     > http://blog.digitalstruct.com/2007/12/24/accelerators-revisited/
>     >
>     > I'd be quite curious to see if things have changed.
>
>
>
>
>
> --
> Philip
> gp@...
> http://www.gpcentre.net/

--
Matthew Weier O'Phinney
Software Architect       | matthew@...
Zend - The PHP Company   | http://www.zend.com/

[Stanislav Malyshev]

Hi!

I've run a benchmark loading 725 framework class files on 5.2 and 5.3
with and without bytecode caching.
The benchmark uses list of 725 Framework classes and loads them one by
one, via require_once and via Zend_Loader::loadClass. You can see the
files here: http://random-bits-of.info/fw-tests/

OK, so here's the results:

Without bytecode cache:
          require_once Zend_Loader
php5.2        4.42      4.42
php5.3        4.96      4.97

With bytecode cache:
          require_once Zend_Loader
php5.2        63.04     56.62
php5.3        61.28     55.52

Numbers are requests per second (more is better). Bytecode cache used in
the benchmark is Zend's one, not APC.

Conclusions from this would be:
0. It is *very* important to understand that it is a narrow-point
benchmark that tests only one function in one specific way. Please do
not draw conclusions on behavior of whole applications based only on
this benchmark.

1. You *do* want to use bytecode caching. You won't get 15x performance
on any real application, but it does speed up loading very significantly.

2. Without bytecode caching, it doesn't matter if you use require_once
or Loader - both are equally slow :)

3. With bytecode caching, Loader has some overhead - explanation for
this is that with file accesses eliminated, require_once of course has
little left, while Loader still does a couple of function
calls. But on real-life apps it'd probably be very small, provided that
it's about 10% even on loading-only huge-class-list benchmark, and your
application probably does something useful instead of loading 700+
framework classes :)

4. 5.3 is still a moving target, to don't put too much stake in current
benchmark results for 5.3, they probably will be different by the time
5.3 is in release cycle (hopefully, better :)

If you have more questions on this, please ask.
--
Stanislav Malyshev, Zend Software Architect
stas@...   http://www.zend.com/
[phone]   MSN: stas@...

[Shekar C Reddy]

Stains,

Great benchmarking and awesome explanations. You are an EXPERT in this area!

Regards,

Shekar

On Thu, May 15, 2008 at 5:56 PM, Stanislav Malyshev <stas@...> wrote:
> Hi!
>
> I've run a benchmark loading 725 framework class files on 5.2 and 5.3 with
> and without bytecode caching.
> The benchmark uses list of 725 Framework classes and loads them one by
> one, via require_once and via Zend_Loader::loadClass. You can see the
> files here: http://random-bits-of.info/fw-tests/
>
> OK, so here's the results:
>
> Without bytecode cache:
>         require_once Zend_Loader
> php5.2        4.42      4.42
> php5.3        4.96      4.97
>
> With bytecode cache:
>         require_once Zend_Loader
> php5.2        63.04     56.62
> php5.3        61.28     55.52
>
> Numbers are requests per second (more is better). Bytecode cache used in
> the benchmark is Zend's one, not APC.
>
> Conclusions from this would be:
> 0. It is *very* important to understand that it is a narrow-point
> benchmark that tests only one function in one specific way. Please do
> not draw conclusions on behavior of whole applications based only on
> this benchmark.
>
> 1. You *do* want to use bytecode caching. You won't get 15x performance
> on any real application, but it does speed up loading very significantly.
>
> 2. Without bytecode caching, it doesn't matter if you use require_once
> or Loader - both are equally slow :)
>
> 3. With bytecode caching, Loader has some overhead - explanation for
> this is that with file accesses eliminated, require_once of course has
> little left, while Loader still does a couple of function
> calls. But on real-life apps it'd probably be very small, provided that
> it's about 10% even on loading-only huge-class-list benchmark, and your
> application probably does something useful instead of loading 700+
> framework classes :)
>
> 4. 5.3 is still a moving target, to don't put too much stake in current
> benchmark results for 5.3, they probably will be different by the time
> 5.3 is in release cycle (hopefully, better :)
>
> If you have more questions on this, please ask.
> --
> Stanislav Malyshev, Zend Software Architect
> stas@...   http://www.zend.com/
> [phone]   MSN: stas@...

[Keith Pope-3]

 
Thx for the benchmarks :) I think there where earlier discussions on this and the ZF team are working on their own set of benchmarks, it will be interesting to see how all this compares.

Looks to me like the performance difference is pretty negligible for most developers to worry about :)

I was thinking about this after the earlier discussions and was wondering what people thought about phar? Would this improve performance if you really needed it?

Keith Pope
Web Developer

-----Original Message-----
From: Stanislav Malyshev [mailto:stas@...]
Sent: 15 May 2008 23:56
To: fw-general@...
Subject: Re: [fw-general] Zend Loader performance - benchmark

Hi!

I've run a benchmark loading 725 framework class files on 5.2 and 5.3 with and without bytecode caching.
The benchmark uses list of 725 Framework classes and loads them one by one, via require_once and via Zend_Loader::loadClass. You can see the files here: http://random-bits-of.info/fw-tests/

OK, so here's the results:

Without bytecode cache:
          require_once Zend_Loader
php5.2        4.42      4.42
php5.3        4.96      4.97

With bytecode cache:
          require_once Zend_Loader
php5.2        63.04     56.62
php5.3        61.28     55.52

Numbers are requests per second (more is better). Bytecode cache used in the benchmark is Zend's one, not APC.

Conclusions from this would be:
0. It is *very* important to understand that it is a narrow-point benchmark that tests only one function in one specific way. Please do not draw conclusions on behavior of whole applications based only on this benchmark.

1. You *do* want to use bytecode caching. You won't get 15x performance on any real application, but it does speed up loading very significantly.

2. Without bytecode caching, it doesn't matter if you use require_once or Loader - both are equally slow :)

3. With bytecode caching, Loader has some overhead - explanation for this is that with file accesses eliminated, require_once of course has little left, while Loader still does a couple of function calls. But on real-life apps it'd probably be very small, provided that it's about 10% even on loading-only huge-class-list benchmark, and your application probably does something useful instead of loading 700+ framework classes :)

4. 5.3 is still a moving target, to don't put too much stake in current benchmark results for 5.3, they probably will be different by the time
5.3 is in release cycle (hopefully, better :)

If you have more questions on this, please ask.
--
Stanislav Malyshev, Zend Software Architect
stas@...   http://www.zend.com/
[phone]   MSN: stas@...





--

allpay.net Limited, Fortis et Fides, Whitestone Business Park, Whitestone, Hereford, HR1 3SE.

Registered in England No. 02933191. UK VAT Reg. No. 666 9148 88.



Telephone: [phone], Fax: [phone].

Website: www.allpay.net

Email: enquiries@...



This email, and any files transmitted with it, is confidential and intended solely for the use of the individual or entity to whom it is addressed. If you have received this email in error please notify the allpay.net Information Security Manager at the number above.

[Pádraic Brady]

I think the results were inevitable. It seems there is still a lot of developers out there quoting from blog posts and irc discussions that are months, and in some cases, years old. You'd think PHP 5.2 was not being used anywhere ;).

We really do need some more benchmarks out there proving this point. We live in a world where a framework can even be judged by it's download size afterall!

 

Pádraic Brady

http://blog.astrumfutura.com
http://www.patternsforphp.com
OpenID Europe Foundation

----- Original Message ----
From: Keith Pope <Keith.Pope@...>
To: fw-general <fw-general@...>
Sent: Friday, May 16, 2008 8:23:19 AM
Subject: RE: [fw-general] Zend Loader performance - benchmark


Thx for the benchmarks :) I think there where earlier discussions on this and the ZF team are working on their own set of benchmarks, it will be interesting to see how all this compares.

Looks to me like the performance difference is pretty negligible for most developers to worry about :)

I was thinking about this after the earlier discussions and was wondering what people thought about phar? Would this improve performance if you really needed it?

Keith Pope
Web Developer

-----Original Message-----
From: Stanislav Malyshev [mailto:stas@...]
Sent: 15 May 2008 23:56
To: fw-general@...
Subject: Re: [fw-general] Zend Loader performance - benchmark

Hi!

I've run a benchmark loading 725 framework class files on 5.2 and 5.3 with and without bytecode caching.
The benchmark uses list of 725 Framework classes and loads them one by one, via require_once and via Zend_Loader::loadClass. You can see the files here: http://random-bits-of.info/fw-tests/

OK, so here's the results:

Without bytecode cache:
          require_once Zend_Loader
php5.2        4.42      4.42
php5.3        4.96      4.97

With bytecode cache:
          require_once Zend_Loader
php5.2        63.04    56.62
php5.3        61.28    55.52

Numbers are requests per second (more is better). Bytecode cache used in the benchmark is Zend's one, not APC.

Conclusions from this would be:
0. It is *very* important to understand that it is a narrow-point benchmark that tests only one function in one specific way. Please do not draw conclusions on behavior of whole applications based only on this benchmark.

1. You *do* want to use bytecode caching. You won't get 15x performance on any real application, but it does speed up loading very significantly.

2. Without bytecode caching, it doesn't matter if you use require_once or Loader - both are equally slow :)

3. With bytecode caching, Loader has some overhead - explanation for this is that with file accesses eliminated, require_once of course has little left, while Loader still does a couple of function calls. But on real-life apps it'd probably be very small, provided that it's about 10% even on loading-only huge-class-list benchmark, and your application probably does something useful instead of loading 700+ framework classes :)

4. 5.3 is still a moving target, to don't put too much stake in current benchmark results for 5.3, they probably will be different by the time
5.3 is in release cycle (hopefully, better :)

If you have more questions on this, please ask.
--
Stanislav Malyshev, Zend Software Architect
stas@...  http://www.zend.com/
[phone]  MSN: stas@...





--
allpay.net Limited, Fortis et Fides, Whitestone Business Park, Whitestone, Hereford, HR1 3SE.
Registered in England No. 02933191. UK VAT Reg. No. 666 9148 88.

Telephone: [phone], Fax: [phone].
Website: www.allpay.net
Email: enquiries@...

This email, and any files transmitted with it, is confidential and intended solely for the use of the individual or entity to whom it is addressed. If you have received this email in error please notify the allpay.net Information Security Manager at the number above.

[weierophinney]

-- Keith Pope <Keith.Pope@...> wrote
(on Friday, 16 May 2008, 08:23 AM +0100):
>  
> Thx for the benchmarks :) I think there where earlier discussions on
> this and the ZF team are working on their own set of benchmarks, it
> will be interesting to see how all this compares.

Just for reference, Keith, Stas is a Zend employee, and is helping
develop those benchmarks. Consider this a preview. :-)

> Looks to me like the performance difference is pretty negligible for
> most developers to worry about :)

Which is precisely the point I've been trying to get across. You're more
likely going to notice performance degradation from database and web
service access than from Zend_Loader.

> I was thinking about this after the earlier discussions and was
> wondering what people thought about phar? Would this improve
> performance if you really needed it?

It may, it may not. I think when using the C extension (ext/phar), there
is a good possibility it would increase performance, but in large part
because _all_ files would be loaded on each request -- and the C
extension interoperates with the opcode cache well. But packaging and
deployment of phars is a whole other barrel of monkeys.

> -----Original Message-----
> From: Stanislav Malyshev [mailto:stas@...]
> Sent: 15 May 2008 23:56
> To: fw-general@...
> Subject: Re: [fw-general] Zend Loader performance - benchmark
>
> Hi!
>
> I've run a benchmark loading 725 framework class files on 5.2 and 5.3 with and without bytecode caching.
> The benchmark uses list of 725 Framework classes and loads them one by one, via require_once and via Zend_Loader::loadClass. You can see the files here: http://random-bits-of.info/fw-tests/
>
> OK, so here's the results:
>
> Without bytecode cache:
>           require_once Zend_Loader
> php5.2        4.42      4.42
> php5.3        4.96      4.97
>
> With bytecode cache:
>           require_once Zend_Loader
> php5.2        63.04     56.62
> php5.3        61.28     55.52
>
> Numbers are requests per second (more is better). Bytecode cache used in the benchmark is Zend's one, not APC.
>
> Conclusions from this would be:
> 0. It is *very* important to understand that it is a narrow-point benchmark that tests only one function in one specific way. Please do not draw conclusions on behavior of whole applications based only on this benchmark.
>
> 1. You *do* want to use bytecode caching. You won't get 15x performance on any real application, but it does speed up loading very significantly.
>
> 2. Without bytecode caching, it doesn't matter if you use require_once or Loader - both are equally slow :)
>
> 3. With bytecode caching, Loader has some overhead - explanation for this is that with file accesses eliminated, require_once of course has little left, while Loader still does a couple of function calls. But on real-life apps it'd probably be very small, provided that it's about 10% even on loading-only huge-class-list benchmark, and your application probably does something useful instead of loading 700+ framework classes :)
>
> 4. 5.3 is still a moving target, to don't put too much stake in current benchmark results for 5.3, they probably will be different by the time
> 5.3 is in release cycle (hopefully, better :)
>
> If you have more questions on this, please ask.
> --
> Stanislav Malyshev, Zend Software Architect
> stas@...   http://www.zend.com/
> [phone]   MSN: stas@...
>
>
>
>
>
> --
> allpay.net Limited, Fortis et Fides, Whitestone Business Park, Whitestone, Hereford, HR1 3SE.
> Registered in England No. 02933191. UK VAT Reg. No. 666 9148 88.
>
> Telephone: [phone], Fax: [phone].
> Website: www.allpay.net
> Email: enquiries@...
>
> This email, and any files transmitted with it, is confidential and intended solely for the use of the individual or entity to whom it is addressed. If you have received this email in error please notify the allpay.net Information Security Manager at the number above.
>

--
Matthew Weier O'Phinney
Software Architect       | matthew@...
Zend - The PHP Company   | http://www.zend.com/

[Stanislav Malyshev]

Hi!

> I was thinking about this after the earlier discussions and was
> wondering what people thought about phar? Would this improve
> performance if you really needed it?

phar is a bit more complicated story... We'll be doing some work on
performance-testing phar too, but that'll take some time. From what I
understand, phar just stabilized and acquired the features it needed to
be able to run with frameworks like ZF, and I'm not sure yet how
bytecode caching solutions work with it. But we definitely intend to
address this topic, now that phar looks like it's ready for the prime time.
--
Stanislav Malyshev, Zend Software Architect
stas@...   http://www.zend.com/
[phone]   MSN: stas@...

[Steph Fox]

Hi,

>> I was thinking about this after the earlier discussions and was
>> wondering what people thought about phar? Would this improve
>> performance if you really needed it?
>
> phar is a bit more complicated story... We'll be doing some work on
> performance-testing phar too, but that'll take some time. From what I
> understand, phar just stabilized and acquired the features it needed to be
> able to run with frameworks like ZF, and I'm not sure yet how bytecode
> caching solutions work with it. But we definitely intend to address this
> topic, now that phar looks like it's ready for the prime time.

Phar works happily with APC, according to Greg. Beyond that - you tell us
(please :)

Performance optimization work has been saved to the last. The most recent
benchmarks gave a performance hit of ~40% - this running phpMyAdmin from an
uncompressed phar and using APC - but Greg has plenty of ideas about how to
turn that around for web apps. Expect to see them in place before Phar 2.0
is declared stable; we only just made it past the first beta release.

- Steph

> --
> Stanislav Malyshev, Zend Software Architect
> stas@...   http://www.zend.com/
> [phone]   MSN: stas@...