The performance test of 6 leading frameworks

[dab-3]

hi
 I do not know if this test has already been posted on the list
 Why these bad performances ?
 http://www.alrond.com/en/2007/jan/25/performance-test-of-6-leading-frameworks/

thank

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Mike Whitaker]

On 29 Feb 2008, at 16:21, dab@... wrote:

> hi
>  I do not know if this test has already been posted on the list

Several times.

>  Why these bad performances ?
>  http://www.alrond.com/en/2007/jan/25/performance-test-of-6-leading- 
> frameworks/

Yes, it's slower. But let's just look at one figure there in a real  
world context:

Catalyst: req/s 132.96

That's:

roughly 480,000 req/hour
roughly 11 milion req/day
roughly 330 million req/month

If your site's doing that many requests a month, and you can't afford  
to move off one machine, your site's revenue model is seriously  
screwed. The last site I worked on that did a quarter billion plus a  
month did it on 40 servers!

On top of that, of course, that presupposes all your users are  
accepting data as fast as you can throw it at them: rarely, if ever,  
is that the case. So, the question is, if the road you're on is  
limited to 70mph, does it matter if you're driving a car that'll do  
110 or 230? Choose one that you're comfortable driving :)
--
Mike Whitaker - mike@...



_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[J. Shirley]

On Fri, Feb 29, 2008 at 8:41 AM, Mike Whitaker <mike@...> wrote:

On 29 Feb 2008, at 16:21, dab@... wrote:

> hi
>  I do not know if this test has already been posted on the list

Several times.

>  Why these bad performances ?
>  http://www.alrond.com/en/2007/jan/25/performance-test-of-6-leading-
> frameworks/

Yes, it's slower. But let's just look at one figure there in a real
world context:

Catalyst: req/s 132.96

That's:

roughly 480,000 req/hour
roughly 11 milion req/day
roughly 330 million req/month

If your site's doing that many requests a month, and you can't afford
to move off one machine, your site's revenue model is seriously
screwed. The last site I worked on that did a quarter billion plus a
month did it on 40 servers!

On top of that, of course, that presupposes all your users are
accepting data as fast as you can throw it at them: rarely, if ever,
is that the case. So, the question is, if the road you're on is
limited to 70mph, does it matter if you're driving a car that'll do
110 or 230? Choose one that you're comfortable driving :)
--
Mike Whitaker - mike@...

I think that the focus should be on template engines, not the framework.

Code Irony has it right: http://www.codeirony.com/?p=9

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Andy Grundman]

On Feb 29, 2008, at 11:21 AM, dab@... wrote:

> hi
> I do not know if this test has already been posted on the list
> Why these bad performances ?
> http://www.alrond.com/en/2007/jan/25/performance-test-of-6-leading-frameworks/


All in all seems like a well-done benchmark with lots of good data.

Note he is benchmarking TT performance as well.  Although I guess he  
is benchmarking everyone else's template libraries too.  To benchmark  
the framework itself, you should just send some static data.  I think  
with 5 FastCGI procs he should get more requests per second though.  
I'm not familiar with nginx to know if that has something to do with  
the numbers.  Will be worth benchmarking it with lighttpd as well.

Another interesting thing is in the hello.tt file he is not using a  
variable for the 'hello' string like he did in most other frameworks.  
Wouldn't help performance, though.  Also don't know if he's using  
Template::Stash::XS or template caching, since these are not enabled  
by default in View::TT.

I'll see about posting some benchmarks of our various engines with  
this same test data, including a fast new engine I'm working on. :)


_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Andy Grundman]

On Feb 29, 2008, at 12:14 PM, Andy Grundman wrote:

>
> On Feb 29, 2008, at 11:21 AM, dab@... wrote:
>
>> hi
>> I do not know if this test has already been posted on the list
>> Why these bad performances ?
>> http://www.alrond.com/en/2007/jan/25/performance-test-of-6-leading-frameworks/
>
>
> All in all seems like a well-done benchmark with lots of good data.
>
> Note he is benchmarking TT performance as well.  Although I guess he  
> is benchmarking everyone else's template libraries too.  To  
> benchmark the framework itself, you should just send some static  
> data.  I think with 5 FastCGI procs he should get more requests per  
> second though.  I'm not familiar with nginx to know if that has  
> something to do with the numbers.  Will be worth benchmarking it  
> with lighttpd as well.
>
> Another interesting thing is in the hello.tt file he is not using a  
> variable for the 'hello' string like he did in most other  
> frameworks.  Wouldn't help performance, though.  Also don't know if  
> he's using Template::Stash::XS or template caching, since these are  
> not enabled by default in View::TT.
>
> I'll see about posting some benchmarks of our various engines with  
> this same test data, including a fast new engine I'm working on. :)

Heh, I didn't realize how old this was... still, I think we could use  
a good list of benchmarks for various engines.

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[dab-3]

Thank you for your answers
If asked that question is that I make a small application that takes 3
seconds per request with only one user. It has only 7 tables and uses a
lot of TT.
I forget to use /Template::Stash::XS, (thank /Andy) I will make a comparison

Sorry for my poor english

Andy Grundman a écrit :

>
> On Feb 29, 2008, at 12:14 PM, Andy Grundman wrote:
>
>>
>> On Feb 29, 2008, at 11:21 AM, dab@... wrote:
>>
>>> hi
>>> I do not know if this test has already been posted on the list
>>> Why these bad performances ?
>>> http://www.alrond.com/en/2007/jan/25/performance-test-of-6-leading-frameworks/ 
>>>
>>
>>
>> All in all seems like a well-done benchmark with lots of good data.
>>
>> Note he is benchmarking TT performance as well.  Although I guess he
>> is benchmarking everyone else's template libraries too.  To benchmark
>> the framework itself, you should just send some static data.  I think
>> with 5 FastCGI procs he should get more requests per second though.  
>> I'm not familiar with nginx to know if that has something to do with
>> the numbers.  Will be worth benchmarking it with lighttpd as well.
>>
>> Another interesting thing is in the hello.tt file he is not using a
>> variable for the 'hello' string like he did in most other
>> frameworks.  Wouldn't help performance, though.  Also don't know if
>> he's using Template::Stash::XS or template caching, since these are
>> not enabled by default in View::TT.
>>
>> I'll see about posting some benchmarks of our various engines with
>> this same test data, including a fast new engine I'm working on. :)
>
> Heh, I didn't realize how old this was... still, I think we could use
> a good list of benchmarks for various engines.
>
> _______________________________________________
> List: Catalyst@...
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@.../
> Dev site: http://dev.catalyst.perl.org/

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Andy Grundman]

On Feb 29, 2008, at 2:20 PM, Dab wrote:

> Thank you for your answers
> If asked that question is that I make a small application that takes  
> 3 seconds per request with only one user. It has only 7 tables and  
> uses a lot of TT.
> I forget to use /Template::Stash::XS, (thank /Andy) I will make a  
> comparison

3 seconds per request is definitely an app-level problem, maybe your  
database queries?  Debug output shows execution time per method, may  
help you locate the problem.


_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Jonathan Rockway]

* On Fri, Feb 29 2008, Dab wrote:
> Thank you for your answers
> If asked that question is that I make a small application that takes 3
> seconds per request with only one user. It has only 7 tables and uses
> a lot of TT.
> I forget to use /Template::Stash::XS, (thank /Andy) I will make a comparison

Rather than guessing, just fire up the profiler (Devel::DProf) and see
what's taking a long time.  Then you can speed that part up.

I usually do this:

  in Controller::Root:

  sub quit :Local { exit(0) if $ENV{MYAPP_EXIT_OK} }

Then:

  $ MYAPP_EXIT_OK=1 perl -Ilib -d:DProf script/myapp_server.pl
  $ ab -n 1000 http://localhost:3000/slow/page
  $ GET http://localhost:3000/quit
  $ dprofpp

  <listing of slow routines>

An improvement would be to just add this to your Makefile.PL:

  sub MY::postamble {
     return <<"EOM";
     profile:
     \tMYAPP_EXIT_OK=1 perl -Ilib -d:DProf script/myapp_server.pl
     \tab -n 1000 http://localhost:3000/slow/page
     \tGET http://localhost:3000/quit
     \tdprofpp
     \trm tmon.out # if you want
  EOM
  }

Then "make profile".  Untested.

Regards,
Jonathan Rockway

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Ashley Pond V]

That's a hot tip. Put it in the mojomojo. Even if untested now,  
someone will eventually and update it if necessary.

On Feb 29, 2008, at 2:03 PM, Jonathan Rockway wrote:

> * On Fri, Feb 29 2008, Dab wrote:
>> Thank you for your answers
>> If asked that question is that I make a small application that  
>> takes 3
>> seconds per request with only one user. It has only 7 tables and uses
>> a lot of TT.
>> I forget to use /Template::Stash::XS, (thank /Andy) I will make a  
>> comparison
>
> Rather than guessing, just fire up the profiler (Devel::DProf) and see
> what's taking a long time.  Then you can speed that part up.
>
> I usually do this:
>
>   in Controller::Root:
>
>   sub quit :Local { exit(0) if $ENV{MYAPP_EXIT_OK} }
>
> Then:
>
>   $ MYAPP_EXIT_OK=1 perl -Ilib -d:DProf script/myapp_server.pl
>   $ ab -n 1000 http://localhost:3000/slow/page
>   $ GET http://localhost:3000/quit
>   $ dprofpp
>
>   <listing of slow routines>
>
> An improvement would be to just add this to your Makefile.PL:
>
>   sub MY::postamble {
>      return <<"EOM";
>      profile:
>      \tMYAPP_EXIT_OK=1 perl -Ilib -d:DProf script/myapp_server.pl
>      \tab -n 1000 http://localhost:3000/slow/page
>      \tGET http://localhost:3000/quit
>      \tdprofpp
>      \trm tmon.out # if you want
>   EOM
>   }
>
> Then "make profile".  Untested.
>
> Regards,
> Jonathan Rockway
>
> _______________________________________________
> List: Catalyst@...
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/ 
> catalyst@.../
> Dev site: http://dev.catalyst.perl.org/

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[dab-3]

Thanks very much Jonathan for your tips

With this and Catalyst::Model::DBIC::Schema::QueryLog i think i have found the problem.
oups ... Queries: 2027 :)



Jonathan Rockway a écrit :

> * On Fri, Feb 29 2008, Dab wrote:
>  
>> Thank you for your answers
>> If asked that question is that I make a small application that takes 3
>> seconds per request with only one user. It has only 7 tables and uses
>> a lot of TT.
>> I forget to use /Template::Stash::XS, (thank /Andy) I will make a comparison
>>    
>
> Rather than guessing, just fire up the profiler (Devel::DProf) and see
> what's taking a long time.  Then you can speed that part up.
>
> I usually do this:
>
>   in Controller::Root:
>
>   sub quit :Local { exit(0) if $ENV{MYAPP_EXIT_OK} }
>
> Then:
>
>   $ MYAPP_EXIT_OK=1 perl -Ilib -d:DProf script/myapp_server.pl
>   $ ab -n 1000 http://localhost:3000/slow/page
>   $ GET http://localhost:3000/quit
>   $ dprofpp
>
>   <listing of slow routines>
>
> An improvement would be to just add this to your Makefile.PL:
>
>   sub MY::postamble {
>      return <<"EOM";
>      profile:
>      \tMYAPP_EXIT_OK=1 perl -Ilib -d:DProf script/myapp_server.pl
>      \tab -n 1000 http://localhost:3000/slow/page
>      \tGET http://localhost:3000/quit
>      \tdprofpp
>      \trm tmon.out # if you want
>   EOM
>   }
>
> Then "make profile".  Untested.
>
> Regards,
> Jonathan Rockway
>
> _______________________________________________
> List: Catalyst@...
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst@.../
> Dev site: http://dev.catalyst.perl.org/
>  

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Matt S Trout-2]

On Sat, Mar 01, 2008 at 01:13:13AM +0100, Dab wrote:
> Thanks very much Jonathan for your tips
>
> With this and Catalyst::Model::DBIC::Schema::QueryLog i think i have found
> the problem.
> oups ... Queries: 2027 :)

This is why we're not overly worried about being slower than other things
at 'Hello World' - that's really not the point, the point is to create an
abstraction that lets you write the rest of your app faster and more
maintainably, then you'll have more time to spend finding the things that
are -actually- making your app slow - which is almost always either the
templating or the DB stuff :)

--
      Matt S Trout       Need help with your Catalyst or DBIx::Class project?
   Technical Director                    http://www.shadowcat.co.uk/catalyst/
 Shadowcat Systems Ltd.  Want a managed development or deployment platform?
http://chainsawblues.vox.com/            http://www.shadowcat.co.uk/servers/

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/