how would you scale a cake app?

[keymaster]

.. just dreaming, but..

If one wanted to use cake to build an app which is scalable to the
extreme, ie. flickr.com or yourtube.com or myspace.com, how would you
do it?

Would you just separate out the db server and replicate the rest on
identical nodes in a cluster? Would you break up the app into view
servers, controller servers, and model servers?

Imagine you were tasked with building one of these systems, what system
architecture would you have?


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[Langdon Stevenson]

It's an interesting question.  I am working on an application that (if
it works out for the client) may grow to support thousands of users with
requirements for high availability.

If it ever grows that far, then I expect that we will need to redevelop
at least some parts of the application .

For me, the first step (to a small cluster) would be to hive off the
database onto a cluster of three to five servers.  DB access on this
system is heavy, so that will be the likely first bottleneck.

After that, I am not sure.  I think that I would likely just replicate
the entire application across multiple web servers, and hand out the
requests in a round robin to the web servers.

I would be very interested to hear anyone else's thoughts on this too.
Does the development team have any recommendations about the best
approach to scaling up a site?

Regards,
Langdon


keymaster wrote:

>
> .. just dreaming, but..
>
> If one wanted to use cake to build an app which is scalable to the
> extreme, ie. flickr.com or yourtube.com or myspace.com, how would you
> do it?
>
> Would you just separate out the db server and replicate the rest on
> identical nodes in a cluster? Would you break up the app into view
> servers, controller servers, and model servers?
>
> Imagine you were tasked with building one of these systems, what system
> architecture would you have?

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[Nimrod A. Abing]

On 12/24/06, keymaster <admin@...> wrote:
>
> .. just dreaming, but..
>
> If one wanted to use cake to build an app which is scalable to the
> extreme, ie. flickr.com or yourtube.com or myspace.com, how would you
> do it?

http://www.onjava.com/pub/a/onjava/2003/10/15/php_scalability.html

http://www.oreillynet.com/onlamp/blog/2006/04/digg_phps_scalability_and_perf.html

> Would you just separate out the db server and replicate the rest on
> identical nodes in a cluster? Would you break up the app into view
> servers, controller servers, and model servers?

That depends. Scalability is a difficult issue to handle if it is not
considered during the initial application design stage. If you want to
be able to scale up your application, you might as well decide on a
strategy *now*.
--
_nimrod_a_abing_

[?] http://abing.gotdns.com

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[Chris Lamb-3]

Langdon Stevenson <langdon@...> wrote:
> I think that I would likely just replicate the entire application
> across multiple web servers, and hand out the requests in a round
> robin to the web servers.

Or maybe FastCGI's load balancing?

> I would be very interested to hear anyone else's thoughts on this
> too.

Good use of caching would help prolong the time before having to scale
up, as well as reducing hardware requirements. Cake's own caching is
really neat, as are things like lighttpd's mod_cml or traditional
options like squid. Remember that web servers are *extremely* good at
serving static content!

I should really add the general warning against premature optimisation
here, but I'm sure you've heard it before.


Best wishes,

--
 Chris Lamb, Cambridgeshire, UK                        GPG: 0x634F9A20

[Mariano Iglesias]

Check this out, it's not that high of an explanation but provides some
useful tips on how to improve PHP performance:

http://talks.php.net/show/hpp

For some reason the creators of that presentation decided that they don't
support IE so you'll need to use FF or something else to view it.
 
In regards to your question I also think that scalability should be on the
table right from the get-go. CakePHP is a framework that allows you to
easily develop your application thinking about scalability all the way, so
take advantage of that.

I believe that in time, as your traffic hits high numbers, the first
optimization you'll need is to allow the DB to be on its own clustered
servers, so will your webserver, having giga connection amongst the two.
Also if you are using MySQL as your backend server move CakePHP towards
using mysqli extension instead of mysql, as it is faster. Also don't neglect
the connectivity your server has to the outside world, some people improve
their application performance and forget about the backbones where their
application is running.

Separating Models / Controllers: oh well I don't think you'll need such a
thing. I would have DB on one side, CakePHP application on another side,
work my way to PHP optimization on the CakePHP side and enable caching to
places where it can be used.

-MI

---------------------------------------------------------------------------

Remember, smart coders answer ten questions for every question they ask.
So be smart, be cool, and share your knowledge.

BAKE ON!


-----Mensaje original-----
De: cake-php@... [mailto:cake-php@...] En nombre
de Langdon Stevenson
Enviado el: Domingo, 24 de Diciembre de 2006 05:33 a.m.
Para: cake-php@...
Asunto: Re: how would you scale a cake app?

I would be very interested to hear anyone else's thoughts on this too.
Does the development team have any recommendations about the best
approach to scaling up a site?


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[Olivier percebois-Garve]

A simple way to let a webapp scale is to moved session's and db's data
to cookies as much as possible.

olivvv


Nimrod A. Abing wrote:

>
> On 12/24/06, keymaster <admin@...> wrote:
>>
>> .. just dreaming, but..
>>
>> If one wanted to use cake to build an app which is scalable to the
>> extreme, ie. flickr.com or yourtube.com or myspace.com, how would you
>> do it?
>
> http://www.onjava.com/pub/a/onjava/2003/10/15/php_scalability.html
>
> http://www.oreillynet.com/onlamp/blog/2006/04/digg_phps_scalability_and_perf.html 
>
>
>> Would you just separate out the db server and replicate the rest on
>> identical nodes in a cluster? Would you break up the app into view
>> servers, controller servers, and model servers?
>
> That depends. Scalability is a difficult issue to handle if it is not
> considered during the initial application design stage. If you want to
> be able to scale up your application, you might as well decide on a
> strategy *now*.

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[keymaster]

On Dec 24, 4:01 pm, "Mariano Iglesias" <mariano.igles...@...>
wrote:

> CakePHP is a framework that allows you to
> easily develop your application thinking about scalability all the way, so
> take advantage of that.

I understand how the MVC separation and strict coding conventions make
maintainability easier as the application gets larger (in terms of
lines of code). This I know cake is outstanding at.

But you seem to be alluding to a feature of cake I have not yet
digested (pardon the pun), which is an aspect of cake which "allows you
to easily develop your application thinking about scalability all the
way". If you were referring to scalability from a traffic perspective,
together with the associated underlying system architectural changes
needed, can you please elaborate for us what you meant, Mariano ?

Thanks.


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[bgmill]

I'm running a cake app on 3 webservers (round robin dns) and 1 mysql
server right now, traffic yesterday was around 200,000 uniques and
400,000 page views.

Caching is being used on some of the heavier hit controllers but all in
all, load is minimal across the board -- we could easily handle 2 or 3
times the traffic on the current setup. Scaling up in terms of adding
webservers and db servers when required will be easy to accomplish.


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[Mariano Iglesias]

Your situation sounds like a great source for a potential bakery article,
mainly for:

1. Understanding how CakePHP is behaving on a high-traffic/clustered server.

2. Giving a short tutorial on how you managed to add new servers and how it
impacted on your application.

Just a thought :)

-MI

---------------------------------------------------------------------------

Remember, smart coders answer ten questions for every question they ask.
So be smart, be cool, and share your knowledge.

BAKE ON!


-----Mensaje original-----
De: cake-php@... [mailto:cake-php@...] En nombre
de bgmill
Enviado el: Domingo, 24 de Diciembre de 2006 01:29 p.m.
Para: Cake PHP
Asunto: Re: how would you scale a cake app?

I'm running a cake app on 3 webservers (round robin dns) and 1 mysql
server right now, traffic yesterday was around 200,000 uniques and
400,000 page views.


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[Mariano Iglesias]

I am talking about what the CakePHP framework offers you in terms of
scalability, things like:

1. View caching: you can cache views as a whole or even specify which parts
of a view should not be cached: http://manual.cakephp.org/chapter/view_cache

2. Model/Object caching: cache model queries or even complex objects
returned by components,
http://rossoft.wordpress.com/2006/03/15/cache-objects-component/

3. Database optimization: it is easy, if you need it, to have a DB engine
running your updates and another DB engine (that replicates previous record
updates) to be used only for querying:
http://groups-beta.google.com/group/cake-php/browse_thread/thread/58ea010f93
0fab6c

MVC helps maintain your code, but a powerful framework can also help you
boost performance by making code optimizations that are transparent to your
application logic. That is one of the things CakePHP is great at.

-MI

---------------------------------------------------------------------------

Remember, smart coders answer ten questions for every question they ask.
So be smart, be cool, and share your knowledge.

BAKE ON!


-----Mensaje original-----
De: cake-php@... [mailto:cake-php@...] En nombre
de keymaster
Enviado el: Domingo, 24 de Diciembre de 2006 01:21 p.m.
Para: Cake PHP
Asunto: Re: how would you scale a cake app?

I understand how the MVC separation and strict coding conventions make
maintainability easier as the application gets larger (in terms of
lines of code). This I know cake is outstanding at.

But you seem to be alluding to a feature of cake I have not yet
digested (pardon the pun), which is an aspect of cake which "allows you
to easily develop your application thinking about scalability all the
way". If you were referring to scalability from a traffic perspective,
together with the associated underlying system architectural changes
needed, can you please elaborate for us what you meant, Mariano ?


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[bgmill]

I may well write an article for the bakery after the holiday season -
are there any other high traffic sites known to be using cake?


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[Langdon Stevenson]

Hi bgmill

Sounds like you have a good system/site there.  Can you tell me what
sort of hardware it is running on?

Regards,
Langdon

> I'm running a cake app on 3 webservers (round robin dns) and 1 mysql
> server right now, traffic yesterday was around 200,000 uniques and
> 400,000 page views.
>
> Caching is being used on some of the heavier hit controllers but all in
> all, load is minimal across the board -- we could easily handle 2 or 3
> times the traffic on the current setup. Scaling up in terms of adding
> webservers and db servers when required will be easy to accomplish.

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[the_woodsman]

Caching, and lots of it, and DB clustering/replication are definitely
the quickest fixes.

One minor point is to avoid using Cake's multi join SQL, which (I
believe) can literally add thousands of queries when a single one would
do.

In such cases, don't be afraid to use query() instead of findAll -
although examining your DB for low performance or often repeated
queries is great for deciding if the benefits are relly worth the
effort.

And remember, hardware is less expensive than development time, so
don't sacrifice code maintainability and clarity for tiny performance
increases, as you'll pay a lot more in the long run.


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[Olexandr Melnyk]

Also, check if Cake doesn't get more than you need, and consider dynamic unbinding where you don't need as much info as you get using default models' options.

----------------------------------
Olexandr Melnyk,
http://omelnyk.net/


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[Dr. Tarique Sani]

On 12/24/06, bgmill <bmilleare@...> wrote:
> I'm running a cake app on 3 webservers (round robin dns) and 1 mysql
> server right now, traffic yesterday was around 200,000 uniques and
> 400,000 page views.

What kind of hardware are you using?

Not to sound like a prick but 400,000 page views are not much for a
typical 3+1 configuration. Given your estimate of higher performance
you say you will reach about 1.2 million page views on the same setup
which again makes me want to know your hardware configuration.

To put things from my perspective - we have conventional php
application which manages 12 to 15 million page views on a 4+2
cluster. Yes the hardware on this is awesome!

However this is a conventional php application which started life
about 4 years ago,  now maintaining it is an daunting task - adding
functionality to it gives best of my programmers shivers  :) and that
is where a framework comes in...

Looking forward to hear from you

Thanks and merry Christmas

Tarique


--
=============================================================
PHP Applications for E-Biz: http://www.sanisoft.com
Coppermine Picture Gallery: http://coppermine.sf.net
=============================================================

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[Langdon Stevenson]

Hi Woodsman

> Caching, and lots of it, and DB clustering/replication are definitely
> the quickest fixes.

Caching (both on the server and the client side) are certainly looking
good for my current project.  I haven't used Cake's view caching yet,
but am definitely going to look into it.  I can well imagine the benefits


> One minor point is to avoid using Cake's multi join SQL, which (I
> believe) can literally add thousands of queries when a single one would
> do.

I have spent some time optimising queries just with unbinding and have
seen substantial improvements in DB performance (like half or one third
of the processing time), so I can well believe the potential for
improvement there.


> And remember, hardware is less expensive than development time, so
> don't sacrifice code maintainability and clarity for tiny performance
> increases, as you'll pay a lot more in the long run.

This has become one of my main mantra.  Yes, hardware is very cheap,
especially compared to coding labor.  And there is also nothing stopping
you optimising code over time as well to take full advantage of that
hardware.

Regards,
Langdon

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[bigty]

That's a dream!
http://www.mytechsky.com

On 12 24 ,   3 10 , "keymaster" <a...@...>
wrote:

> .. just dreaming, but..
>
> If one wanted to use cake to build an app which is scalable to the
> extreme, ie. flickr.com or yourtube.com or myspace.com, how would you
> do it?
>
> Would you just separate out the db server and replicate the rest on
> identical nodes in a cluster? Would you break up the app into view
> servers, controller servers, and model servers?
>
> Imagine you were tasked with building one of these systems, what system
> architecture would you have?

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[bgmill]

Dr. Tarique Sani wrote:

> Not to sound like a prick but 400,000 page views are not much for a
> typical 3+1 configuration. Given your estimate of higher performance
> you say you will reach about 1.2 million page views on the same setup
> which again makes me want to know your hardware configuration.

Firstly, I don't claim to be a hardware guru, and my estimations are
purely based on brief conversations with the techs who run the servers.
So with that in mind... all the machines are dual xeon 2.8GHz with 2Gb
of memory. But like I said, the machines have not flinched even at peak
traffic so far (I've rarely seen them above 0.1), whether we could
handle 1.2mil or 10mil page views a day on the current hardware was
just speculation on my end.

The decision to use 3+1 straight off the bat was mainly due to the need
for handling rapid expansion -- the site only launched 2 weeks ago. Add
to the mix the need for video transcoding and an element of the unknown
in this respect meant it was easier to start with more than less.

> To put things from my perspective - we have conventional php
> application which manages 12 to 15 million page views on a 4+2
> cluster. Yes the hardware on this is awesome!

Sounds like a nice setup you have there :)

Regards.


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[Langdon Stevenson]

Hi bgmill

> The decision to use 3+1 straight off the bat was mainly due to the need
> for handling rapid expansion

Thanks for that info, much appreciated.  One more question though, when
you say "3+1" do you mean a cluster of 3 database servers and one web
server?

Regards,
Langdon

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---

[Dr. Tarique Sani]

On 12/27/06, Langdon Stevenson <langdon@...> wrote:
> Thanks for that info, much appreciated.  One more question though, when
> you say "3+1" do you mean a cluster of 3 database servers and one web
> server?

I think he said 3 web servers + 1 MySQL, MySQL server is probably a
master and handles all the writes and then replicated back on the 3
web servers which are used for subsequent reads. but I am just
speculating here

Cheers
Tarique

--
=============================================================
PHP Applications for E-Biz: http://www.sanisoft.com
Cheesecake-Photoblog: http://cheesecake-photoblog.org
=============================================================

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups "Cake PHP" group.
To post to this group, send email to cake-php@...
To unsubscribe from this group, send email to cake-php-unsubscribe@...
For more options, visit this group at http://groups.google.com/group/cake-php?hl=en
-~----------~----~----~----~------~----~------~--~---