horizontal scaling

[mulc]

 

Scaling horizontally.

I am currently using my own home-brewed python web-application framework,
but I would like to migrate to a better option.  I have always admired Zope
from afar ( very far ) but I am very intrigued by Zope3 and ( by extension )
Grok.   I bought Philipp and Stephan's books and I have been trying to
determine if Zope3 technologies are what I am looking for.

I have discarded the other python "Rails-alike" frameworks because as far as
I can tell, they ( and Rails ) suffer from the same problem.   Every
web-application ( no matter how small or large ) requires its own
long-running process or processes.   The larger an application becomes, the
more functionality it acquires, the larger its resource foot-print becomes
even if it is not handling any requests.   This does not scale.

I wrote my own python framework because although I loathe PHP, I recognized
that it had some undeniable strengths.   I own a small web-hosting company,
and I buy, install,  and administer the machines my applications run on, so
I have a very SysAdmin ( as opposed to Web-Developer ) way of looking at
things.

PHP's strength?   Horizontal Scaling ( for lack of a better word ).

A web-server might have 200-odd virtual-hosts serving 200 different crappy
PHP bulletin-boards, dodgy shopping carts, and bug-ridden CMS applications.  
The run-time costs of hosting these 200 different applications in 200
different Apache virtual hosts at 2:30 AM when the requests dribble in at
1/second or so?   Next to nothing.   The resource bloat of mod_php in all
thoses httpds is neglible ( the dodgy code bloat is something else. )    
Under load, the "cost" of hosting all these disparate apps with 3 to 300 php
scripts each, is simply the runtime cost of each script request.    It

Note, I am not advocating for PHP.  Its weaknesses are too numerous to
mention, and as a matter of fact I don't host hundreds of crappy PHP
bulletin-boards, blogs, and CMS apps but it is for reasons of security, the
prospect of painful unpaid code-maintainence, and general aesthetics, not
resource utilization.   All I am saying is that it is possible to host
hundreds of websites each with hundreds of possible dynamic URLs with
processor and memory to spare on 4 year-old hardware because the general
"machinery" for processing all these requests is  ready and running ( I
don't want to say "in-process" because I don't believe in the fat buggy
httpds brought on by the whole mod_perl, mod_python thing and "external
process" techniques like SCGI work very well for me ) and GENERIC, each
mod_php enabled httpd, or php fast_cgi process capable of serving one
"application" as well as another from request to request.

Zope3 is very large and I haven't been able to get the 30,000 foot view yet.

Most simply I suppose:
Is it possible for 1 zope3 application server ( or a family of zope3
application servers sharing a ZEO or relstorage-like back-end ) to provide
multiple Grok application instances for many ( maybe not 200, but 30 or 40 )
different virtual hosts ( fronted by squid, apache et.al ) .

Can an application that has not been accessed in hours ( or days ) be
"passivated" and its runtime resource costs be reduced to disk-space and a
tiny entry in an in-memory hashtable?

I know that zope3  ( and thereby Grok ) is a lot heavier than my own
framework, but it brings a lot more to the table.
I just can't afford to run one for every site ( large or small ) that needs
dynamic content.

I have a vision of  one redundant zope3-cluster ( backed by relstorage on a
slony-replicated postgresql cluster ) that can provide all of the
web-application infrastructure for all of my development work for various
disparate customers going forward.

If this is feasible?

If this were possible it would be a slam-dunk for the adoption of grok/zope3
over rails, turbogears, django, et.al.

Regards,

Christopher  Mulcahy

_______________________________________________
Grok-dev mailing list
Grok-dev@...
http://mail.zope.org/mailman/listinfo/grok-dev

[Martijn Faassen-2]

Hey,

Christopher Mulcahy wrote:
[snip discussion of horizontal scaling]
> Zope3 is very large and I haven't been able to get the 30,000 foot view
> yet.
> Most simply I suppose:
> Is it possible for 1 zope3 application server ( or a family of zope3
> application servers sharing a ZEO or relstorage-like back-end ) to
> provide multiple Grok application instances for many ( maybe not 200,
> but 30 or 40 ) different virtual hosts ( fronted by squid, apache et.al ).

Good question. I don't have experience with this amount of scaling. I
think the best chance to get people who do have such experience (if they
exist) is to ask this question on zope-dev as well. But there are some
of those people hanging out on grok-dev, so let's see whether someone
else replies. Christian Theune (who will be at the grokkerdam sprint
tomorrow) may be able to give interesting feedback, for instance.

> Can an application that has not been accessed in hours ( or days ) be
> "passivated" and its runtime resource costs be reduced to disk-space and
> a tiny entry in an in-memory hashtable?

ZODB (and ZEO) do keep a cache around of objects that have been
accessed, so I don't that this will happen on a Zope level. That said,
the operating system may swap out the application if it isn't used a
lot, but that's probably not what you ask for. I don't know what the
memory characteristics of RelStorage are - zope-dev or zodb-dev should
get you better answers.

There are ways emerging to run Zope without the ZODB at all (or just
with a very tiny ZODB), and store the data in a relational database, for
instance using Storm or SQLAlchemy. This may have better memory
characteristics. It's likely a typical relational object mapper still
has an object cache around, but it may be smaller, or at least it may be
easier to make this smaller. I imagine a typical PHP application would
still have a smaller amounts of objects around, as no relational object
mapper is used at all, and PHP also tends to have a "run the page and
then forget about whatever happened" nature.

> I know that zope3  ( and thereby Grok ) is a lot heavier than my own
> framework, but it brings a lot more to the table.
> I just can't afford to run one for every site ( large or small ) that
> needs dynamic content.
> I have a vision of  one redundant zope3-cluster ( backed by relstorage
> on a slony-replicated postgresql cluster ) that can provide all of the
> web-application infrastructure for all of my development work for
> various disparate customers going forward.
> If this is feasible?

It's possible to run multiple applications in a single Zope instance.
There are drawbacks to this - if you upgrade Grok to a newer version (or
just a library), you would need to update all your underlying code, not
just for one application. Also a restart will be interesting - you could
restart each front-end separately until all are using the newer version
of your code.

I think it would also be possible to run different ZEO or RelStorage
frontends that share the same database backend. This way you could do
incremental updates while sharing the same database backend. I think
this would need some special approach though, as even though you'd use
one big database in the back, an individual frontend server would not be
able to access all the objects that are in that ZODB, only the ones that
make sense in the context of its application. I don't think there's
anything fundamentally stopping you from doing so, but I also suspect
there are few tools to support you there.

> If this were possible it would be a slam-dunk for the adoption of
> grok/zope3 over rails, turbogears, django, et.al.

Unfortunately I doubt that there's an easy "yes" or "no" to this answer.
   It also would need to be measured whether such a setup would have
reliability and memory advantages in the real world.

The question would be most easily answered if someone already has done
things like this. I'm not really the right person to answer, but I still
wanted to provide you some feedback. :) zope-dev again would be a good
place to ask.

Thanks for your questions - it's interesting to see this kind of
question coming in; gives us new perspectives and news ideas, even if,
sadly, we may not end up being able to do what you're asking for.

Regards,

Martijn

_______________________________________________
Grok-dev mailing list
Grok-dev@...
http://mail.zope.org/mailman/listinfo/grok-dev