CiviMail message delivery performance

[Fen Labalme-2]

Hi -

I'm looking for some real world numbers on CiviMail performance, particularly
with large mailouts.  We are seeing about 180 message deliveries per second on
a lightly loaded 2 GHz (older architecture) Xeon using CiviCRM v1.6 with
Drupal 5.1, MySQL 5, PHP 5.1, and Apache 2.2.

What I'm looking for is an order of magnitude performance increase eventually,
which I don't think I'll get with reasonable hardware upgrades.  But can I get
a 2 or 3 times speed boost?

Lobo has suggested moving the CiviCRM DB (and perhaps also the Drupal DB) to
another machine to give the web server maximum access to CPU & memory cache
and reduce context switching.  And even going to multiple front-end web
servers.  This is quite an expense (the latter is certainly outside our
budget) but I'm wondering: are others finding significant response gains from
moving the database to another machine?

Is anyone out there seeing 500 or more message deliveries per second?  If so,
what is your hardware platform/architecture?

Thanks in advance,
=Fen
____________________________________________________________
You received this message as a subscriber on the list:
     civicrm-mail@...
To be removed from the list, send any message to:
     civicrm-mail-unsubscribe@...

For all list information and functions, see:
     http://lists.civicrm.org/lists/info/civicrm-mail

[Donald Lobo]

(cc;ing civicrm-dev since this might be of interest to that list also)

a few clarifications and some of my own benchmarks:

1. We really have not spent a lot of time optimizing the speed and
throughput of CiivCRM. I suspect if we spend a week or so, we should
be able to squeeze out a 2-3x improvement. All numbers approximate.
Would be awesome for someone from the community to step up and either
lead the charge or sponsor the effort or both. Ideally our goal should
be to send at least 100K - 250K mails/hour on a 2 or 3 machine
architecture (modern hardware, web server one, db server on the other,
email server on the third (or share with db). This basically comes
down to 1666 - 4166 emails / minute

2. I did a few test email on my local Powerbook with 1G ram and a 2.16
G Intel Core Duo. Running php5.2.2 mysql5.0.37  (and on trunk). I also
have zend platform installed (they have a free developer version and
comes highly recommended!)

3. generated sample data with first name / last name and email. My
mail messages had a few tokens in there in addition to the required
tokens. all the email addresses ended in example.com, i.e. the local
smtp server could not deliver them

4. sending a 1000 email via civimail took between 55-65 seconds. I
also tested this with a 10000 email blast. so we are pretty close to
1000 mails / min which is not bad for a labtop :). I also have mysql
logging all queries and not really optimized with large cache sizes
etc

So i think we are in a pretty good place to begin with :) I think we
can really make civicrm industrial strength and rock solid if we
invest the time and resources on this. If you or your organization can
contribute significantly to this please let us know. I think the
community will benefit greatly with more changes and optimization :)

lobo

On 5/10/07, Fen Labalme <fen@...> wrote:

> Hi -
>
> I'm looking for some real world numbers on CiviMail performance, particularly
> with large mailouts.  We are seeing about 180 message deliveries per second on
> a lightly loaded 2 GHz (older architecture) Xeon using CiviCRM v1.6 with
> Drupal 5.1, MySQL 5, PHP 5.1, and Apache 2.2.
>
> What I'm looking for is an order of magnitude performance increase eventually,
> which I don't think I'll get with reasonable hardware upgrades.  But can I get
> a 2 or 3 times speed boost?
>
> Lobo has suggested moving the CiviCRM DB (and perhaps also the Drupal DB) to
> another machine to give the web server maximum access to CPU & memory cache
> and reduce context switching.  And even going to multiple front-end web
> servers.  This is quite an expense (the latter is certainly outside our
> budget) but I'm wondering: are others finding significant response gains from
> moving the database to another machine?
>
> Is anyone out there seeing 500 or more message deliveries per second?  If so,
> what is your hardware platform/architecture?
>
> Thanks in advance,
> =Fen
> ____________________________________________________________
> You received this message as a subscriber on the list:
>      civicrm-mail@...
> To be removed from the list, send any message to:
>      civicrm-mail-unsubscribe@...
>
> For all list information and functions, see:
>      http://lists.civicrm.org/lists/info/civicrm-mail
>

--
lobo

http://civicrm.org/blog/
http://civicrm.org/
http://lobostravel.blogspot.com/
____________________________________________________________
You received this message as a subscriber on the list:
     civicrm-mail@...
To be removed from the list, send any message to:
     civicrm-mail-unsubscribe@...

For all list information and functions, see:
     http://lists.civicrm.org/lists/info/civicrm-mail

[Donald Lobo]

forgot another point

5. I wanted to see how much time we were spending handing the mail off
to the smtp server. We are spending pretty close to 50% of the time in
the smtp transaction. We could optimize this quite a bit by having
multiple processes crunch the mail queue at the same time

I'll document and blog about this soon ..

lobo


On 5/10/07, Donald Lobo <donald.lobo@...> wrote:

> (cc;ing civicrm-dev since this might be of interest to that list also)
>
> a few clarifications and some of my own benchmarks:
>
> 1. We really have not spent a lot of time optimizing the speed and
> throughput of CiivCRM. I suspect if we spend a week or so, we should
> be able to squeeze out a 2-3x improvement. All numbers approximate.
> Would be awesome for someone from the community to step up and either
> lead the charge or sponsor the effort or both. Ideally our goal should
> be to send at least 100K - 250K mails/hour on a 2 or 3 machine
> architecture (modern hardware, web server one, db server on the other,
> email server on the third (or share with db). This basically comes
> down to 1666 - 4166 emails / minute
>
> 2. I did a few test email on my local Powerbook with 1G ram and a 2.16
> G Intel Core Duo. Running php5.2.2 mysql5.0.37  (and on trunk). I also
> have zend platform installed (they have a free developer version and
> comes highly recommended!)
>
> 3. generated sample data with first name / last name and email. My
> mail messages had a few tokens in there in addition to the required
> tokens. all the email addresses ended in example.com, i.e. the local
> smtp server could not deliver them
>
> 4. sending a 1000 email via civimail took between 55-65 seconds. I
> also tested this with a 10000 email blast. so we are pretty close to
> 1000 mails / min which is not bad for a labtop :). I also have mysql
> logging all queries and not really optimized with large cache sizes
> etc
>
> So i think we are in a pretty good place to begin with :) I think we
> can really make civicrm industrial strength and rock solid if we
> invest the time and resources on this. If you or your organization can
> contribute significantly to this please let us know. I think the
> community will benefit greatly with more changes and optimization :)
>
> lobo
>
> On 5/10/07, Fen Labalme <fen@...> wrote:
> > Hi -
> >
> > I'm looking for some real world numbers on CiviMail performance, particularly
> > with large mailouts.  We are seeing about 180 message deliveries per second on
> > a lightly loaded 2 GHz (older architecture) Xeon using CiviCRM v1.6 with
> > Drupal 5.1, MySQL 5, PHP 5.1, and Apache 2.2.
> >
> > What I'm looking for is an order of magnitude performance increase eventually,
> > which I don't think I'll get with reasonable hardware upgrades.  But can I get
> > a 2 or 3 times speed boost?
> >
> > Lobo has suggested moving the CiviCRM DB (and perhaps also the Drupal DB) to
> > another machine to give the web server maximum access to CPU & memory cache
> > and reduce context switching.  And even going to multiple front-end web
> > servers.  This is quite an expense (the latter is certainly outside our
> > budget) but I'm wondering: are others finding significant response gains from
> > moving the database to another machine?
> >
> > Is anyone out there seeing 500 or more message deliveries per second?  If so,
> > what is your hardware platform/architecture?
> >
> > Thanks in advance,
> > =Fen
> > ____________________________________________________________
> > You received this message as a subscriber on the list:
> >      civicrm-mail@...
> > To be removed from the list, send any message to:
> >      civicrm-mail-unsubscribe@...
> >
> > For all list information and functions, see:
> >      http://lists.civicrm.org/lists/info/civicrm-mail
> >
>
>
> --
> lobo
>
> http://civicrm.org/blog/
> http://civicrm.org/
> http://lobostravel.blogspot.com/
>

--
lobo

http://civicrm.org/blog/
http://civicrm.org/
http://lobostravel.blogspot.com/
____________________________________________________________
You received this message as a subscriber on the list:
     civicrm-mail@...
To be removed from the list, send any message to:
     civicrm-mail-unsubscribe@...

For all list information and functions, see:
     http://lists.civicrm.org/lists/info/civicrm-mail

[shanehill]

Hi Fen,

I wanted to point out the optimizations for civimail that will be going into 1.9.  I was able to squeeze an order of magnitude of performance out of civimail email generation recently.

also, at some point in the future (3.0?) there will be a postfix queue injector for civimail which will further enhance delivery performance.

does someone want to stand up and provide / sponsor a postfix queue injector?  I will do it eventually.
There is code here that could be used, abused or ported perhaps to php for civimail postfix queue injection:

http://svn.perl.org/viewcvs/qpsmtpd/trunk/lib/Qpsmtpd/Postfix.pm?view=log

the injector library code is linked from this civicrm issue:

http://issues.civicrm.org/jira/browse/CRM-2101

-Shane

Fen Labalme-2 wrote:

Hi -

I'm looking for some real world numbers on CiviMail performance, particularly
with large mailouts.  We are seeing about 180 message deliveries per second on
a lightly loaded 2 GHz (older architecture) Xeon using CiviCRM v1.6 with
Drupal 5.1, MySQL 5, PHP 5.1, and Apache 2.2.

What I'm looking for is an order of magnitude performance increase eventually,
which I don't think I'll get with reasonable hardware upgrades.  But can I get
a 2 or 3 times speed boost?

Lobo has suggested moving the CiviCRM DB (and perhaps also the Drupal DB) to
another machine to give the web server maximum access to CPU & memory cache
and reduce context switching.  And even going to multiple front-end web
servers.  This is quite an expense (the latter is certainly outside our
budget) but I'm wondering: are others finding significant response gains from
moving the database to another machine?

Is anyone out there seeing 500 or more message deliveries per second?  If so,
what is your hardware platform/architecture?

Thanks in advance,
=Fen
____________________________________________________________
You received this message as a subscriber on the list:
     [e-mail]
To be removed from the list, send any message to:
     [e-mail]

For all list information and functions, see:
     http://lists.civicrm.org/lists/info/civicrm-mail