how to increase throughput of postfix to local user?

[devel anaconda]

Hello everybody!

I have a high-performance server (dual quad-core Xeon 2.8Ghz + 16GB RAM + 2SCSI disks 140Gb), RedHat Enterprise Linux 5.4, software RAID1 + Postfix 2.5.9.
This server serves only smtp traffic. The only thing postfix should do - receive mails for one user and send it to local script via pipe. Like this:

user:    |/usr/local/bin/script

The flow is about 250-300 mails per second.Everything is going fine, if there is no queue. But when queue grows to 10000+ letters (for some reasons), postfix loses control of it. deferred queue is null, active queue is almost null, but incoming queue is growing and growing. When incoming queue is about 100k+ letters, postfix just can't pick it up, until I stop incoming traffic. I'm wondering, why such a server can't handle this workload? Even if I do the following:

 user:   /dev/null

postfix can't get the incoming queue over, until I reject incoming smtp traffic... When I tried to figure out the bottleneck, I saw, that there is about 100-150 smtpd processes, 100-150 cleanup process, but there is only 5-8 "local" processes... How to say to qmgr, that it must put as many letters from incoming queue to "local" process, as possible? How to increase parallel delivery? I did this:

default_destination_concurrency_limit   = 200
initial_destination_concurrency         = 200
local_destination_concurrency_limit     = 200

but that didn't help :(

Thank you very much and sorry for my bad English.

[Kenneth Marshall-3]

On Fri, Nov 06, 2009 at 02:19:34AM +0300, devel anaconda wrote:

> Hello everybody!
>
> I have a high-performance server (dual quad-core Xeon 2.8Ghz + 16GB RAM + 2SCSI disks 140Gb), RedHat Enterprise Linux 5.4, software RAID1 + Postfix 2.5.9.
> This server serves only smtp traffic. The only thing postfix should do - receive mails for one user and send it to local script via pipe. Like this:
>
> user:    |/usr/local/bin/script
>
> The flow is about 250-300 mails per second.Everything is going fine, if there is no queue. But when queue grows to 10000+ letters (for some reasons), postfix loses control of it. deferred queue is null, active queue is almost null, but incoming queue is growing and growing. When incoming queue is about 100k+ letters, postfix just can't pick it up, until I stop incoming traffic. I'm wondering, why such a server can't handle this workload? Even if I do the following:
>
>  user:   /dev/null
>

Your system is not a high-performance server I/O-wise. Your two disks can only
handle 200-300 fsync's to disk per second and postfix will always sync your
mail to disk before passing it on for local processing. You will need a
battery backed caching RAID controller or fast SSD drive for the
/var/spool/postfix directories to allow you to go faster. You could maybe
move the spool directory to a RAM or tmpfs file system as well. You lose
the safety net for messages if you have a power or other hardware problem.

> postfix can't get the incoming queue over, until I reject incoming smtp traffic... When I tried to figure out the bottleneck, I saw, that there is about 100-150 smtpd processes, 100-150 cleanup process, but there is only 5-8 "local" processes... How to say to qmgr, that it must put as many letters from incoming queue to "local" process, as possible? How to increase parallel delivery? I did this:
>
> default_destination_concurrency_limit   = 200
> initial_destination_concurrency         = 200
> local_destination_concurrency_limit     = 200
>
> but that didn't help :(
>

It probably never needed more than 5-8 local processes to sink only 200-300
messages per second. That is why you do not see more. Add a "sleep 1" to
your script and see what happens then. :)

Good luck,
Ken

[Corey Chandler]

Kenneth Marshall wrote:
>
> Your system is not a high-performance server I/O-wise. Your two disks can only
> handle 200-300 fsync's to disk per second and postfix will always sync your
> mail to disk before passing it on for local processing. You will need a
> battery backed caching RAID controller or fast SSD drive for the
> /var/spool/postfix directories to allow you to go faster. You could maybe
> move the spool directory to a RAM or tmpfs file system as well. You lose
> the safety net for messages if you have a power or other hardware problem.
>  
For this kind of issue I generally split the laod across multiple
boxes.  It's a lot easier to scale out than it is to keep throwing
hardware at the problem.

-- Corey / KB1JWQ

[Kenneth Marshall-3]

On Thu, Nov 05, 2009 at 04:03:13PM -0800, Corey Chandler wrote:

> Kenneth Marshall wrote:
>>
>> Your system is not a high-performance server I/O-wise. Your two disks can
>> only
>> handle 200-300 fsync's to disk per second and postfix will always sync
>> your
>> mail to disk before passing it on for local processing. You will need a
>> battery backed caching RAID controller or fast SSD drive for the
>> /var/spool/postfix directories to allow you to go faster. You could maybe
>> move the spool directory to a RAM or tmpfs file system as well. You lose
>> the safety net for messages if you have a power or other hardware problem.
>>  
> For this kind of issue I generally split the laod across multiple boxes.  
> It's a lot easier to scale out than it is to keep throwing hardware at the
> problem.
>
> -- Corey / KB1JWQ
>

I just thought of another distasteful option, use a pre-queue content
filter to submit the E-mail and never pass it on to the postfix queue. :)

Ken

[Wietse Venema]

devel anaconda:
> Hello everybody!
>
> I have a high-performance server (dual quad-core Xeon 2.8Ghz + 16GB RAM + 2SCSI disks 140Gb), RedHat Enterprise Linux 5.4, software RAID1 + Postfix 2.5.9.
> This server serves only smtp traffic. The only thing postfix should do - receive mails for one user and send it to local script via pipe. Like this:
>
> user:    |/usr/local/bin/script
>
> The flow is about 250-300 mails per second.Everything is going fine, if there is no queue. But when queue grows to 10000+ letters (for some reasons), postfix loses control of it. deferred queue is null, active queue is almost null, but incoming queue is growing and growing. When incoming queue is about 100k+ letters, postfix just can't pick it up, until I stop incoming traffic. I'm wondering, why such a server can't handle this workload? Even if I do the following:

Because you load the DISK for 100% with incoming mail.

        Wietse

[Victor Duchovni]

On Fri, Nov 06, 2009 at 02:19:34AM +0300, devel anaconda wrote:

> user:    |/usr/local/bin/script
>
> The flow is about 250-300 mails per second.

Stop right there. At 200 msgs/sec, fork/exec of scripts via local(8)
is never going to give you the right performance. You need (in addition
to the faster disks already mentioned), to use a resident (pre-forked)
SMTP or LMTP server script to receive the mail in question.

    - Do not use local(8)
    - Do not start a new interpreter process for each message

--
        Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:majordomo@...?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.

[Stan Hoeppner]

devel anaconda put forth on 11/5/2009 5:19 PM:
> Hello everybody!
>
> I have a high-performance server (dual quad-core Xeon 2.8Ghz + 16GB RAM + 2SCSI disks 140Gb), RedHat Enterprise Linux 5.4, software RAID1 + Postfix 2.5.9.
> This server serves only smtp traffic. The only thing postfix should do - receive mails for one user and send it to local script via pipe. Like this:

How many free (empty) SCSI disk bays do you have in this server?  Is it
a 1U server chassis?  Post the make/model of the server please, or
preferably a link to a photo of the server on the manufacturer's
website.  The answer to this may or may not limit your options in
increasing your disk throughput, and thus solving your problem.

If it's a 1U server, I can tell you already that you are screwed, and
that you need a bigger chassis with more SCSI/SAS disk bays, at least 8
disks given your load of 250-300 msgs/sec.

--
Stan

[devel anaconda]

05.11.09, 17:58, "Kenneth Marshall" <ktm@...>:

> On Fri, Nov 06, 2009 at 02:19:34AM +0300, devel anaconda wrote:
> > Hello everybody!
> >
> > I have a high-performance server (dual quad-core Xeon 2.8Ghz + 16GB RAM + 2SCSI disks 140Gb), RedHat Enterprise Linux 5.4, software RAID1 + Postfix 2.5.9.
> > This server serves only smtp traffic. The only thing postfix should do - receive mails for one user and send it to local script via pipe. Like this:
> >
> > user:    |/usr/local/bin/script
> >
> > The flow is about 250-300 mails per second.Everything is going fine, if there is no queue. But when queue grows to 10000+ letters (for some reasons), postfix loses control of it. deferred queue is null, active queue is almost null, but incoming queue is growing and growing. When incoming queue is about 100k+ letters, postfix just can't pick it up, until I stop incoming traffic. I'm wondering, why such a server can't handle this workload? Even if I do the following:
> >
> >  user:   /dev/null
> >
> Your system is not a high-performance server I/O-wise. Your two disks can only
> handle 200-300 fsync's to disk per second and postfix will always sync your
> mail to disk before passing it on for local processing. You will need a
> battery backed caching RAID controller or fast SSD drive for the
> /var/spool/postfix directories to allow you to go faster. You could maybe
> move the spool directory to a RAM or tmpfs file system as well. You lose
> the safety net for messages if you have a power or other hardware problem.

Firs of, thank you Kenneth, Wietse, Victor and Corey for quick answer.

So I must confess, all looks like the bottle neck for now is a IO performance of my system. Unfortunately, I cannot move all the postfix's spool to tmpfs or /dev/shmem because of large size of queue (sometimes about a 80-100Gb), and I cannot move only incoming queue to RAM, because of postfix's specific file moves (I need to patch postfix, so it can move messages in split-partitions-queue). Please, correct me, if I'm wrong, but if the current bottle neck is limited fsync, so I can do the following (at my own risk):


--- postfix-2.5.9/src/global/mail_stream.c.orig 2009-11-05 23:27:23.000000000 +0300
+++ postfix-2.5.9/src/global/mail_stream.c      2009-11-05 23:27:34.000000000 +0300
@@ -292,9 +292,9 @@
        || (want_stamp && stamp_path(VSTREAM_PATH(info->stream), want_stamp))
 #endif
        || fchmod(vstream_fileno(info->stream), 0700 | info->mode)
-#ifdef HAS_FSYNC
+/* #ifdef HAS_FSYNC
        || fsync(vstream_fileno(info->stream))
-#endif
+#endif */
        || (check_incoming_fs_clock
            && fstat(vstream_fileno(info->stream), &st) < 0)
        )

?

It disables fsync() on each incoming mail. Plus, if I mount my ext3 partition with option commit=30 or even commit=100, can it helps a bit?



> It probably never needed more than 5-8 local processes to sink only 200-300
> messages per second. That is why you do not see more. Add a "sleep 1" to
> your script and see what happens then. :)

Yes, it did the trick :)

[devel anaconda]

06.11.09, 00:02, "Stan Hoeppner" <stan@...>:

> devel anaconda put forth on 11/5/2009 5:19 PM:
> > Hello everybody!
> >
> > I have a high-performance server (dual quad-core Xeon 2.8Ghz + 16GB RAM + 2SCSI disks 140Gb), RedHat Enterprise Linux 5.4, software RAID1 + Postfix 2.5.9.
> > This server serves only smtp traffic. The only thing postfix should do - receive mails for one user and send it to local script via pipe. Like this:
> How many free (empty) SCSI disk bays do you have in this server?  Is it
> a 1U server chassis?  Post the make/model of the server please, or
> preferably a link to a photo of the server on the manufacturer's
> website.  The answer to this may or may not limit your options in
> increasing your disk throughput, and thus solving your problem.
> If it's a 1U server, I can tell you already that you are screwed, and
> that you need a bigger chassis with more SCSI/SAS disk bays, at least 8
> disks given your load of 250-300 msgs/sec.

This is an 1U PowerEdge server with 4 slots for SCSI disks

[Victor Duchovni]

On Fri, Nov 06, 2009 at 12:02:45AM -0600, Stan Hoeppner wrote:

> If it's a 1U server, I can tell you already that you are screwed, and
> that you need a bigger chassis with more SCSI/SAS disk bays, at least 8
> disks given your load of 250-300 msgs/sec.

Does this application really need Postfix and a queue? Why not just
turn the script into an SMTP server that pre-forks a fixed number of
copies and loops receiving/processing email? For 200-300 msgs/sec on
an I/O constrained server with a single "mailbox", one really does not
need Postfix, and can't afford the I/O cost of a local queue.

--
        Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:majordomo@...?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.

[devel anaconda]

06.11.09, 01:58, "Victor Duchovni" <Victor.Duchovni@...>:

> On Fri, Nov 06, 2009 at 12:02:45AM -0600, Stan Hoeppner wrote:
> > If it's a 1U server, I can tell you already that you are screwed, and
> > that you need a bigger chassis with more SCSI/SAS disk bays, at least 8
> > disks given your load of 250-300 msgs/sec.
> Does this application really need Postfix and a queue? Why not just
> turn the script into an SMTP server that pre-forks a fixed number of
> copies and loops receiving/processing email? For 200-300 msgs/sec on
> an I/O constrained server with a single "mailbox", one really does not
> need Postfix, and can't afford the I/O cost of a local queue.

Actually, I didn't tell the complete story.
The point is:

MTA (postfix) recieves the message. Then I need to put this message to 3 scripts (I must notify three different services about incoming mail). Now it looks like this:

mail comes to user@localhost on my Postfix, on aliases file I have:

usrer:  user1,user2,user3

user1:  |/usr/local/bin/script1
user2:  |/usr/local/bin/script2
user3:  |/usr/local/bin/script3

probably it'll be more scripts soon. When script1 fails with exit 75, but script2 and script3 have success, I must return mail for script1 to queue and try again later, and remove from queue messages for script2 and script3. That's why I use postfix and that's why I can't use pre-queued filter. I thought about different scheme, but postfix is the "easy way", so I'm trying speedup things a bit :)

[Victor Duchovni]

On Fri, Nov 06, 2009 at 10:16:35AM +0300, devel anaconda wrote:

> Actually, I didn't tell the complete story.
> The point is:
>
> MTA (postfix) recieves the message. Then I need to put this message
> to 3 scripts (I must notify three different services about incoming
> mail). Now it looks like this:
>
> mail comes to user@localhost on my Postfix, on aliases file I have:
>
> user:  user1,user2,user3
>
> user1:  |/usr/local/bin/script1
> user2:  |/usr/local/bin/script2
> user3:  |/usr/local/bin/script3
>
> probably it'll be more scripts soon. When script1 fails with exit 75,
> but script2 and script3 have success, I must return mail for script1 to
> queue and try again later, and remove from queue messages for script2 and
> script3.

Well, the Postfix "local" delivery agent does not do that for you,
it will retry all the scripts again, because local alias expansion is
done on each delivery attempt. The queue file stores only the original
recipient "user", so that can either be marked completed or re-processed
each time.

You could attempt to create appropriate "owner-alias" entries to make
this work, but with complex alias trees, I can't remember how. Far
easier to make your alias expansion to 3 users happen in virtual(5)
(pre-queue) not aliases(5) (post-queue).

For the throughput you are expecting, and especially running 3 three
(and later more) scripts for each message, you are not getting the job
done on the hardware you propose without serious optimization and
a design that reduces (rather than increases) the number of scripts
going forward if at all possible.

> That's why I use postfix and that's why I can't use pre-queued
> filter. I thought about different scheme, but postfix is the "easy way",
so I'm trying speedup things a bit :)

You are not going to succeed without more spindles and likely more CPUs.
Just launching a "Perl" hello-world script takes >10ms on a fast machine,
do that 100 times a second and you've burned a CPU, and non-trivial
scripts are slower still.

--
        Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:majordomo@...?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.

[Stan Hoeppner]

devel anaconda put forth on 11/6/2009 12:18 AM:

> This is an 1U PowerEdge server with 4 slots for SCSI disks

Ouch, 4 disks is very limiting.  Given that the spool files are
temporary in nature, your best option short of purchasing a new server
or an external SCSI array enclosure, is to add two more disks as an
mdadm RAID 0 (striping only, no data protection), and move the postfix
spool to that volume.  That will give you a 2x increase in spool
throughput, which might not be enough additional throughput for you.  If
one of the RAID 0 disks fails, you lose your spool.  Unlikely, but it
happens.

A better option would be to migrate to a new 2U or 3U server with more
disk bays (and disks), or get something like this, which is much less
than a new server:

http://cgi.ebay.com/Dell-PowerVault-220S-14x36GB-U320-SCSI-Array-Enclosure_W0QQitemZ380068671603QQcmdZViewItemQQptZLH_DefaultDomain_0?hash=item587ddbb073

Used external SCSI disk arrays are pretty cheap today, such as the one
above at Ebay.  It comes with a PCI RAID card, cables, and rails.  If
you choose not to use the RAID card but plain SCSI and mdadm instead,
you can just cable to the external SCSI connector on the back of your
current server (which you said is SCSI, not SAS, so it should have a
SCSI connector on back).

Anyway, with this particular array, configured by you as RAID 10, you'd
get 7 spindles worth of throughput (14/2 due to RAID 10) and roughly
250GB of space for your spool.  250GB may seem small given the cost and
availability of 500GB SATA disks, but you're after speed not size, and
the only way to get it is lots of disks.  The ad doesn't say if these
are 7.2, 10, or 15K RPM disks, but even if only 7.2K RPM, these seven
spindles would still give you around a 4x-5x throughput increase over
what you currently have, especially if you use the included PERC4/DC
RAID card.  If you get something like this, put the spool on it, and any
other disk intensive stuff like user mail dirs, assuming all this mail
traffic is destined to local users.

If you do upgrade some hardware to solve this issue, please let us know
what you do.

Hope this info helps you out.

--
Stan

**DISCLAIMER**:  I'm not plugging this particular Ebay item.  I have no
connection to it.  I'm just using this gear as an example as it is the
same brand as his server, meets the OP's needs and is relatively
inexpensive.

[Seth Mattinen]

devel anaconda wrote:
>
> It disables fsync() on each incoming mail. Plus, if I mount my ext3 partition with option commit=30 or even commit=100, can it helps a bit?
>


Have you tried a filesystem other than ext3 like Reiser or XFS? The
performance of ext3 really, really sucks if you have a lot of files in
one directory as a mail spool is bound to have.

~Seth

[Stan Hoeppner]

Seth Mattinen put forth on 11/7/2009 3:18 AM:
> devel anaconda wrote:
>> It disables fsync() on each incoming mail. Plus, if I mount my ext3 partition with option commit=30 or even commit=100, can it helps a bit?
>>
>
>
> Have you tried a filesystem other than ext3 like Reiser or XFS? The
> performance of ext3 really, really sucks if you have a lot of files in
> one directory as a mail spool is bound to have.

I'd stay away from Reiser, for multiple reasons, but XFS would
definitely be an improvement over ext3.  However, changing file systems
alone isn't going to solve his problem--it won't even double his
throughput, and he needs something like at least a 3x increase in disk
throughput at a minimum.

Unless he changes his filters so as to not write so darn many spool
files, he will absolutely require more spindles.  SSD is an option for
speed, but I doubt he'd want to spend the money required for 100+ GB of
SSD.  I'd say he needs at least 3 more SCSI drives in a Linux RAID 0
setup, or 6 more drives in a Linux RAID 10 setup, with XFS atop those.

IMHO, his next step should be adding a couple of 74GB 10K or 15K RPM
drives into his two empty disk bays, configure for RAID 0 with XFS atop
that, and test.  If that's still not enough, then he should go with
something like that cheap 14 drive external array I previously
mentioned, or SSD if he's flush with cash.

--
Stan

[Victor Duchovni]

On Sat, Nov 07, 2009 at 01:18:35AM -0800, Seth Mattinen wrote:

> Have you tried a filesystem other than ext3 like Reiser or XFS? The
> performance of ext3 really, really sucks if you have a lot of files in
> one directory as a mail spool is bound to have.

This is not so in most cases. Ext3 is just fine for Postfix.

--
        Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:majordomo@...?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.

[Seth Mattinen]

Victor Duchovni wrote:
> On Sat, Nov 07, 2009 at 01:18:35AM -0800, Seth Mattinen wrote:
>
>> Have you tried a filesystem other than ext3 like Reiser or XFS? The
>> performance of ext3 really, really sucks if you have a lot of files in
>> one directory as a mail spool is bound to have.
>
> This is not so in most cases. Ext3 is just fine for Postfix.
>

The only reason I mention it is because I had a problem with ext3
performance on my Postfix systems. The array controller is a 3ware
9550SX with 256M cache+BBU. Switching to XFS helped my case.

~Seth

[devel anaconda]

07.11.09, 13:41, "Stan Hoeppner" <stan@...>:

> > Have you tried a filesystem other than ext3 like Reiser or XFS? The
> > performance of ext3 really, really sucks if you have a lot of files in
> > one directory as a mail spool is bound to have.
> I'd stay away from Reiser, for multiple reasons, but XFS would
> definitely be an improvement over ext3.  However, changing file systems
> alone isn't going to solve his problem--it won't even double his
> throughput, and he needs something like at least a 3x increase in disk
> throughput at a minimum.
> Unless he changes his filters so as to not write so darn many spool
> files, he will absolutely require more spindles.  SSD is an option for
> speed, but I doubt he'd want to spend the money required for 100+ GB of
> SSD.  I'd say he needs at least 3 more SCSI drives in a Linux RAID 0
> setup, or 6 more drives in a Linux RAID 10 setup, with XFS atop those.
> IMHO, his next step should be adding a couple of 74GB 10K or 15K RPM
> drives into his two empty disk bays, configure for RAID 0 with XFS atop
> that, and test.  If that's still not enough, then he should go with
> something like that cheap 14 drive external array I previously
> mentioned, or SSD if he's flush with cash.

The problem is not in filters. It's not the bottleneck for now. The bottleneck for now is moves messages from incoming to active queue. I'll try to use two other HDD's for RAID0. Thank you.

[devel anaconda]

06.11.09, 02:32, "Victor Duchovni" <Victor.Duchovni@...>:

> > That's why I use postfix and that's why I can't use pre-queued
> > filter. I thought about different scheme, but postfix is the "easy way",
> > so I'm trying speedup things a bit :)

> You are not going to succeed without more spindles and likely more CPUs.
> Just launching a "Perl" hello-world script takes >10ms on a fast machine,
> do that 100 times a second and you've burned a CPU, and non-trivial
> scripts are slower still.

As I said above - scripts is not the point at a time. If it will, I'll rewrite them, so my scripts will listen unix socket and I'll put messages directly into sockets.

I think, that things will go faster, if postfix, instead of saving new messages to queue, will try to deliver it to deliver agent "in memory", and only if that fails, save the message to queue. Don't you think, this will be faster? Is that possible? I can make a patch (not for upstream, only for me).

[devel anaconda]

08.11.09, 12:28, "James MacLachlan" <james@...>:

> Is your queue disk mounted aysnc?  If so you are in ram for the most
> part anyway,

Well, actually - not.
Yes, it's mounted with async. I straced the cleanup daemon. Two things slows down HDD's: write() and fsync() on each message. So, I disabled fsync() and didn't get any appreciable benefits. So, the write() is the thing.

If moving all queues to RAID0 won't help, I'll change postfix a bit for moving only "incoming" queue to RAM disk. "Vanilla" postfix does not allow that.
If that won't help, I'll try to move all queues to RAM and see.
Thanks.