Crashes running SA as milter in Postfix

[Patrick Ben Koetter]

We regularly experience SA crashes on a Ubuntu Hardy machine. The setup is as
follows:

  Postfix (2.5.1) -> SpamAssassin Milter (0.3.1-6) -> SpamAssassin (3.2.4-1ubuntu1.1)

The milter is run like this:

/usr/sbin/spamass-milter -P /var/run/spamass/spamass.pid -f \
  -p /var/spool/postfix/spamass/spamass.sock \
  -u spamass-milter -i 127.0.0.1 -r 10

SpamAssassin is run like this:

/usr/bin/perl -T -w /usr/sbin/spamd -s local5 -u spamassassin \
  --nouser-config --max-children 10 --debug=spamd -d \
  --pidfile=/var/run/spamd.pid

It crashed again this weekend. This is what I found in the log:

Oct 29 08:01:51 mail01 spamd[10249]: spamd: fork: Cannot allocate memory at /usr/sbin/spamd line 999.
Oct 29 08:01:53 mail01 spamd[301]: auto-whitelist: open of auto-whitelist file failed: locker: safe_lock: cannot create tmp lockfile /nonexistent/.spamassassin/auto-whitelist.lo
ck.mail01.example.com.301 for /nonexistent/.spamassassin/auto-whitelist.lock: No such file or directory
Oct 29 08:01:53 mail01 spamd[301]: spamd: clean message (1.1/5.0) for singer-paf:65534 in 2.3 seconds, 28868 bytes.
Oct 29 08:01:53 mail01 spamd[301]: spamd: result: . 1 - EXTRA_MPART_TYPE,HTML_MESSAGE,RDNS_NONE scantime=2.3,size=28868,user=singer-paf,uid=65534,required_score=5.0,rhost=localhost.localdomain,raddr=127.0.0.1,rport=42576,mid=<542376CEA87A0943B958AFD2BF4636CD166E24@...>,autolearn=no
Oct 29 08:01:53 mail01 spamd[301]: syswrite() to parent failed: Broken pipe at /usr/share/perl5/Mail/SpamAssassin/SpamdForkScaling.pm line 576.

Something that annoys me, is that it keeps complaining "cannot create tmp
lockfile /nonexistent/.spamassassin/", while I keep it running as user
"spamassassin". I am purley speculating: Could this be in relation to my crash
problem?

Thanks,

p@rick

--
state of mind
Digitale Kommunikation

http://www.state-of-mind.de

Franziskanerstraße 15      Telefon [phone]
81669 München              Telefax [phone]

Amtsgericht München        Partnerschaftsregister PR 563

[d.hill]

Quoting Patrick Ben Koetter <p@...>:

> We regularly experience SA crashes on a Ubuntu Hardy machine. The setup is as
> follows:
>
>   Postfix (2.5.1) -> SpamAssassin Milter (0.3.1-6) -> SpamAssassin  
> (3.2.4-1ubuntu1.1)
>
> The milter is run like this:
>
> /usr/sbin/spamass-milter -P /var/run/spamass/spamass.pid -f \
>   -p /var/spool/postfix/spamass/spamass.sock \
>   -u spamass-milter -i 127.0.0.1 -r 10
>
> SpamAssassin is run like this:
>
> /usr/bin/perl -T -w /usr/sbin/spamd -s local5 -u spamassassin \
>   --nouser-config --max-children 10 --debug=spamd -d \
>   --pidfile=/var/run/spamd.pid
>
> It crashed again this weekend. This is what I found in the log:
>
> Oct 29 08:01:51 mail01 spamd[10249]: spamd: fork: Cannot allocate  
> memory at /usr/sbin/spamd line 999.
> Oct 29 08:01:53 mail01 spamd[301]: auto-whitelist: open of  
> auto-whitelist file failed: locker: safe_lock: cannot create tmp  
> lockfile /nonexistent/.spamassassin/auto-whitelist.lo
> ck.mail01.example.com.301 for  
> /nonexistent/.spamassassin/auto-whitelist.lock: No such file or  
> directory
> Oct 29 08:01:53 mail01 spamd[301]: spamd: clean message (1.1/5.0)  
> for singer-paf:65534 in 2.3 seconds, 28868 bytes.
> Oct 29 08:01:53 mail01 spamd[301]: spamd: result: . 1 -  
> EXTRA_MPART_TYPE,HTML_MESSAGE,RDNS_NONE  
> scantime=2.3,size=28868,user=singer-paf,uid=65534,required_score=5.0,rhost=localhost.localdomain,raddr=127.0.0.1,rport=42576,mid=<542376CEA87A0943B958AFD2BF4636CD166E24@...>,autolearn=no
> Oct 29 08:01:53 mail01 spamd[301]: syswrite() to parent failed:  
> Broken pipe at  
> /usr/share/perl5/Mail/SpamAssassin/SpamdForkScaling.pm line 576.
>
> Something that annoys me, is that it keeps complaining "cannot create tmp
> lockfile /nonexistent/.spamassassin/", while I keep it running as user
> "spamassassin". I am purley speculating: Could this be in relation  
> to my crash
> problem?

The home directory for the username spamassassin is probably set to  
/nonexistant in the passwd file (or whatever it is in Ubuntu).

[Patrick Ben Koetter]

* d.hill@... <d.hill@...>:
> The home directory for the username spamassassin is probably set to
> /nonexistant in the passwd file (or whatever it is in Ubuntu).

Thanks for the reply. I wish, it was that easy, but it is not. The $HOME is
/home/spamassassin.

p@rick

--
state of mind
Digitale Kommunikation

http://www.state-of-mind.de

Franziskanerstraße 15      Telefon [phone]
81669 München              Telefax [phone]

Amtsgericht München        Partnerschaftsregister PR 563

[d.hill]

Quoting Patrick Ben Koetter <p@...>:

> * d.hill@... <d.hill@...>:
>> The home directory for the username spamassassin is probably set to
>> /nonexistant in the passwd file (or whatever it is in Ubuntu).
>
> Thanks for the reply. I wish, it was that easy, but it is not. The $HOME is
> /home/spamassassin.

That's the only place I could think where the /nonexistant path would  
be coming from.

[Mark Martinec]

Patrick,

> We regularly experience SA crashes on a Ubuntu Hardy machine. The setup is
> as follows:
>  
> Postfix (2.5.1) -> SpamAssassin Milter (0.3.1-6) -> SpamAssassin
> (3.2.4-1ubuntu1.1)
>  
> The milter is run like this:
>  
> /usr/sbin/spamass-milter -P /var/run/spamass/spamass.pid -f \
>    -p /var/spool/postfix/spamass/spamass.sock \
>    -u spamass-milter -i 127.0.0.1 -r 10
>  
> /usr/bin/perl -T -w /usr/sbin/spamd -s local5 -u spamassassin \
>    --nouser-config --max-children 10 --debug=spamd -d \
>    --pidfile=/var/run/spamd.pid
>  
> It crashed again this weekend. This is what I found in the log:
>  
> Oct 29 08:01:51 mail01 spamd[10249]: spamd: fork: Cannot allocate memory
>   at /usr/sbin/spamd line 999.

/usr/include/errno.h
#define ENOMEM 12 /* Cannot allocate memory */

man 2 fork
  [ENOMEM] There is insufficient swap space for the new process.


Mark

On Sun, 2009-11-01 at 22:31 +0100, Patrick Ben Koetter wrote:

> We regularly experience SA crashes on a Ubuntu Hardy machine. The setup is as
> follows:
>
>   Postfix (2.5.1) -> SpamAssassin Milter (0.3.1-6) -> SpamAssassin (3.2.4-1ubuntu1.1)
>
> The milter is run like this:
>
> /usr/sbin/spamass-milter -P /var/run/spamass/spamass.pid -f \
>   -p /var/spool/postfix/spamass/spamass.sock \
>   -u spamass-milter -i 127.0.0.1 -r 10
>
> SpamAssassin is run like this:
>
> /usr/bin/perl -T -w /usr/sbin/spamd -s local5 -u spamassassin \
>   --nouser-config --max-children 10 --debug=spamd -d \
>   --pidfile=/var/run/spamd.pid
>
> It crashed again this weekend. This is what I found in the log:
>
> Oct 29 08:01:51 mail01 spamd[10249]: spamd: fork: Cannot allocate memory at /usr/sbin/spamd line 999.
> Oct 29 08:01:53 mail01 spamd[301]: auto-whitelist: open of auto-whitelist file failed: locker: safe_lock: cannot create tmp lockfile /nonexistent/.spamassassin/auto-whitelist.lo
> ck.mail01.example.com.301 for /nonexistent/.spamassassin/auto-whitelist.lock: No such file or directory
> Oct 29 08:01:53 mail01 spamd[301]: spamd: clean message (1.1/5.0) for singer-paf:65534 in 2.3 seconds, 28868 bytes.
> Oct 29 08:01:53 mail01 spamd[301]: spamd: result: . 1 - EXTRA_MPART_TYPE,HTML_MESSAGE,RDNS_NONE scantime=2.3,size=28868,user=singer-paf,uid=65534,required_score=5.0,rhost=localhost.localdomain,raddr=127.0.0.1,rport=42576,mid=<542376CEA87A0943B958AFD2BF4636CD166E24@...>,autolearn=no
> Oct 29 08:01:53 mail01 spamd[301]: syswrite() to parent failed: Broken pipe at /usr/share/perl5/Mail/SpamAssassin/SpamdForkScaling.pm line 576.
>
> Something that annoys me, is that it keeps complaining "cannot create tmp
> lockfile /nonexistent/.spamassassin/", while I keep it running as user
> "spamassassin". I am purley speculating: Could this be in relation to my crash
> problem?
>
> Thanks,
>
> p@rick
>

Just to be clear - this is a proper hardware server, and not a VPS of
some kind?

[Matus UHLAR - fantomas]

> * d.hill@... <d.hill@...>:
> > The home directory for the username spamassassin is probably set to
> > /nonexistant in the passwd file (or whatever it is in Ubuntu).

On 01.11.09 23:23, Patrick Ben Koetter wrote:
> Thanks for the reply. I wish, it was that easy, but it is not. The $HOME is
> /home/spamassassin.

you have passed the "-u spamass-milter" option. That means
that SA-milter will pass username of the recipient to spamd if there's
one recipient, so the recipient's homedir will be used. However, if there
are more recipients, SA-milter will pass the provided username
(spamass-milter) to the spamd, so it's apparently the spamass-milter user
whose directory is /nonexistent.

However, that should not be a reason why your SA crashes.

--
Matus UHLAR - fantomas, uhlar@... ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
If Barbie is so popular, why do you have to buy her friends?

[Patrick Ben Koetter]

Matus,

* Matus UHLAR - fantomas <uhlar@...>:

> > * d.hill@... <d.hill@...>:
> > > The home directory for the username spamassassin is probably set to
> > > /nonexistant in the passwd file (or whatever it is in Ubuntu).
>
> On 01.11.09 23:23, Patrick Ben Koetter wrote:
> > Thanks for the reply. I wish, it was that easy, but it is not. The $HOME is
> > /home/spamassassin.
>
> you have passed the "-u spamass-milter" option. That means
> that SA-milter will pass username of the recipient to spamd if there's
> one recipient, so the recipient's homedir will be used. However, if there

That explains it! I never bothered to question this setting, as I thought it
would instruct the milter to run as user spamass-milter, which seemed kind of
wishful to me.


> are more recipients, SA-milter will pass the provided username
> (spamass-milter) to the spamd, so it's apparently the spamass-milter user
> whose directory is /nonexistent.

I will track this a little to see that it really fixes this particular
problem.

> However, that should not be a reason why your SA crashes.

Agreed. Any ideas how I could trap this better? I run spamd in debug mode
debugging "spamd". Running debug with option "all" seemed a little too heavy
to me. Is there best practice to trace the crashes?

p@rick

[Greg Troxel]

I have had problems in postfix with spamass-milter and milter-greylist,
on NetBSD/sparc64 5-stable.  I think I saw a SIGBUS in the log, which
would indicate an unaligned access.  This recently started - it used to
be ok.
Not sure if this is relevant to your issue....

[Patrick Ben Koetter]

Mark,

* Mark Martinec <Mark.Martinec+sa@...>:

> > We regularly experience SA crashes on a Ubuntu Hardy machine. The setup is
> > as follows:
> >  
> > Postfix (2.5.1) -> SpamAssassin Milter (0.3.1-6) -> SpamAssassin
> > (3.2.4-1ubuntu1.1)
> >  
> > The milter is run like this:
> >  
> > /usr/sbin/spamass-milter -P /var/run/spamass/spamass.pid -f \
> >    -p /var/spool/postfix/spamass/spamass.sock \
> >    -u spamass-milter -i 127.0.0.1 -r 10
> >  
> > /usr/bin/perl -T -w /usr/sbin/spamd -s local5 -u spamassassin \
> >    --nouser-config --max-children 10 --debug=spamd -d \
> >    --pidfile=/var/run/spamd.pid
> >  
> > It crashed again this weekend. This is what I found in the log:
> >  
> > Oct 29 08:01:51 mail01 spamd[10249]: spamd: fork: Cannot allocate memory
> >   at /usr/sbin/spamd line 999.
>
> /usr/include/errno.h
> #define ENOMEM 12 /* Cannot allocate memory */
>
> man 2 fork
>   [ENOMEM] There is insufficient swap space for the new process.

it seems your diagnosis hit the spot. The filter seems to run stable now.

thanks,

p@rick

--
state of mind
Digitale Kommunikation

http://www.state-of-mind.de

Franziskanerstraße 15      Telefon [phone]
81669 München              Telefax [phone]

Amtsgericht München        Partnerschaftsregister PR 563