Nabble - SpamAssassin - Dev

[Bug 6176] [review] spamc truncates lines read from spamc.conf

2009-12-21T18:31:55Z

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6176

Mark Martinec <Mark.Martinec@...> changed:

What |Removed |Added
----------------------------------------------------------------------------
Priority|P2 |P1

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6176] [review] spamc truncates lines read from spamc.conf

2009-12-21T18:29:01Z

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6176

Mark Martinec <Mark.Martinec@...> changed:

What |Removed |Added
----------------------------------------------------------------------------
Summary|REGRESSION spamc truncates |[review] spamc truncates
|lines read from spamc.conf |lines read from spamc.conf
Status Whiteboard| |needs 2 votes

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6176] REGRESSION spamc truncates lines read from spamc.conf

2009-12-21T18:27:27Z

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6176

Mark Martinec <Mark.Martinec@...> changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #4611|0 |1
is obsolete| |

--- Comment #17 from Mark Martinec <Mark.Martinec@...> 2009-12-21 18:27:21 UTC ---
Created an attachment (id=4613)
--> (https://issues.apache.org/SpamAssassin/attachment.cgi?id=4613)
proposed patch, take 2

Apparently nobody is brave enough to touch the spamc.conf, so I'm
reluctantly doing it. Also fixed a buffer overflow when there
are more than 24 arguments present (command line + from a config file),
and fixed one spelling.

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6247] Replace HABEAS, BSP, SSC with RP Whitelist rules

2009-12-21T07:40:54Z

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6247

John Hardin <jhardin@...> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |jhardin@...

--- Comment #56 from John Hardin <jhardin@...> 2009-12-21 07:40:50 UTC ---
(In reply to comment #55)
>
> when looked at this way, they look more practical. a reliable way to cut the
> FP rate by 17% is definitely useful, although obviously it'd be nice if the
> 6.7% additional FNs was reduced.

Perhaps the suggested metas with the DNSBLs might help with that.

http://ruleqa.spamassassin.org/?rule=%2F[N_]_RP_

Unfortunately the last network masscheck seems to have only used one corpus so
the results aren't interesting yet; they certainly don't track the earlier
quick analysis I did by pulling the overlaps from the masscheck results (before
I wrote the rules).

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6247] Replace HABEAS, BSP, SSC with RP Whitelist rules

2009-12-21T05:20:00Z

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6247

--- Comment #55 from Justin Mason <jm@...> 2009-12-21 05:19:51 UTC ---
(In reply to comment #54)

> > rescore masscheck logs with mcsnapshot + today's trunk Scores
> > HABEAS, BSP and SSC and DNSWL Disabled
> > ==========================
> > # SUMMARY for threshold 5.0:
> > # Correctly non-spam: 703899 99.93%
> > # Correctly spam: 2565063 98.49%
> > # False positives: 467 0.07%
> > # False negatives: 39257 1.51%
> > # TCR(l=50): 41.597904 SpamRecall: 98.493% SpamPrec: 99.982%
>
> So that means that
>
> 467-387
> --> 80
> 80/(703899+467)
> --> 0.000113577316338381
>
> => 0.01135% of hams were rescued from being FPs by the DNSWL rules.
>
> 41888-39257
> --> 2631
> 2631/(2565063+39257)
> --> 0.00101024451680285
>
> => 0.101% of spams were, conversely, allowed through by them.

actually, I've realised this may be measuring the wrong thing -- since FP% is
much lower (by design) than FN%, it's quite likely that we'll see a
comparatively high rate of hits against spam, even if the rules are useful. A
better way to think about it is in terms of those values compared against the
FP%/FN% rates.

=> 80 hams rescued from the 467 FPs = 17.13% of FPs rescued

=> 2631 spams added to the 39257 FNs = 6.70% additional FNs

when looked at this way, they look more practical. a reliable way to cut the
FP rate by 17% is definitely useful, although obviously it'd be nice if the
6.7% additional FNs was reduced.

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

RE: Perl SA util: setuid: oops: fileno(STDIN) [6]

2009-12-21T03:59:25Z

In addition to this, the line “Message:MonDec212147542009-1037461244 SPAM Results - Score 2.729 Tests - AWL=-2.064,EMPTY_MESSAGE=0.607,HTML_MESSAGE=0.001,MIME_HTML_MOSTLY=0.001,MISSING_SUBJECT=1.285,TVD_SPACE_RATIO=2.899” Is the result from my application

And at the top of my application I have

use Mail::SpamAssassin;

#Create and compile a SpamAssassin object to be used by all threads

my $MainSA = new Mail::SpamAssassin;

$MainSA->compile_now();

From: Cory Hawkless [mailto:cory@...]
Sent: Monday, 21 December 2009 9:54 PM
To: dev@...
Subject: Perl SA util: setuid: oops: fileno(STDIN) [6]

Hi All,

I am writing an app in perl that gets mail passed from Postfix then runs the mail through SA and will run various actions based on the score(Very much like Amavisd-new)

I’m struggling a bit my SpamAssassin code though and was wondering if anyone here could help. I’m getting the following error when using my SpamAssassinCheck subroutine

util: setuid: oops: fileno(STDIN) [6] != 0 at /usr/lib/perl5/vendor_perl/5.10.0/Mail/SpamAssassin/Util.pm line 1375.

Message:MonDec212147542009-1037461244 SPAM Results - Score 2.729 Tests - AWL=-2.064,EMPTY_MESSAGE=0.607,HTML_MESSAGE=0.001,MIME_HTML_MOSTLY=0.001,MISSING_SUBJECT=1.285,TVD_SPACE_RATIO=2.899

As you can see i’m still getting a result, if I let me app continue to process a few more emails I eventually get a “Alarm Clock” error and perl bombs out.

Here is my SpamAssassinCheck code(I also don’t think i’m using the eval correctly)

P.S Let me know if this is not the appropriate place to be posting this message, but I couldn’t think of a better group of people to help solve my problem.

Regards,

Cory

sub SpamAssassinScan {

eval {

#Record the time when this program was started

#my $starttime = [gettimeofday];

#User to run SPAM Assasin as

my $SAUser;

$SAUser=$_[0];

#Subroutine expects the second paramater to be the content of the email to be scanned

my $rawemail;

$rawemail=$_[1];

my $messageID = $_[2];

my $ThisThreadSA=$MainSA;

##print "Beginning test\n"

if ($SAUser eq "") { $SAUser="root" }

logit("info", "Message:$messageID Running SA test as: $SAUser");

$ThisThreadSA->{username}=$SAUser;

my $mail = $ThisThreadSA->parse($rawemail);

my $status = $ThisThreadSA->check($mail);

##print "SA scanning done\n";

logit("info","Message:$messageID SPAM Results - Score ".$status->get_score()." Tests - ".$status->get_tag('TESTSSCORES',','));

};

if ($@){

### Catch block

logit("info", "Failed SA Scan");;

};

}

Perl SA util: setuid: oops: fileno(STDIN) [6]

2009-12-21T03:24:21Z

Hi All,

I am writing an app in perl that gets mail passed from Postfix then runs the mail through SA and will run various actions based on the score(Very much like Amavisd-new)

I’m struggling a bit my SpamAssassin code though and was wondering if anyone here could help. I’m getting the following error when using my SpamAssassinCheck subroutine

util: setuid: oops: fileno(STDIN) [6] != 0 at /usr/lib/perl5/vendor_perl/5.10.0/Mail/SpamAssassin/Util.pm line 1375.

Message:MonDec212147542009-1037461244 SPAM Results - Score 2.729 Tests - AWL=-2.064,EMPTY_MESSAGE=0.607,HTML_MESSAGE=0.001,MIME_HTML_MOSTLY=0.001,MISSING_SUBJECT=1.285,TVD_SPACE_RATIO=2.899

As you can see i’m still getting a result, if I let me app continue to process a few more emails I eventually get a “Alarm Clock” error and perl bombs out.

Here is my SpamAssassinCheck code(I also don’t think i’m using the eval correctly)

P.S Let me know if this is not the appropriate place to be posting this message, but I couldn’t think of a better group of people to help solve my problem.

Regards,

Cory

sub SpamAssassinScan {

eval {

#Record the time when this program was started

#my $starttime = [gettimeofday];

#User to run SPAM Assasin as

my $SAUser;

$SAUser=$_[0];

#Subroutine expects the second paramater to be the content of the email to be scanned

my $rawemail;

$rawemail=$_[1];

my $messageID = $_[2];

my $ThisThreadSA=$MainSA;

##print "Beginning test\n"

if ($SAUser eq "") { $SAUser="root" }

logit("info", "Message:$messageID Running SA test as: $SAUser");

$ThisThreadSA->{username}=$SAUser;

my $mail = $ThisThreadSA->parse($rawemail);

my $status = $ThisThreadSA->check($mail);

##print "SA scanning done\n";

logit("info","Message:$messageID SPAM Results - Score ".$status->get_score()." Tests - ".$status->get_tag('TESTSSCORES',','));

};

if ($@){

### Catch block

logit("info", "Failed SA Scan");;

};

}

[Bug 6176] REGRESSION spamc truncates lines read from spamc.conf

2009-12-20T14:59:16Z

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6176

--- Comment #16 from Warren Togami <wtogami@...> 2009-12-20 14:59:11 UTC ---
I backed this out just in case folks are dogfooding the trunk.

I am traveling most of Dec 21st. I suppose rc1.proposed2 can be cut after you
folks decide what to do about this bug. I will be able to cut it on Tuesday,
unless jm or somebody wants to do the cut on Monday.

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6176] REGRESSION spamc truncates lines read from spamc.conf

2009-12-19T22:47:12Z

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6176

--- Comment #15 from Warren Togami <wtogami@...> 2009-12-19 22:47:08 UTC ---
NOTE: Output of nothing with zero return value means blackholing of mail. Very
dangerous. This should be a very simple test-case possible with "make test".
A working spamd is not necessary to test this.

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6176] REGRESSION spamc truncates lines read from spamc.conf

2009-12-19T21:56:55Z

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6176

Warren Togami <wtogami@...> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |REOPENED
Resolution|FIXED |
Summary|[review] spamc truncates |REGRESSION spamc truncates
|lines read from spamc.conf |lines read from spamc.conf

--- Comment #14 from Warren Togami <wtogami@...> 2009-12-19 21:56:52 UTC ---
[warren@computer trunk]$ cat sample-nonspam.txt |spamc
[warren@computer trunk]$ echo $?
0

spamd -D sees no connection attempt from spamc. Same result if spamd is
running or not.

http://svn.apache.org/viewvc?view=revision&revision=891935
I backed out this patch and spamc works again.

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

Re: ABORT Re: PROPOSED: Apache SpamAssassin 3.3.0-rc1.proposed1

2009-12-19T20:25:44Z

On 12/19/2009 10:36 PM, Warren Togami wrote:
>
> I am aborting the release of rc1.proposed1. There is some problem
> causing spamc/spamd to fail completely. I am attempting to figure this
> out a possible cause. Downgrading to beta1 seems to fix my server.
>
> Warren

[warren@newcaprica trunk]$ cat sample-nonspam.txt |spamc
[warren@newcaprica trunk]$ echo $?
0

spamd -D sees no connection attempt from spamc. It seems that spamc is
broken? I am attempting to backout parts between beta1 and rc1 to
narrow this down.

Reproduced on ...
perl-5.8.8-27.el5.x86_64
perl-5.10.0-82.fc12.x86_64

Since spamc was returning with a zero return value and blank, all of the
legitimate mail goes into a blackhole. =(

Warren

ABORT Re: PROPOSED: Apache SpamAssassin 3.3.0-rc1.proposed1

2009-12-19T19:36:38Z

On 12/18/2009 08:57 PM, Warren Togami wrote:
> This will be released if we go three days without an objection as per build/README procedure. At that point these archives will be renamed to "rc1" and the announcements will go out. Please suggest improvements to this announcement text as well.
>
> Hey users list, now would be a very good time to begin testing 3.3.0 if you haven't already. At this point it has been tested in production on many production servers (including my own production server since March 2009), but it is possible we missed a corner case of some non-standard configuration that you folks rely upon. We could use your feedback, even if it is only "It works!" Now is last chance to complain if you find a problem.
>
> All of the Priority P1 blocker bugs targeted for 3.3.0 are now closed. I suspect there might be a few minor things we might want to polish before 3.3.0 final, but otherwise this is VERY CLOSE to what 3.3.0 will be.
>
> Warren Togami
> wtogami@...
>

I am aborting the release of rc1.proposed1. There is some problem
causing spamc/spamd to fail completely. I am attempting to figure this
out a possible cause. Downgrading to beta1 seems to fix my server.

Warren

Re: PROPOSED: Apache SpamAssassin 3.3.0-rc1.proposed1

2009-12-19T11:06:32Z

On 19/12/2009 10:24 AM, Warren Togami wrote:

> On 12/18/2009 11:38 PM, Daryl C. W. O'Shea wrote:
>> Warren,
>>
>> I don't think that it's necessary to cross-post so much to the users@
>> list. In particular, I don't think proposed release announcements
>> belong on the users@ list.
>>
>> Thoughts?
>>
>> Daryl
>
> Respectfully, I believe we need more testers and participants, and our
> user list is our nearest opportunity to do recruitment.
>
> Warren

I agree we need more testers and participants. Is it common for other
projects to ask their users' lists to test things like betas and release
candidates before the project members have agreed to cut such a beta or
release candidate? I don't know. Maybe the new ".proposed1" scheme
makes it irrelevant.

Daryl

Re: PROPOSED: Apache SpamAssassin 3.3.0-rc1.proposed1

2009-12-19T07:24:39Z

On 12/18/2009 11:38 PM, Daryl C. W. O'Shea wrote:
> Warren,
>
> I don't think that it's necessary to cross-post so much to the users@
> list. In particular, I don't think proposed release announcements
> belong on the users@ list.
>
> Thoughts?
>
> Daryl

Respectfully, I believe we need more testers and participants, and our
user list is our nearest opportunity to do recruitment.

Warren

Re: pyzor error during spamd startup

2009-12-19T04:21:08Z

On Fri 18 Dec 2009 10:10:31 PM CET, "Kevin A. McGrail" wrote

> Why is pyzor running as root?

spamd start up as root ?, and dropspriv to late before calling pyzor ?

spamassassin -t test.msg creates for root also here a dir in /.spamassassin

--
xpoint

attachment0 (205 bytes) Download Attachment

Re: PROPOSED: Apache SpamAssassin 3.3.0-rc1.proposed1

2009-12-18T20:38:20Z

Warren,

I don't think that it's necessary to cross-post so much to the users@
list. In particular, I don't think proposed release announcements
belong on the users@ list.

Thoughts?

Daryl

On 18/12/2009 8:57 PM, Warren Togami wrote:

> This will be released if we go three days without an objection as per build/README procedure. At that point these archives will be renamed to "rc1" and the announcements will go out. Please suggest improvements to this announcement text as well.
>
> Hey users list, now would be a very good time to begin testing 3.3.0 if you haven't already. At this point it has been tested in production on many production servers (including my own production server since March 2009), but it is possible we missed a corner case of some non-standard configuration that you folks rely upon. We could use your feedback, even if it is only "It works!" Now is last chance to complain if you find a problem.
>
> All of the Priority P1 blocker bugs targeted for 3.3.0 are now closed. I suspect there might be a few minor things we might want to polish before 3.3.0 final, but otherwise this is VERY CLOSE to what 3.3.0 will be.
>
> Warren Togami
> wtogami@...
>
> [DRAFT DRAFT DRAFT - NOT YET RELEASED - DRAFT DRAFT DRAFT]
>
> Apache SpamAssassin 3.3.0-rc1 is now available for testing.
>
> Downloads are available from:
> http://people.apache.org/~wtogami/devel/
> md5sum of archive files:
>
> fabccde7f09dcdf4c06afbd4cc0687c2 Mail-SpamAssassin-3.3.0-rc1.proposed1.tar.bz2
> a36c646742c0cd5960e7be222cc91200 Mail-SpamAssassin-3.3.0-rc1.proposed1.tar.gz
> 8c7f92167f2c2016164aafeac925ac58 Mail-SpamAssassin-3.3.0-rc1.proposed1.zip
> 0bf1e2c0fc70dc05104acec8a723cbdb Mail-SpamAssassin-rules-3.3.0-rc1.r892385.tgz
>
> sha1sum of archive files:
>
> 7101ea1ab28b47e557583b914e242651f6cd761e Mail-SpamAssassin-3.3.0-rc1.proposed1.tar.bz2
> 344dd5887772abc0fc0aea11a30dff5d77d67db8 Mail-SpamAssassin-3.3.0-rc1.proposed1.tar.gz
> d9e852d2c64cf23e86b2ae0e7f3e6082a028c1a2 Mail-SpamAssassin-3.3.0-rc1.proposed1.zip
> ad71dbf4250dddd00d885311b14d39c0c03e2ae1 Mail-SpamAssassin-rules-3.3.0-rc1.r892385.tgz
>
>
> Note that the *-rules-*.tgz files are only necessary if you cannot, or do not
> wish to, run "sa-update" after install to download the latest fresh rules.
>
> The release files also have a .asc accompanying them. The file serves
> as an external GPG signature for the given release file. The signing
> key is available via the wwwkeys.pgp.net key server, as well as
> http://www.apache.org/dist/spamassassin/KEYS
>
> The key information is:
>
> pub 4096R/F7D39814 2009-12-02
> Key fingerprint = D809 9BC7 9E17 D7E4 9BC2 1E31 FDE5 2F40 F7D3 9814
> uid SpamAssassin Project Management Committee <private@...>
> uid SpamAssassin Signing Key (Code Signing Key, replacement for 1024D/265FA05B) <dev@...>
> sub 4096R/7B3265A5 2009-12-02
>
> See the INSTALL and UPGRADE files in the distribution for important
> installation notes.
>
>
> Summary of major changes since 3.2.5
> ------------------------------------
>
> COMPATIBILITY WITH 3.2.5
>
> - rules are no longer distributed with the package, but installed by
> sa-update - either automatically fetched from the network (preferably),
> or from a tar archive, which is available for downloading separately
>
> - CPAN module requirements:
> - minimum required version of ExtUtils::MakeMaker is 6.17
> - modules now required: Time::HiRes, NetAddr::IP, Archive::Tar
> - minimal version of Mail::DKIM is 0.31 (preferred: 0.37 or later);
> expect some tests in t/dkim2.t to fail with versions older than 0.36_5;
> - no longer used: Mail::DomainKeys, Mail::SPF::Query
> - if module Digest::SHA is not available, a module Digest::SHA1
> will be used, but at least one of them must be installed;
> a DKIM plugin requires Digest::SHA (the older Digest::SHA1 does not
> support sha256 hashes), so in practice the Digest::SHA is required
>
> - if keeping AWL database in SQL, the field awl.ip must be extended to
> 40 characters. The change is necessary to allow AWL to keep track of IPv6
> addresses which may appear in a mail header even on non-IPv6 -enabled host.
> While at it, consider also adding a field 'signedby' to the SQL table 'awl'
> (and adding 'auto_whitelist_distinguish_signed 1' to local.cf);
> See sql/README.awl for details. The change need not be undone even if
> downgrading back to 3.2.* for some reason;
>
> - fixing a protocol implementation error regarding a PING command required
> bumping up the SPAMC protocol version to 1.5. Spamd retains compatibility
> with older spamc clients. Combining new spamc clients with pre-3.3 versions
> of a spamd daemon is not supported (but happens to work, except for the
> PING and SKIP commands).
>
> - it may be worth mentioning that a rule DKIM_VERIFIED has been renamed
> to DKIM_VALID, to match its semantics;
>
> - due to a change in internal data structure (Bug 6185, 6254), third-party
> plugins which accesss the $pms->{main}->{conf}->{headers_spam} (and ham)
> need to be updated. One such example is the ClamAVPlugin plugin - please
> find a fresh version on its wiki page. It retains backwards compatibility,
> so can be used with both 3.2.5 as well as with SpamAssassin 3.3.0;
>
> - versions of amavisd-new between 2.5.2 and 2.6.1 (inclusive) are incompatible
> with SpamAssassin 3.3; please upgrade amavisd to 2.6.2 or later, or apply
> a workaround https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6257
>
> - support for versions of perl 5.6.* is being gradually revoked
> (may still work, but no promises and no support);
>
> - preferred versions of perl are 5.8.8, 5.8.9, and 5.10.1 or later
>
>
> MAIN NEW FEATURES
>
> - IPv6 support was substantially improved (see below);
>
> - many improvements to the DKIM plugin (understands author domain signatures,
> supports multiple signatures, ADSP support with overrides) - (see below);
>
> - added 'if can(Class::method)' conditional statement, allowing configuration
> settings to be conditionalised on plugin capabilities without requiring
> new version releases to do so;
>
> - added a configuration option 'time_limit', defaulting to 300 seconds
> or whatever the caller (like spamd) provides; attempting to gracefully
> terminate the checking when a time limit is reached, reporting the score
> and test hits that were collected so far, along with an added hit on
> a rule TIME_LIMIT_EXCEEDED;
>
> - more expensive code sections are now instrumented with timing measurements;
> timing report is logged as a debug message by the end of processing,
> and made available to a caller and to 'add_header' directives through
> a TIMING tag;
>
> - added a configuration option skip_uribl_checks to the URIDNSBL plugin,
> cross-document it with skip_rbl_checks;
>
> - preserve order of declared 'add_header' header fields;
>
> - configurable network mask length for the AWL plugin (see below);
>
> - added support for DCC reputations (see below);
>
> - improved error handling and robustness (see below);
>
> - added timestamps when logging on stderr;
>
> - allowed debug areas to be excluded from debugging,
> e.g.: -D all,norules,noconfig,nodcc
>
>
> BUILDING AND PACKAGING
>
> - rules are no longer distributed with the package, but installed by
> sa-update
>
> - Makefile.PL has been simplified and a bug fixed in a DESTDIR support
> by increasing the minimum required version of ExtUtils::MakeMaker to 6.17
>
> - tools check_whitelist and check_spamd are now included in the distribution,
> now called 'sa-awl' and 'sa-check_spamd'
>
>
> WORKAROUNDS TO PERL BUGS AND LIMITATIONS
>
> - modified the Check.pm plugin to produce smaller chunks of source code
> from rules (60 kB) to avoid Perl compiler crashing on exceeding stack size
>
> - localized global variables $1, $2, etc at several places, avoiding taint
> issue from propagating
>
> - avoided Perl I/O bug by replacing line-by-line reading with read() where
> suitable, or played down the EBADF status in other places and only report
> it as a dbg instead of a die - while also providing a little speedup
> (10 .. 25 %) on reading a message
>
> - provided a new sub Message::split_into_array_of_short_lines to split
> a text into array of paragraph chunks of sizes between 1 kB and 2 kB,
> giving less opportunity to runaway regular expressions in rules;
> fixes bugs: 5717, 5644, 5795, 5486, 5801, 5041
>
>
> MEMORY FOOTPRINT
>
> - as a side-effect of compiling rules in smaller chunks (to avoid compiler
> crashes), virtual memory footprint of SpamAssassin is reduced;
>
> - saved some memory by not importing the Pod::Usage unless it is needed;
>
> - saved 350k+ of memory in sa-compile by replacing DynaLoader with XSLoader;
>
> - removed unneeded index from MySQL bayes_token table;
>
>
> IPv6 SUPPORT
>
> - added IPv6 support for trusted_networks, internal_networks, msa_networks,
> whitelist_from_rcvd, and other stuff that uses NetSet and the Received
> header field parser, using NetAddr::IP;
>
> - allowed usage of a remote dccifd host through an INET or INET6 socket;
>
> - added IPv6 support to AWL plugin and its utility modules; a network
> mask length is now configurable and defaults to /48, which controls
> what data is stored in an AWL database;
>
> - sql/README.awl and sql/awl_*.sql: increased suggested awl.ip field width
> to 40 characters to be able to hold IPv6 addresses;
>
> - IP_PRIVATE now includes ipv6 variants of private address space,
> as well as the ipv6-mapped ipv4 addresses.
>
> - NetSet now understands that ::ffff:192.168.1.2 and 192.168.1.2 are
> the same address;
>
> - IPv6 addresses are now recognised in Received header fields;
>
> - when reading Received header fields, the "IPv6:" prefix is stripped from
> IPv6 addresses, and "::ffff:" is removed from IPv6-mapped IPv4 addresses
> (so strings can match them as simply IPv4 addresses);
>
> - ::1/128 is always included in the trusted_networks/internal_networks set
> similar to 127.0.0.0/8;
>
> - some of the IPv6 functionality in SpamAssassin requires that a perl module
> IO::Socket::INET6 is available (like accessing a DNS resolver over inet6,
> talking to a dccifd host over inet6 socket, SPAMC protocol);
>
>
> SPAMC
>
> - Mail::SpamAssasin::Client ping may erroneously result in broken pipe;
> bump spamc protocol version to 1.5, updated spamd, spamc and Client.pm;
>
> - added -n / --connect-timeout switch to spamc, allowing separate
> connection timeout from communication timeout;
>
> - added --filter-retries and --filter-retry-sleep
>
> - increased allowed line length in spamc.conf files to 8 KiB and report
> an error when the limit is exceeded
>
> - spamc would not time out connections to a hung spamd, fixed
>
> - spamc client library leaked the zlib compression buffer if compression
> is used
>
> - spamc long option '--dest' was broken
>
>
> SPAMD
>
> - when spamd is started with the daemonize option do not exit the parent
> until a child signals that it has logged the pid, to allow a wrapper
> script to simply continue immediately after starting spamd;
>
> - additional tempfile cleanup in kill_handler;
>
> - added SPAMD_LOCALHOST option to "make test" to allow specifying
> non-127.0.0.1 IP address for use in FreeBSD jail
>
>
> API
>
> - adding one optional argument to Mail::SpamAssassin::parse allows caller
> to pass additional out-of-band information to SpamAssassin (such as a
> deadline time, DKIM verification results, information about a SMTP session,
> or dynamic rule hits); this information is made available to plugins and
> the rest of the code through a 'suppl_attrib' hash;
>
> - Plugin::Check - pick up 'rule_hits' from caller via the new mechanism
> and call got_hit() on them;
>
> - simplified adding dynamic score hits and dynamic rules by plugins
> (such as AWL, CRM114, FuzzyOcr, Check) by letting got_hit() accept
> options tflags and description, and letting it store a supplied
> dynamic score for proper reporting;
>
> - let the timing breakdown information be accessible to a caller through
> the existing get_tag mechanism (tag TIMING);
>
> - let the generated header fields ('add_header' configuration options)
> be accessible to a caller through the existing get_tag mechanism
> (tags ADDEDHEADER, ADDEDHEADERHAM, ADDEDHEADERSPAM);
>
>
> RULES
>
> - rules are no longer distributed with the package;
>
> - new scores have been generated by a GA algorithm and then manually tweaked,
> based on cleaned datasets supplied by a dozen of volunteers;
>
> - dropped redundant rules or rules causing too many false positives;
>
> - added or updated many rules; incomplete list in no particular order:
> vbounce, lotsa_money, muchmoney, image spam, fill_this_form, FreeMail,
> European Parliament, HTML attachments, uri_obfu*, urinsrhsbl, urinsrhssub,
> urifullnsrhsbl, URI_OBFU_X9_WS, rDNS=localhost, INVALID_DATE_TZ_ABSURD,
> KHOP_SC, RCVD_IN_PSBL, FRT_VALIUM*, BOUNCE_MESSAGE, VBOUNCE_MESSAGE,
> __BOUNCE_UNDELIVERABLE, HELO_STATIC_HOST, FILL_THIS_FORM_FRAUD_PHISH,
> CHALLENGE_RESPONSE, DKIM_VALID, DKIM_VALID_AU, DKIM_ADSP_*,
> NML_ADSP_CUSTOM_{LOW,MED,HIGH}, __VIA_ML, MIME_BASE64_TEXT, LOTTO_URI,
> FORGED_MUA_THEBAT_BOUN, FORGED_MUA_THEBAT_CS, UNRESOLVED_TEMPLATE,
> __THEBAT_MUA, __ANY_OUTLOOK_MUA, RP_MATCHES_RCVD, one-word X-Mailer,
> advance_fee update, tweak SPAN rules, tweak skype and misquoted-HTML rules,
> added some new HTML obfuscation and Google feedproxy URI rules,
> tweak reevolved advance fee second-order metarules,
> added a test rule for postmaster+abuse missing, FROM_MISSPACED,
> fix FROM_CONTAINS_TAB, added Facebook redirector pattern,
> avoided ISO-2022-JP FPs on TVD_SPACE_RATIO, GAPPY_SUBJECT, PLING_QUERY
> and FM_FRM_RN_L_BRACK rules, FP fix for one-word mails on TVD_SPACE_RATIO,
> RATWARE_BOUNDARY plus variant, supersede all previous RATWARE_OUTLOOK
> stuff, added exclusion for __ISO_2022_JP_DELIM to OBFUSCATING_COMMENT,
> FP in obfuscated URI rule, fixed breakage in tbird image rule, fixed
> SUBJECT_FUZZY_MEDS FP on unobfuscated "meds", added misspaced From header
> field rule, numeric+cctld URI rule, ...
>
> - added PSBL blacklist - http://psbl.surriel.com/
>
> - added support for http://www.spamhaus.org/css/
>
> - replaces HABEAS, BSP and SSC with RP CERTIFIED;
>
> - use ReturnPath's RNBL, replacing SSBL;
>
> - added rule for plain text attachments with octet-stream MIME type;
>
> - avoided false positives on ISO-2022-JP messages in several rules;
>
> - removed massmailers from uridnsbl_skip_domain in 25_uribl.cf;
>
> - updated various default whitelists, uridnsbl_skip_domain, adsp_override, ...
>
>
> PLUGINS
>
> - new plugins: FreeMail, PhishTag, Reuse
>
> - now enabled by default: DKIM
>
> - now disabled by default: AWL
>
> - retired plugin: DomainKeys
>
>
> AWL PLUGIN
>
> - plugin AWL is now disabled by default;
>
> - added new configuration options auto_whitelist_ipv4_mask_len and
> auto_whitelist_ipv6_mask_len to allow more control on what part of
> an IP address is stored into an AWL database;
>
> - README.awl: increased a suggested awl.ip field width to 40 characters
> to support IPv6 addresses;
>
> - AutoWhitelist.pm: allowed storing a canonicalized IPv6 address, cropped
> to a configurable network mask (previously causing SQL server errors:
> 'value too long')
>
> - let AWL with SQL keep separate records for DKIM-signed and unsigned mail
> (when auto_whitelist_distinguish_signed configuration option is true,
> and a field awl.signedby exists);
>
> - avoided a race condition in SQLBasedAddrList.pm when multiple processes
> try to insert-or-update an awl SQL record: trying INSERT first, and if
> that fails go for UPDATE;
>
> - gracefully handle NaN from corrupted database or a broken emulator or
> virtualizer;
>
>
> DCC PLUGIN
>
> - added support for DCC reputations, added setting dcc_rep_percent,
> new test check_dcc_reputation_range(), new tag DCCREP
> (DCC servers supply reputation data only to licensed clients);
>
> - allowed usage of a remote dccifd host through an INET or INET6 socket;
>
>
> DKIM PLUGIN
>
> - the plugin is now enabled by default;
>
> - absolute minimal version of Mail::DKIM is 0.31;
> support for ADSP requires Mail::DKIM 0.34;
> a DNS test (and rule) for NXDOMAIN is operational since Mail::DKIM 0.36_5
>
> - a perl module Digest::SHA is required if the DKIM plugin is enabled
> (if a perl module Digest::SHA is available, the module Digest::SHA1
> becomes optional as far as SpamAssassin is concerned (but is still
> needed by Razor agents));
>
> - added support for multiple signatures (useful for whitelisting);
>
> - plugin now distinguishes author domain signatures from third party
> signatures (useful for whitelisting);
>
> - provides a tag DKIMIDENTITY (in addition to DKIMDOMAIN);
>
> - DKIM now supports Author Domain Signing Practices - ADSP (RFC 5617);
>
> - use the Mail::DKIM::AuthorDomainPolicy instead of Mail::DKIM::DkimPolicy,
> when available (since Mail::DKIM 0.34);
>
> - implements an 'adsp_override' configuration directive and adds
> an eval:check_dkim_adsp check, which is used by new DKIM_ADSP_* rules;
>
> - rules contain an initial set of 'adsp_override' directives, listing
> some of the more popular target domains for phishing (applicable only to
> domains which sign all their direct mail with a DKIM or DK signature);
>
> - this plugin can now re-use Mail::DKIM verification results if made
> available by a caller, which saves resources and makes it possible
> for SpamAssassin to work on a truncated large mail without breaking
> DKIM signatures;
>
> - check_dkim_signed and check_dkim_adsp eval rules can now take an optional
> list of domain names, which limits their action to listed domains only.
> It facilitates building DKIM-based rules for specific domains, without
> having to resort to meta rules;
>
> - draft-ietf-dkim-ssp-10/RFC-5617 made Author Domain Signature based on 'd':
> updated ADSP code accordingly; changed whitelisting code to be based on
> SDID ('d') instead of AUID ('i');
>
> - Plugin/DKIM.pm: terminology changes in comments and logging according
> to RFC 5617 and draft-ietf-dkim-rfc4871-errata-07;
>
>
> BUG FIXES
>
> - fixed Rule2XSBody segfaults;
>
> - no longer treat user data as perl booleans (a string "0" is a false);
>
> - avoid data from the wild be interpreted as perl regular expressions;
>
> - ArchiveIterator: prevent _scan_directory from passing directories
> to _scan_file (on NFS it would fail with EISDIR on read(2);
>
> - fixed vpopmail support;
>
> - fixed incorrect mode bits when creating lock files for AWL;
>
> - fixed some cases where :addr headers were parsed incorrectly;
>
> - fixed leakage of 'whitelist_from_rcvd' entries between spamd users;
>
> - fixing run_and_catch, which failed to catch a non-timed run;
>
> - 127/8 isn't an illegal IP;
>
> - reworked the M::S::Timeout module to deal with nested timers as one would
> expect: an inner timer shouldn't be able to extend an outer timer's limit;
> account for time elapsed in the submitted subroutine when restarting an
> outer timer; reset() should have accounted for time already spent;
> deal with nested timed runs where alarm(0) does not provide remaining time;
>
> - the 'exists:' evaluator in HEADER rules now works as documented
> and tests for existence of a header field, instead of testing for
> a header field body being nonempty; internally, the pms->get can
> also now distinguish between empty and nonexistent header fields;
>
> - applied fixes to header fields parsing in several places: header field
> names are case-insensitive, whitespace is not required after a colon,
> obsolete rfc822 syntax allowed whitespace before a colon;
> VBounce: match "Received:" only at the beginning of a line;
>
> - fixed bug 6237: 2.0.0.0/8 is now an allocated address range,
> fixed RCVD_ILLEGAL_IP with IP 2.0.0.0/8 (and 223.0.0.0/8);
>
> - fixed bug 6205 comment 5 in URIDetail.pm;
>
> - 'pyzor_options' in Plugin/Pyzor.pm was not untainted;
>
> - URIDetail plugin was not taint safe, fixed;
>
> - fixed parsing of multi-line Received header fields for
> BOUNCE_MESSAGE/VBOUNCE_MESSAGE et al;
>
> - Bug 6206, Bug 2536: spamd: untaint directory as obtained from a password
> file or from vpopmail utilities, avoid implicit untainting; report error
> if user preferences file exists but cannot be accessed;
>
> - avoid using raw data from DNS as a regexp in Plugin/ASN.pm;
>
> - ensured the dbg() and info() calls always return the same value (true)
> regardless of log level;
>
> - suppress logging of $& when its value is not available (i.e. when
> no regexp has been evaluated during rule evaluation);
>
> - Exporter never really worked in SA, was not enclosed in BEGIN {};
>
> - masses/runGA and masses/mk-baseline-results: prevent a shell 'source'
> command from loading an unrelated file named 'config' which happens to be
> in the current PATH - must use a ./ in an arg to a 'source' command;
>
>
> ERROR HANDLING, ROBUSTNESS
>
> - improved error detection and reporting: test status of all system calls
> and I/O operations (or explicitly document where not), and report
> unexpected failures;
>
> - eval calls now check for eval result instead of testing the $@, which
> is not always reliable;
>
> - localized $@ and $! in DESTROY methods to prevent potential calls to eval
> and calls to system routines in code executed from a DESTROY method
> from clobbering global variables $@ and $!;
>
> - Util::helper_app_pipe_open_unix: contain a failing exec with an eval
> to prevent additional cases of process cloning. The exec could fail
> this way when given tainted arguments;
>
> - Util::helper_app_pipe_open_unix: flush stdout and stderr before forking,
> otherwise an error reported by exec (such as 'insecure dependency')
> was lost in a buffer;
>
> - eval-protected an open($fh,'-|') to capture implied fork failures
> due to lack of system resource;
>
> - explicit untainting: combine "use re 'taint'" with untaint_var(),
> avoiding implicit perl untainting, along with workarounds to prevent it;
>
> - added 'use strict' where missing;
>
> - avoided a bunch of warnings on "Use of uninitialized value"
>
> - clearly report reasons for helper application process failures
>
> - t/SATest.pm: provide information about the process failure reason
> if a system() call fails; improved its reporting of failures;
>
> - improved error reporting in Plugin/DCC.pm on finding a DCC home directory
> to facilitate troubleshooting;
>
>
> OTHER CHANGES
>
> - pseudoheader "ALL:raw" returns a pristine header section,
> and pseudoheader "ALL" returns a cleaned header section
>
> - total rewrite of URI detection in plain text body;
>
> - many updates to the list of top level domains;
>
> - added 'util_rb_3tld', allowing 3-level TLDs to be listed in URIBLs and
> allowing new 3TLDs to be added from rule updates;
>
> - avoided trusted_networks bog down due to O(n^2) loop with millions
> of entries;
>
> - applied fixes to Plugin/VBounce.pm, updated VBounce ruleset;
>
> - added support for a 'Communigate Pro' Received header field;
>
> - parse Communigate Pro "with HTTPU" auth token;
>
> - let DependencyInfo.pm understand a concept of recommended module version,
> besides a required version;
>
> - provided a workaround for Net::DNS::Packet::new inconsistency;
>
> - let SpamAssassin use either Digest::SHA or Digest::SHA1, whichever is
> available (the Digest::SHA is now a base module since perl 5.10.0);
>
> - improved parsing of eval-type rules: allow unquoted domain names,
> disallow unmatched quotes;
>
> - provided a new module Mail::SpamAssassin::BayesStore::BDB. It should be
> treated as alpha-quality (needs more testing) and is not yet ready for
> production use;
>
> - exposed existing function 'received_within_months' as an eval function
> in Plugin/HeaderEval.pm;
>
> - use /var/lock/subsys/spamd instead of /var/lock/subsys/spamassassin for
> rc script, so that 'service spamd status' will work;
>
> - re-download MIRRRORED.BY files at least once a week, or if
> 'sa-update --refreshmirrors' switch is used;
>
> - input delimiter $/ can be corrupted by a plugin, localize $/ and $\ before
> calling a plugin;
>
> - takes almost a minute to start spamd on a slow machine, bumped up the
> retry counter to 180 seconds;
>
> - resolved Bug 5325: syslog severity level in spamc/libspamc.c for max
> message size (changed LOG_ERR into LOG_NOTICE for the message:
> "skipped message, greater than max message size");
>
> - avoid taint warnings if hostname is returned as '(none)';
>
> - produce an error message if an sa-update channel doesn't exist;
>
> - Bug 6150, Bug 6127, Bug 5981, Bug 5950, Bug 6191: let spamd log/report
> a child process exit status or aborting condition in an informative way;
>
> - detect accidental match-everything regexps in rules;
>
> - updated garescorer for 3.3.0: use more epochs in GA runs for better scores;
> clarify some mass-check warning output, ensure rule name always appears at
> start of line; if a rule had no default/existing score in 50_scores.cf,
> don't tell the GA that 1.0 is an appropriate default value, instead pick
> the midway point of its score range. this produces better results;
> remove some dead code from masses/score-ranges-from-freqs;
>
> - report performance as iterations per second in garescorer.c;
>
> - added test to ensure that all config settings are correctly handled when
> switching between users; added more config setting type metadata to enable
> those tests to work; and fix URIDetail to store config on the {conf} object,
> not on the plugin;
>
> - moved 'release tests' to xt/ directory; mirror long-running, net-tests and
> stress tests with xt/50_testname.t scripts to enforce their run before a
> release;
>
> - numerous additional and updated self-tests;
>
> - added a Test::Perl::Critic release-test;
>
> - some code cleanups based on suggestions by a perl module Test::Perl::Critic,
> among others:
> . enable TestingAndDebugging::ProhibitNoStrict test but allow the
> use of 'no strict "refs"';
> . deal with BuiltinFunctions::RequireGlobFunction;
> . deal with ControlStructures::ProhibitMutatingListFunctions
> removing this exception from xt/60_perlcritic.t;
> . deal with BayesStore/BDB.pm, Variables::ProhibitConditionalDeclarations
> . now that the module Time::HiRes is a required module, we can afford
> to replace a select() with Time::HiRes::sleep, and remove exception
> BuiltinFunctions::ProhibitSleepViaSelect from xt/60_perlcritic.t
>
> - documentation was updated, fixing numerous typos and mistakes in
> documentation text and in log messages;
>
> - extensive improvements to development process:
> automated testing through Hudson, improvements to mass-check and rules

PROPOSED: Apache SpamAssassin 3.3.0-rc1.proposed1

2009-12-18T17:57:27Z

This will be released if we go three days without an objection as per build/README procedure. At that point these archives will be renamed to "rc1" and the announcements will go out. Please suggest improvements to this announcement text as well.

Hey users list, now would be a very good time to begin testing 3.3.0 if you haven't already. At this point it has been tested in production on many production servers (including my own production server since March 2009), but it is possible we missed a corner case of some non-standard configuration that you folks rely upon. We could use your feedback, even if it is only "It works!" Now is last chance to complain if you find a problem.

All of the Priority P1 blocker bugs targeted for 3.3.0 are now closed. I suspect there might be a few minor things we might want to polish before 3.3.0 final, but otherwise this is VERY CLOSE to what 3.3.0 will be.

Warren Togami
wtogami@...

[DRAFT DRAFT DRAFT - NOT YET RELEASED - DRAFT DRAFT DRAFT]

Apache SpamAssassin 3.3.0-rc1 is now available for testing.

Downloads are available from:
http://people.apache.org/~wtogami/devel/
md5sum of archive files:

fabccde7f09dcdf4c06afbd4cc0687c2 Mail-SpamAssassin-3.3.0-rc1.proposed1.tar.bz2
a36c646742c0cd5960e7be222cc91200 Mail-SpamAssassin-3.3.0-rc1.proposed1.tar.gz
8c7f92167f2c2016164aafeac925ac58 Mail-SpamAssassin-3.3.0-rc1.proposed1.zip
0bf1e2c0fc70dc05104acec8a723cbdb Mail-SpamAssassin-rules-3.3.0-rc1.r892385.tgz

sha1sum of archive files:

7101ea1ab28b47e557583b914e242651f6cd761e Mail-SpamAssassin-3.3.0-rc1.proposed1.tar.bz2
344dd5887772abc0fc0aea11a30dff5d77d67db8 Mail-SpamAssassin-3.3.0-rc1.proposed1.tar.gz
d9e852d2c64cf23e86b2ae0e7f3e6082a028c1a2 Mail-SpamAssassin-3.3.0-rc1.proposed1.zip
ad71dbf4250dddd00d885311b14d39c0c03e2ae1 Mail-SpamAssassin-rules-3.3.0-rc1.r892385.tgz

Note that the *-rules-*.tgz files are only necessary if you cannot, or do not
wish to, run "sa-update" after install to download the latest fresh rules.

The release files also have a .asc accompanying them. The file serves
as an external GPG signature for the given release file. The signing
key is available via the wwwkeys.pgp.net key server, as well as
http://www.apache.org/dist/spamassassin/KEYS

The key information is:

pub 4096R/F7D39814 2009-12-02
Key fingerprint = D809 9BC7 9E17 D7E4 9BC2 1E31 FDE5 2F40 F7D3 9814
uid SpamAssassin Project Management Committee <private@...>
uid SpamAssassin Signing Key (Code Signing Key, replacement for 1024D/265FA05B) <dev@...>
sub 4096R/7B3265A5 2009-12-02

See the INSTALL and UPGRADE files in the distribution for important
installation notes.

Summary of major changes since 3.2.5
------------------------------------

COMPATIBILITY WITH 3.2.5

- rules are no longer distributed with the package, but installed by
sa-update - either automatically fetched from the network (preferably),
or from a tar archive, which is available for downloading separately

- CPAN module requirements:
- minimum required version of ExtUtils::MakeMaker is 6.17
- modules now required: Time::HiRes, NetAddr::IP, Archive::Tar
- minimal version of Mail::DKIM is 0.31 (preferred: 0.37 or later);
expect some tests in t/dkim2.t to fail with versions older than 0.36_5;
- no longer used: Mail::DomainKeys, Mail::SPF::Query
- if module Digest::SHA is not available, a module Digest::SHA1
will be used, but at least one of them must be installed;
a DKIM plugin requires Digest::SHA (the older Digest::SHA1 does not
support sha256 hashes), so in practice the Digest::SHA is required

- if keeping AWL database in SQL, the field awl.ip must be extended to
40 characters. The change is necessary to allow AWL to keep track of IPv6
addresses which may appear in a mail header even on non-IPv6 -enabled host.
While at it, consider also adding a field 'signedby' to the SQL table 'awl'
(and adding 'auto_whitelist_distinguish_signed 1' to local.cf);
See sql/README.awl for details. The change need not be undone even if
downgrading back to 3.2.* for some reason;

- fixing a protocol implementation error regarding a PING command required
bumping up the SPAMC protocol version to 1.5. Spamd retains compatibility
with older spamc clients. Combining new spamc clients with pre-3.3 versions
of a spamd daemon is not supported (but happens to work, except for the
PING and SKIP commands).

- it may be worth mentioning that a rule DKIM_VERIFIED has been renamed
to DKIM_VALID, to match its semantics;

- due to a change in internal data structure (Bug 6185, 6254), third-party
plugins which accesss the $pms->{main}->{conf}->{headers_spam} (and ham)
need to be updated. One such example is the ClamAVPlugin plugin - please
find a fresh version on its wiki page. It retains backwards compatibility,
so can be used with both 3.2.5 as well as with SpamAssassin 3.3.0;

- versions of amavisd-new between 2.5.2 and 2.6.1 (inclusive) are incompatible
with SpamAssassin 3.3; please upgrade amavisd to 2.6.2 or later, or apply
a workaround https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6257

- support for versions of perl 5.6.* is being gradually revoked
(may still work, but no promises and no support);

- preferred versions of perl are 5.8.8, 5.8.9, and 5.10.1 or later

MAIN NEW FEATURES

- IPv6 support was substantially improved (see below);

- many improvements to the DKIM plugin (understands author domain signatures,
supports multiple signatures, ADSP support with overrides) - (see below);

- added 'if can(Class::method)' conditional statement, allowing configuration
settings to be conditionalised on plugin capabilities without requiring
new version releases to do so;

- added a configuration option 'time_limit', defaulting to 300 seconds
or whatever the caller (like spamd) provides; attempting to gracefully
terminate the checking when a time limit is reached, reporting the score
and test hits that were collected so far, along with an added hit on
a rule TIME_LIMIT_EXCEEDED;

- more expensive code sections are now instrumented with timing measurements;
timing report is logged as a debug message by the end of processing,
and made available to a caller and to 'add_header' directives through
a TIMING tag;

- added a configuration option skip_uribl_checks to the URIDNSBL plugin,
cross-document it with skip_rbl_checks;

- preserve order of declared 'add_header' header fields;

- configurable network mask length for the AWL plugin (see below);

- added support for DCC reputations (see below);

- improved error handling and robustness (see below);

- added timestamps when logging on stderr;

- allowed debug areas to be excluded from debugging,
e.g.: -D all,norules,noconfig,nodcc

BUILDING AND PACKAGING

- rules are no longer distributed with the package, but installed by
sa-update

- Makefile.PL has been simplified and a bug fixed in a DESTDIR support
by increasing the minimum required version of ExtUtils::MakeMaker to 6.17

- tools check_whitelist and check_spamd are now included in the distribution,
now called 'sa-awl' and 'sa-check_spamd'

WORKAROUNDS TO PERL BUGS AND LIMITATIONS

- modified the Check.pm plugin to produce smaller chunks of source code
from rules (60 kB) to avoid Perl compiler crashing on exceeding stack size

- localized global variables $1, $2, etc at several places, avoiding taint
issue from propagating

- avoided Perl I/O bug by replacing line-by-line reading with read() where
suitable, or played down the EBADF status in other places and only report
it as a dbg instead of a die - while also providing a little speedup
(10 .. 25 %) on reading a message

- provided a new sub Message::split_into_array_of_short_lines to split
a text into array of paragraph chunks of sizes between 1 kB and 2 kB,
giving less opportunity to runaway regular expressions in rules;
fixes bugs: 5717, 5644, 5795, 5486, 5801, 5041

MEMORY FOOTPRINT

- as a side-effect of compiling rules in smaller chunks (to avoid compiler
crashes), virtual memory footprint of SpamAssassin is reduced;

- saved some memory by not importing the Pod::Usage unless it is needed;

- saved 350k+ of memory in sa-compile by replacing DynaLoader with XSLoader;

- removed unneeded index from MySQL bayes_token table;

IPv6 SUPPORT

- added IPv6 support for trusted_networks, internal_networks, msa_networks,
whitelist_from_rcvd, and other stuff that uses NetSet and the Received
header field parser, using NetAddr::IP;

- allowed usage of a remote dccifd host through an INET or INET6 socket;

- added IPv6 support to AWL plugin and its utility modules; a network
mask length is now configurable and defaults to /48, which controls
what data is stored in an AWL database;

- sql/README.awl and sql/awl_*.sql: increased suggested awl.ip field width
to 40 characters to be able to hold IPv6 addresses;

- IP_PRIVATE now includes ipv6 variants of private address space,
as well as the ipv6-mapped ipv4 addresses.

- NetSet now understands that ::ffff:192.168.1.2 and 192.168.1.2 are
the same address;

- IPv6 addresses are now recognised in Received header fields;

- when reading Received header fields, the "IPv6:" prefix is stripped from
IPv6 addresses, and "::ffff:" is removed from IPv6-mapped IPv4 addresses
(so strings can match them as simply IPv4 addresses);

- ::1/128 is always included in the trusted_networks/internal_networks set
similar to 127.0.0.0/8;

- some of the IPv6 functionality in SpamAssassin requires that a perl module
IO::Socket::INET6 is available (like accessing a DNS resolver over inet6,
talking to a dccifd host over inet6 socket, SPAMC protocol);

SPAMC

- Mail::SpamAssasin::Client ping may erroneously result in broken pipe;
bump spamc protocol version to 1.5, updated spamd, spamc and Client.pm;

- added -n / --connect-timeout switch to spamc, allowing separate
connection timeout from communication timeout;

- added --filter-retries and --filter-retry-sleep

- increased allowed line length in spamc.conf files to 8 KiB and report
an error when the limit is exceeded

- spamc would not time out connections to a hung spamd, fixed

- spamc client library leaked the zlib compression buffer if compression
is used

- spamc long option '--dest' was broken

SPAMD

- when spamd is started with the daemonize option do not exit the parent
until a child signals that it has logged the pid, to allow a wrapper
script to simply continue immediately after starting spamd;

- additional tempfile cleanup in kill_handler;

- added SPAMD_LOCALHOST option to "make test" to allow specifying
non-127.0.0.1 IP address for use in FreeBSD jail

API

- adding one optional argument to Mail::SpamAssassin::parse allows caller
to pass additional out-of-band information to SpamAssassin (such as a
deadline time, DKIM verification results, information about a SMTP session,
or dynamic rule hits); this information is made available to plugins and
the rest of the code through a 'suppl_attrib' hash;

- Plugin::Check - pick up 'rule_hits' from caller via the new mechanism
and call got_hit() on them;

- simplified adding dynamic score hits and dynamic rules by plugins
(such as AWL, CRM114, FuzzyOcr, Check) by letting got_hit() accept
options tflags and description, and letting it store a supplied
dynamic score for proper reporting;

- let the timing breakdown information be accessible to a caller through
the existing get_tag mechanism (tag TIMING);

- let the generated header fields ('add_header' configuration options)
be accessible to a caller through the existing get_tag mechanism
(tags ADDEDHEADER, ADDEDHEADERHAM, ADDEDHEADERSPAM);

RULES

- rules are no longer distributed with the package;

- new scores have been generated by a GA algorithm and then manually tweaked,
based on cleaned datasets supplied by a dozen of volunteers;

- dropped redundant rules or rules causing too many false positives;

- added or updated many rules; incomplete list in no particular order:
vbounce, lotsa_money, muchmoney, image spam, fill_this_form, FreeMail,
European Parliament, HTML attachments, uri_obfu*, urinsrhsbl, urinsrhssub,
urifullnsrhsbl, URI_OBFU_X9_WS, rDNS=localhost, INVALID_DATE_TZ_ABSURD,
KHOP_SC, RCVD_IN_PSBL, FRT_VALIUM*, BOUNCE_MESSAGE, VBOUNCE_MESSAGE,
__BOUNCE_UNDELIVERABLE, HELO_STATIC_HOST, FILL_THIS_FORM_FRAUD_PHISH,
CHALLENGE_RESPONSE, DKIM_VALID, DKIM_VALID_AU, DKIM_ADSP_*,
NML_ADSP_CUSTOM_{LOW,MED,HIGH}, __VIA_ML, MIME_BASE64_TEXT, LOTTO_URI,
FORGED_MUA_THEBAT_BOUN, FORGED_MUA_THEBAT_CS, UNRESOLVED_TEMPLATE,
__THEBAT_MUA, __ANY_OUTLOOK_MUA, RP_MATCHES_RCVD, one-word X-Mailer,
advance_fee update, tweak SPAN rules, tweak skype and misquoted-HTML rules,
added some new HTML obfuscation and Google feedproxy URI rules,
tweak reevolved advance fee second-order metarules,
added a test rule for postmaster+abuse missing, FROM_MISSPACED,
fix FROM_CONTAINS_TAB, added Facebook redirector pattern,
avoided ISO-2022-JP FPs on TVD_SPACE_RATIO, GAPPY_SUBJECT, PLING_QUERY
and FM_FRM_RN_L_BRACK rules, FP fix for one-word mails on TVD_SPACE_RATIO,
RATWARE_BOUNDARY plus variant, supersede all previous RATWARE_OUTLOOK
stuff, added exclusion for __ISO_2022_JP_DELIM to OBFUSCATING_COMMENT,
FP in obfuscated URI rule, fixed breakage in tbird image rule, fixed
SUBJECT_FUZZY_MEDS FP on unobfuscated "meds", added misspaced From header
field rule, numeric+cctld URI rule, ...

- added PSBL blacklist - http://psbl.surriel.com/

- added support for http://www.spamhaus.org/css/

- replaces HABEAS, BSP and SSC with RP CERTIFIED;

- use ReturnPath's RNBL, replacing SSBL;

- added rule for plain text attachments with octet-stream MIME type;

- avoided false positives on ISO-2022-JP messages in several rules;

- removed massmailers from uridnsbl_skip_domain in 25_uribl.cf;

- updated various default whitelists, uridnsbl_skip_domain, adsp_override, ...

PLUGINS

- new plugins: FreeMail, PhishTag, Reuse

- now enabled by default: DKIM

- now disabled by default: AWL

- retired plugin: DomainKeys

AWL PLUGIN

- plugin AWL is now disabled by default;

- added new configuration options auto_whitelist_ipv4_mask_len and
auto_whitelist_ipv6_mask_len to allow more control on what part of
an IP address is stored into an AWL database;

- README.awl: increased a suggested awl.ip field width to 40 characters
to support IPv6 addresses;

- AutoWhitelist.pm: allowed storing a canonicalized IPv6 address, cropped
to a configurable network mask (previously causing SQL server errors:
'value too long')

- let AWL with SQL keep separate records for DKIM-signed and unsigned mail
(when auto_whitelist_distinguish_signed configuration option is true,
and a field awl.signedby exists);

- avoided a race condition in SQLBasedAddrList.pm when multiple processes
try to insert-or-update an awl SQL record: trying INSERT first, and if
that fails go for UPDATE;

- gracefully handle NaN from corrupted database or a broken emulator or
virtualizer;

DCC PLUGIN

- added support for DCC reputations, added setting dcc_rep_percent,
new test check_dcc_reputation_range(), new tag DCCREP
(DCC servers supply reputation data only to licensed clients);

- allowed usage of a remote dccifd host through an INET or INET6 socket;

DKIM PLUGIN

- the plugin is now enabled by default;

- absolute minimal version of Mail::DKIM is 0.31;
support for ADSP requires Mail::DKIM 0.34;
a DNS test (and rule) for NXDOMAIN is operational since Mail::DKIM 0.36_5

- a perl module Digest::SHA is required if the DKIM plugin is enabled
(if a perl module Digest::SHA is available, the module Digest::SHA1
becomes optional as far as SpamAssassin is concerned (but is still
needed by Razor agents));

- added support for multiple signatures (useful for whitelisting);

- plugin now distinguishes author domain signatures from third party
signatures (useful for whitelisting);

- provides a tag DKIMIDENTITY (in addition to DKIMDOMAIN);

- DKIM now supports Author Domain Signing Practices - ADSP (RFC 5617);

- use the Mail::DKIM::AuthorDomainPolicy instead of Mail::DKIM::DkimPolicy,
when available (since Mail::DKIM 0.34);

- implements an 'adsp_override' configuration directive and adds
an eval:check_dkim_adsp check, which is used by new DKIM_ADSP_* rules;

- rules contain an initial set of 'adsp_override' directives, listing
some of the more popular target domains for phishing (applicable only to
domains which sign all their direct mail with a DKIM or DK signature);

- this plugin can now re-use Mail::DKIM verification results if made
available by a caller, which saves resources and makes it possible
for SpamAssassin to work on a truncated large mail without breaking
DKIM signatures;

- check_dkim_signed and check_dkim_adsp eval rules can now take an optional
list of domain names, which limits their action to listed domains only.
It facilitates building DKIM-based rules for specific domains, without
having to resort to meta rules;

- draft-ietf-dkim-ssp-10/RFC-5617 made Author Domain Signature based on 'd':
updated ADSP code accordingly; changed whitelisting code to be based on
SDID ('d') instead of AUID ('i');

- Plugin/DKIM.pm: terminology changes in comments and logging according
to RFC 5617 and draft-ietf-dkim-rfc4871-errata-07;

BUG FIXES

- fixed Rule2XSBody segfaults;

- no longer treat user data as perl booleans (a string "0" is a false);

- avoid data from the wild be interpreted as perl regular expressions;

- ArchiveIterator: prevent _scan_directory from passing directories
to _scan_file (on NFS it would fail with EISDIR on read(2);

- fixed vpopmail support;

- fixed incorrect mode bits when creating lock files for AWL;

- fixed some cases where :addr headers were parsed incorrectly;

- fixed leakage of 'whitelist_from_rcvd' entries between spamd users;

- fixing run_and_catch, which failed to catch a non-timed run;

- 127/8 isn't an illegal IP;

- reworked the M::S::Timeout module to deal with nested timers as one would
expect: an inner timer shouldn't be able to extend an outer timer's limit;
account for time elapsed in the submitted subroutine when restarting an
outer timer; reset() should have accounted for time already spent;
deal with nested timed runs where alarm(0) does not provide remaining time;

- the 'exists:' evaluator in HEADER rules now works as documented
and tests for existence of a header field, instead of testing for
a header field body being nonempty; internally, the pms->get can
also now distinguish between empty and nonexistent header fields;

- applied fixes to header fields parsing in several places: header field
names are case-insensitive, whitespace is not required after a colon,
obsolete rfc822 syntax allowed whitespace before a colon;
VBounce: match "Received:" only at the beginning of a line;

- fixed bug 6237: 2.0.0.0/8 is now an allocated address range,
fixed RCVD_ILLEGAL_IP with IP 2.0.0.0/8 (and 223.0.0.0/8);

- fixed bug 6205 comment 5 in URIDetail.pm;

- 'pyzor_options' in Plugin/Pyzor.pm was not untainted;

- URIDetail plugin was not taint safe, fixed;

- fixed parsing of multi-line Received header fields for
BOUNCE_MESSAGE/VBOUNCE_MESSAGE et al;

- Bug 6206, Bug 2536: spamd: untaint directory as obtained from a password
file or from vpopmail utilities, avoid implicit untainting; report error
if user preferences file exists but cannot be accessed;

- avoid using raw data from DNS as a regexp in Plugin/ASN.pm;

- ensured the dbg() and info() calls always return the same value (true)
regardless of log level;

- suppress logging of $& when its value is not available (i.e. when
no regexp has been evaluated during rule evaluation);

- Exporter never really worked in SA, was not enclosed in BEGIN {};

- masses/runGA and masses/mk-baseline-results: prevent a shell 'source'
command from loading an unrelated file named 'config' which happens to be
in the current PATH - must use a ./ in an arg to a 'source' command;

ERROR HANDLING, ROBUSTNESS

- improved error detection and reporting: test status of all system calls
and I/O operations (or explicitly document where not), and report
unexpected failures;

- eval calls now check for eval result instead of testing the $@, which
is not always reliable;

- localized $@ and $! in DESTROY methods to prevent potential calls to eval
and calls to system routines in code executed from a DESTROY method
from clobbering global variables $@ and $!;

- Util::helper_app_pipe_open_unix: contain a failing exec with an eval
to prevent additional cases of process cloning. The exec could fail
this way when given tainted arguments;

- Util::helper_app_pipe_open_unix: flush stdout and stderr before forking,
otherwise an error reported by exec (such as 'insecure dependency')
was lost in a buffer;

- eval-protected an open($fh,'-|') to capture implied fork failures
due to lack of system resource;

- explicit untainting: combine "use re 'taint'" with untaint_var(),
avoiding implicit perl untainting, along with workarounds to prevent it;

- added 'use strict' where missing;

- avoided a bunch of warnings on "Use of uninitialized value"

- clearly report reasons for helper application process failures

- t/SATest.pm: provide information about the process failure reason
if a system() call fails; improved its reporting of failures;

- improved error reporting in Plugin/DCC.pm on finding a DCC home directory
to facilitate troubleshooting;

OTHER CHANGES

- pseudoheader "ALL:raw" returns a pristine header section,
and pseudoheader "ALL" returns a cleaned header section

- total rewrite of URI detection in plain text body;

- many updates to the list of top level domains;

- added 'util_rb_3tld', allowing 3-level TLDs to be listed in URIBLs and
allowing new 3TLDs to be added from rule updates;

- avoided trusted_networks bog down due to O(n^2) loop with millions
of entries;

- applied fixes to Plugin/VBounce.pm, updated VBounce ruleset;

- added support for a 'Communigate Pro' Received header field;

- parse Communigate Pro "with HTTPU" auth token;

- let DependencyInfo.pm understand a concept of recommended module version,
besides a required version;

- provided a workaround for Net::DNS::Packet::new inconsistency;

- let SpamAssassin use either Digest::SHA or Digest::SHA1, whichever is
available (the Digest::SHA is now a base module since perl 5.10.0);

- improved parsing of eval-type rules: allow unquoted domain names,
disallow unmatched quotes;

- provided a new module Mail::SpamAssassin::BayesStore::BDB. It should be
treated as alpha-quality (needs more testing) and is not yet ready for
production use;

- exposed existing function 'received_within_months' as an eval function
in Plugin/HeaderEval.pm;

- use /var/lock/subsys/spamd instead of /var/lock/subsys/spamassassin for
rc script, so that 'service spamd status' will work;

- re-download MIRRRORED.BY files at least once a week, or if
'sa-update --refreshmirrors' switch is used;

- input delimiter $/ can be corrupted by a plugin, localize $/ and $\ before
calling a plugin;

- takes almost a minute to start spamd on a slow machine, bumped up the
retry counter to 180 seconds;

- resolved Bug 5325: syslog severity level in spamc/libspamc.c for max
message size (changed LOG_ERR into LOG_NOTICE for the message:
"skipped message, greater than max message size");

- avoid taint warnings if hostname is returned as '(none)';

- produce an error message if an sa-update channel doesn't exist;

- Bug 6150, Bug 6127, Bug 5981, Bug 5950, Bug 6191: let spamd log/report
a child process exit status or aborting condition in an informative way;

- detect accidental match-everything regexps in rules;

- updated garescorer for 3.3.0: use more epochs in GA runs for better scores;
clarify some mass-check warning output, ensure rule name always appears at
start of line; if a rule had no default/existing score in 50_scores.cf,
don't tell the GA that 1.0 is an appropriate default value, instead pick
the midway point of its score range. this produces better results;
remove some dead code from masses/score-ranges-from-freqs;

- report performance as iterations per second in garescorer.c;

- added test to ensure that all config settings are correctly handled when
switching between users; added more config setting type metadata to enable
those tests to work; and fix URIDetail to store config on the {conf} object,
not on the plugin;

- moved 'release tests' to xt/ directory; mirror long-running, net-tests and
stress tests with xt/50_testname.t scripts to enforce their run before a
release;

- numerous additional and updated self-tests;

- added a Test::Perl::Critic release-test;

- some code cleanups based on suggestions by a perl module Test::Perl::Critic,
among others:
. enable TestingAndDebugging::ProhibitNoStrict test but allow the
use of 'no strict "refs"';
. deal with BuiltinFunctions::RequireGlobFunction;
. deal with ControlStructures::ProhibitMutatingListFunctions
removing this exception from xt/60_perlcritic.t;
. deal with BayesStore/BDB.pm, Variables::ProhibitConditionalDeclarations
. now that the module Time::HiRes is a required module, we can afford
to replace a select() with Time::HiRes::sleep, and remove exception
BuiltinFunctions::ProhibitSleepViaSelect from xt/60_perlcritic.t

- documentation was updated, fixing numerous typos and mistakes in
documentation text and in log messages;

- extensive improvements to development process:
automated testing through Hudson, improvements to mass-check and rules

Re: timeout.t test failure

2009-12-18T13:40:30Z

On 12/18/2009 04:24 PM, Warren Togami wrote:

> t/timeout.........................ok 15/33# Failed test 16 in
> t/timeout.t at line 99
> t/timeout.........................FAILED test 16
> Failed 1/33 tests, 96.97% okay
> <snip>
> Failed Test Stat Wstat Total Fail Failed List of Failed
> -------------------------------------------------------------------------------
>
> t/timeout.t 33 1 3.03% 16
> 61 tests skipped.
> Failed 1/159 test scripts, 99.37% okay. 1/1980 subtests failed, 99.95%
> okay.
> make: *** [test_dynamic] Error 255
>
> This failure happens on:
>
> perl-5.8.8-27.el5.x86_64
>
> All tests pass with:
>
> perl-5.10.0-82.fc12.x86_64
>
> Warren

I've run the test a dozen more times and I can't reproduce the failure.
Sigh.

Warren

timeout.t test failure

2009-12-18T13:24:15Z

t/timeout.........................ok 15/33# Failed test 16 in
t/timeout.t at line 99
t/timeout.........................FAILED test 16

Failed 1/33 tests, 96.97% okay
<snip>
Failed Test Stat Wstat Total Fail Failed List of Failed
-------------------------------------------------------------------------------
t/timeout.t 33 1 3.03% 16
61 tests skipped.
Failed 1/159 test scripts, 99.37% okay. 1/1980 subtests failed, 99.95% okay.
make: *** [test_dynamic] Error 255

This failure happens on:

perl-5.8.8-27.el5.x86_64

All tests pass with:

perl-5.10.0-82.fc12.x86_64

Warren

Re: pyzor error during spamd startup

2009-12-18T13:11:45Z

On 12/18/2009 04:10 PM, Kevin A. McGrail wrote:
> Why is pyzor running as root?

It isn't. This seems to be some self-tests during spamd startup. The
real pyzor launched by spamd runs as the spamc user.

Warren

Re: pyzor error during spamd startup

2009-12-18T13:10:31Z

Why is pyzor running as root?

----- Original Message -----

From: jm@...

To: wtogami@...

Cc: Mark.Martinec+sa@... ; dev@...

Sent: Friday, December 18, 2009 4:07 PM

Subject: Re: pyzor error during spamd startup

it should be harmless, btw, since this is just the preloading-modules step. Ugly, though.

On Fri, Dec 18, 2009 at 20:53, Warren Togami <wtogami@...> wrote:

On 12/18/2009 03:24 PM, Mark Martinec wrote:

There is no error running pyzor directly or via spamassassin -D. It is
only reporting an error during spamd startup.

Perhaps spamd is running under a different uid that your command-line tests?

Run tests under the same uid under which spamd will be running:
su vscan -c 'spamassassin -D -t<0.msg'

Mark

Dec 18 15:49:55 localhost spamd[5096]: pyzor: opening pipe: /usr/bin/pyzor check < /tmp/.spamassassin5096Y9v0nmtmp
Dec 18 15:49:55 localhost spamd[5097]: util: setuid: ruid=0 euid=0
Dec 18 15:49:55 localhost spamd[5096]: pyzor: [5097] finished: exit 1
Dec 18 15:49:55 localhost spamd[5096]: pyzor: got response: Traceback (most recent call last):\n File "/usr/bin/pyzor", line 8, in <module>\n pyzor.client.run()\n File "/usr/lib/python2.6/site-packages/pyzor
/client.py", line 1022, in run\n ExecCall().run()\n File "/usr/lib/python2.6/site-packages/pyzor/client.py", line 180, in run\n os.mkdir(homedir)\nOSError: [Errno 13] Permission denied: '/root/.pyzor'
Dec 18 15:49:55 localhost spamd[5096]: info: leaving helper-app run mode
Dec 18 15:49:55 localhost spamd[5096]: pyzor: check failed: internal error, python traceback seen in response

This is a pyzor + selinux bug, not spamassassin's fault. We have a go for spamassassin-3.3.0-rc1. Doing some further tests before cutting.

Warren

--
--j.

Re: pyzor error during spamd startup

2009-12-18T13:07:28Z

it should be harmless, btw, since this is just the preloading-modules step. Ugly, though.

On Fri, Dec 18, 2009 at 20:53, Warren Togami <wtogami@...> wrote:

On 12/18/2009 03:24 PM, Mark Martinec wrote:

There is no error running pyzor directly or via spamassassin -D. It is
only reporting an error during spamd startup.

Perhaps spamd is running under a different uid that your command-line tests?

Run tests under the same uid under which spamd will be running:
su vscan -c 'spamassassin -D -t<0.msg'

Mark

Dec 18 15:49:55 localhost spamd[5096]: pyzor: opening pipe: /usr/bin/pyzor check < /tmp/.spamassassin5096Y9v0nmtmp
Dec 18 15:49:55 localhost spamd[5097]: util: setuid: ruid=0 euid=0
Dec 18 15:49:55 localhost spamd[5096]: pyzor: [5097] finished: exit 1
Dec 18 15:49:55 localhost spamd[5096]: pyzor: got response: Traceback (most recent call last):\n File "/usr/bin/pyzor", line 8, in <module>\n pyzor.client.run()\n File "/usr/lib/python2.6/site-packages/pyzor
/client.py", line 1022, in run\n ExecCall().run()\n File "/usr/lib/python2.6/site-packages/pyzor/client.py", line 180, in run\n os.mkdir(homedir)\nOSError: [Errno 13] Permission denied: '/root/.pyzor'
Dec 18 15:49:55 localhost spamd[5096]: info: leaving helper-app run mode
Dec 18 15:49:55 localhost spamd[5096]: pyzor: check failed: internal error, python traceback seen in response

This is a pyzor + selinux bug, not spamassassin's fault. We have a go for spamassassin-3.3.0-rc1. Doing some further tests before cutting.

Warren

--
--j.

Re: pyzor error during spamd startup

2009-12-18T12:53:38Z

On 12/18/2009 03:24 PM, Mark Martinec wrote:

>> There is no error running pyzor directly or via spamassassin -D. It is
>> only reporting an error during spamd startup.
>
> Perhaps spamd is running under a different uid that your command-line tests?
>
> Run tests under the same uid under which spamd will be running:
> su vscan -c 'spamassassin -D -t<0.msg'
>
>
> Mark

Dec 18 15:49:55 localhost spamd[5096]: pyzor: opening pipe:
/usr/bin/pyzor check < /tmp/.spamassassin5096Y9v0nmtmp
Dec 18 15:49:55 localhost spamd[5097]: util: setuid: ruid=0 euid=0
Dec 18 15:49:55 localhost spamd[5096]: pyzor: [5097] finished: exit 1
Dec 18 15:49:55 localhost spamd[5096]: pyzor: got response: Traceback
(most recent call last):\n File "/usr/bin/pyzor", line 8, in <module>\n
pyzor.client.run()\n File "/usr/lib/python2.6/site-packages/pyzor
/client.py", line 1022, in run\n ExecCall().run()\n File
"/usr/lib/python2.6/site-packages/pyzor/client.py", line 180, in run\n
os.mkdir(homedir)\nOSError: [Errno 13] Permission denied: '/root/.pyzor'
Dec 18 15:49:55 localhost spamd[5096]: info: leaving helper-app run mode
Dec 18 15:49:55 localhost spamd[5096]: pyzor: check failed: internal
error, python traceback seen in response

This is a pyzor + selinux bug, not spamassassin's fault. We have a go
for spamassassin-3.3.0-rc1. Doing some further tests before cutting.

Warren

Re: pyzor error during spamd startup

2009-12-18T12:24:41Z

> There is no error running pyzor directly or via spamassassin -D. It is
> only reporting an error during spamd startup.

Perhaps spamd is running under a different uid that your command-line tests?

Run tests under the same uid under which spamd will be running:
su vscan -c 'spamassassin -D -t <0.msg'

Mark

Re: pyzor error during spamd startup

2009-12-18T12:01:52Z

On 12/18/2009 02:57 PM, Mark Martinec wrote:
> Try it from a command line: spamassasssin -D -t<0.msg
> which may be able to show more information of a failure.
>
> Also, try it directly: $ pyzor check<0.msg
>
> Mark

There is no error running pyzor directly or via spamassassin -D. It is
only reporting an error during spamd startup.

Warren

Re: pyzor error during spamd startup

2009-12-18T11:57:21Z

> Dec 18 14:23:59 localhost spamd[20801]: pyzor: check failed: internal
> error, python traceback seen in response
>
> Any idea how to see contents of this traceback that spamd reports? This
> apparently has been happening on Fedora 12 since before beta1 and
> continues in svn trunk now. But I don't see this error on RHEL5.
>
> pyzor does appear to work properly with spamc/spamd with no further
> error messages in syslog. This initial error during spamd startup is
> the only error that appears in the logs.

Try it from a command line: spamassasssin -D -t <0.msg
which may be able to show more information of a failure.

Also, try it directly: $ pyzor check <0.msg

Mark

pyzor error during spamd startup

2009-12-18T11:37:55Z

Dec 18 14:23:46 localhost spamd[20766]: spamd: server killed by SIGTERM,
shutting down
Dec 18 14:23:50 localhost spamd[20799]: logger: removing stderr method
Dec 18 14:23:59 localhost spamd[20801]: pyzor: check failed: internal
error, python traceback seen in response
Dec 18 14:23:59 localhost spamd[20801]: spamd: server started on port
783/tcp (running version 3.3.0-beta1)
Dec 18 14:23:59 localhost spamd[20801]: spamd: server pid: 20801
Dec 18 14:23:59 localhost spamd[20801]: spamd: server successfully
spawned child process, pid 20804
Dec 18 14:23:59 localhost spamd[20801]: spamd: server successfully
spawned child process, pid 20805
Dec 18 14:23:59 localhost spamd[20801]: prefork: child states: IS
Dec 18 14:23:59 localhost spamd[20801]: prefork: child states: II

Any idea how to see contents of this traceback that spamd reports? This
apparently has been happening on Fedora 12 since before beta1 and
continues in svn trunk now. But I don't see this error on RHEL5.

pyzor does appear to work properly with spamc/spamd with no further
error messages in syslog. This initial error during spamd startup is
the only error that appears in the logs.

Warren Togami
wtogami@...

Re: Reminder:: 3.3.0 pre-release cut: December 17th

2009-12-18T09:22:01Z

> Seconded. I guess I'm spinning the rc1 unless jm wants to do it.
>
> I'm reluctantly going to call the first cut rc1.proposed1 after the
> previous discussion. After the 3 day period has passed with no
> objections then it will be renamed to rc1 and released.

Fine with me.

3.3.0-rc1.txt: mention some of the recent changes
Sending build/announcements/3.3.0-rc1.txt
Committed revision 892319.

Mark

[Bug 6260] text attachments are not scanned if the MIME type is APPLICATION/OCTET-STREAM

2009-12-18T06:27:28Z

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6260

John Hardin <jhardin@...> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |jhardin@...
Target Milestone|3.3.1 |Future

--- Comment #2 from John Hardin <jhardin@...> 2009-12-18 06:27:27 UTC ---
I think I disagree with that yardstick when in most MUAs it's only a
double-click to view the payload, and in most cases the only automatically
visible (and scoreable) content is "please read the attachment".

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6228] sa-learn --backup is broken

2009-12-18T04:40:14Z

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6228

Mark Martinec <Mark.Martinec@...> changed:

What |Removed |Added
----------------------------------------------------------------------------
Summary|[review] sa-learn --backup |sa-learn --backup is broken
|is broken |

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6228] [review] sa-learn --backup is broken

2009-12-18T04:40:00Z

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6228

Mark Martinec <Mark.Martinec@...> changed:

What |Removed |Added
----------------------------------------------------------------------------
Target Milestone|3.3.0 |3.3.1
Status Whiteboard|needs one more vote |

--- Comment #23 from Mark Martinec <Mark.Martinec@...> 2009-12-18 04:39:57 UTC ---
Retargeting for 3.3.1 for a proper permanent fix.

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6228] [review] sa-learn --backup is broken

2009-12-18T03:53:17Z

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6228

--- Comment #22 from Mark Martinec <Mark.Martinec@...> 2009-12-18 03:53:14 UTC ---
Bug 6228 - sa-learn --backup is broken:
a quick-and-hackish temporary solution to a problem
of Bayes pluginization not (yet) carried out fully
Sending lib/Mail/SpamAssassin/Plugin/Bayes.pm
Sending sa-learn.raw
Committed revision 892229.

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6228] [review] sa-learn --backup is broken

2009-12-17T17:00:10Z

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6228

--- Comment #21 from Warren Togami <wtogami@...> 2009-12-17 17:00:09 UTC ---
+1

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 3208] 3.0.0 bugs to fix before release

2009-12-17T15:54:45Z

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=3208

Bug 3208 depends on bug 3364, which changed state.

Bug 3364 Summary: [review] Use of customized header yields "nan" for _HITS_ variable
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=3364

What |Old Value |New Value
----------------------------------------------------------------------------
Status|REOPENED |RESOLVED
Resolution| |FIXED

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 3364] [review] Use of customized header yields "nan" for _HITS_ variable

2009-12-17T15:54:44Z

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=3364

Mark Martinec <Mark.Martinec@...> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|REOPENED |RESOLVED
Resolution| |FIXED

--- Comment #19 from Mark Martinec <Mark.Martinec@...> 2009-12-17 15:54:40 UTC ---
> is this fixed in 3.3.0?

Yes, there is a workaround in 3.3 (r704140, r703487 and thereabout).
The bug was retargeted to 3.2.6 just in case, although the problem
apparently only happens on faulty (virtual) hardware and is not common.

I think we can just close it.

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.