Development dead

[polloxx]

Hi,

Is the spamassassin development dead?
On the website there's: 2008-06-12: SpamAssassin 3.2.5 has been released.

[Bowie Bailey]

polloxx wrote:
> Hi,
>
> Is the spamassassin development dead?
> On the website there's: 2008-06-12: SpamAssassin 3.2.5 has been released.
>  

Not quite.  If you look at svn, you'll see this:

spamassassin_20091103151200.tar.gz    03-Nov-2009 15:12    2.1M

Doesn't look dead to me!  :)

--
Bowie

[Benny Pedersen]

On ons 04 nov 2009 15:10:45 CET, polloxx wrote
> Is the spamassassin development dead?
> On the website there's: 2008-06-12: SpamAssassin 3.2.5 has been released.

join the dev maillist and ask the same question there, but as i see  
it, it being working on make sa 3.3.x stable for so long time now does  
not mean its dead, where did you get that idear from ?

--
xpoint

[Kent Borg]

I admit I have been ignoring Spamassassin because it seems to work.  I
have been pleased that Spamassassin has been regularly flagging over 94%
of my spam.  And this list seems active (if mostly ignored by me).

But this e-mail caught my eye.  Indeed, the version on spamassassin.org
is old and the version offered by my Ubuntu installation matches.  Well,
if it is working pretty well in the face of rising spam volumes, that
seems OK.

But what about the sa-update rules?  They are probably newer...yes, but
my rules are from 2009-3-13.  Do I have the latest?  Is there a better
source from which to draw rules?  Am I coasting on my ongoing spam/ham
training?


Thanks,

-kb

[Bowie Bailey]

Kent Borg wrote:

> I admit I have been ignoring Spamassassin because it seems to work.  I
> have been pleased that Spamassassin has been regularly flagging over 94%
> of my spam.  And this list seems active (if mostly ignored by me).
>
> But this e-mail caught my eye.  Indeed, the version on spamassassin.org
> is old and the version offered by my Ubuntu installation matches.  Well,
> if it is working pretty well in the face of rising spam volumes, that
> seems OK.
>
> But what about the sa-update rules?  They are probably newer...yes, but
> my rules are from 2009-3-13.  Do I have the latest?  Is there a better
> source from which to draw rules?  Am I coasting on my ongoing spam/ham
> training?

The SA core rules are not updated very often.  For the most part, they
just work.  If you are not already doing so, you may want to consider
Justin's Sought ruleset.  It is dynamically generated and updated every
4 hours or so.

http://wiki.apache.org/spamassassin/SoughtRules

I am also using these two from SARE:

90_2tld.cf.sare.sa-update.dostech.net
90_sare_freemail.cf.sare.sa-update.dostech.net

and some of Adam Katz's khop rules (http://khopesh.com/wiki/Anti-spam):

khop-bl.sa.khopesh.com
khop-blessed.sa.khopesh.com
khop-general.sa.khopesh.com
khop-sc-neighbors.sa.khopesh.com

(which also include Marc Perkel's HostKarma blacklists ... and white
lists and yellow lists and brown lists ...  :)

--
Bowie

[Charles Gregory]

On Wed, 4 Nov 2009, Bowie Bailey wrote:
> The SA core rules are not updated very often.  For the most part, they
> just work.  If you are not already doing so, you may want to consider
> Justin's Sought ruleset.  It is dynamically generated and updated every
> 4 hours or so.
>
> http://wiki.apache.org/spamassassin/SoughtRules

Is there a way to examine the sought rules *before* installing them into
my spamassassin? Or at least a 'readme' so that if I download them via
sa-update I can know which files will be created and how to remove them.
I have a number of custom rules and want to vet the auto-generated rules
for overlap....

- Charles

[Yet Another Ninja]

On 11/4/2009 5:22 PM, Charles Gregory wrote:

> On Wed, 4 Nov 2009, Bowie Bailey wrote:
>> The SA core rules are not updated very often.  For the most part, they
>> just work.  If you are not already doing so, you may want to consider
>> Justin's Sought ruleset.  It is dynamically generated and updated every
>> 4 hours or so.
>>
>> http://wiki.apache.org/spamassassin/SoughtRules
>
> Is there a way to examine the sought rules *before* installing them into
> my spamassassin? Or at least a 'readme' so that if I download them via
> sa-update I can know which files will be created and how to remove them.
> I have a number of custom rules and want to vet the auto-generated rules
> for overlap....
>
> - Charles

sa-update --help will show you how

[Bowie Bailey]

Charles Gregory wrote:

> On Wed, 4 Nov 2009, Bowie Bailey wrote:
>> The SA core rules are not updated very often.  For the most part, they
>> just work.  If you are not already doing so, you may want to consider
>> Justin's Sought ruleset.  It is dynamically generated and updated every
>> 4 hours or so.
>>
>> http://wiki.apache.org/spamassassin/SoughtRules
>
> Is there a way to examine the sought rules *before* installing them into
> my spamassassin? Or at least a 'readme' so that if I download them via
> sa-update I can know which files will be created and how to remove them.

The sa-update channel will put the rules into
"/var/lib/spamassassin/3.002005/sought_rules_yerp_org/20_sought.cf".
You can take a look at the rules there (all 293 of them at the moment).

> I have a number of custom rules and want to vet the auto-generated
> rules for overlap....

Since the rules are constantly changing (every 4 hours), I'm not sure
how useful it will be to look for overlaps.  If you find an overlap in
one version of the ruleset, it may disappear in the next update.

Actually, looking at my stats, I'm not seeing the Sought ruleset being
very helpful recently.  The best rule, JM_SOUGHT_FRAUD_3, hit less than
1% of my spam in the last month and the worst rule, JM_SOUGHT_3, hit
0.33% of spam and 0.45% of ham.  Anyone else seeing this?

The best rules for me at the moment are Razor2, DCC, and URIBL.

--
Bowie

[RW-15]

On Wed, 04 Nov 2009 11:49:26 -0500
Bowie Bailey <Bowie_Bailey@...> wrote:


> Since the rules are constantly changing (every 4 hours), I'm not sure
> how useful it will be to look for overlaps.  If you find an overlap in
> one version of the ruleset, it may disappear in the next update.

What I find a bit more worrying is that whilst  almost all of the sought
rules are specific to the point of being signatures:

   body __SEEK_6Z5IEU  /Offer top rep1ica watches up to 80\% brand name
   at affordable price , select your favorite designer fake watch
   fromRo1ex, Omega, Patek Philippe, Piaget, IWC, Breitling, Tag Heuer,
   Cartier, Chopard, Bvlgari, Chanel, Movado, Ebel, Oris, Michele, Gucci
   and so on Visit here http:\/\//

occasionally some are not:

   body __SEEK_KKIF1_  /Click here to view online version/

[Charles Gregory]

On Thu, 5 Nov 2009, RW wrote: liberally snipped......
> What I find a bit more worrying is that whilst  almost all of the sought
> rules are specific to the point of being signatures:
> occasionally some are not:
>   body __SEEK_KKIF1_  /Click here to view online version/

Another question: The rules are 'regenerated' every 4 hours.
But do old rules get deleted? Phrases like the one above may appear from
time to time in ham, but not every month. How is this hanlded? A large,
long-term corpora? Sorry, maybe this is FAQed someplace....

- Charles

[Bowie Bailey]

Charles Gregory wrote:

> On Thu, 5 Nov 2009, RW wrote: liberally snipped......
>> What I find a bit more worrying is that whilst  almost all of the sought
>> rules are specific to the point of being signatures:
>> occasionally some are not:
>>   body __SEEK_KKIF1_  /Click here to view online version/
>
> Another question: The rules are 'regenerated' every 4 hours.
> But do old rules get deleted? Phrases like the one above may appear
> from time to time in ham, but not every month. How is this hanlded? A
> large, long-term corpora? Sorry, maybe this is FAQed someplace....

I think the rules are generated from scratch each time, so there is no
such thing as an old rule.  The question would be if and when are
messages expired from the spam/ham corpus used to generate the rules.

Here is a basic description of the process:
http://taint.org/2007/03/05/134447a.html

Another question is why they have not been updated recently.  My sought
rule set right now is dated Nov 1 (and is not catching much of anything,
as I mentioned in a previous post).  Is there a problem?

--
Bowie

[Charles Gregory]

On Thu, 5 Nov 2009, Bowie Bailey wrote:
> I think the rules are generated from scratch each time, so there is no
> such thing as an old rule.  The question would be if and when are
> messages expired from the spam/ham corpus used to generate the rules.

That's the essence of my question, yes. with false positives always being
a concern, I think the retention of ham for longer periods is of
importance.

> Here is a basic description of the process:
> http://taint.org/2007/03/05/134447a.html

(nod) Yeah, I read that already. Sounds good, but leaves me with that
lingering feeling that FP's might occur. I know I had one custom rule here
at my server that worked very well until Micro$oft made some change to
their mail client, and suddenly all sorts of mail was FP'ing.... :(

> Another question is why they have not been updated recently.  My sought
> rule set right now is dated Nov 1 (and is not catching much of anything,
> as I mentioned in a previous post).  Is there a problem?

Hmmmm. I guess I'm not reading too closely. I saw your percentage figures,
but overlooked any mention of them being of date....

- Charles

[Bowie Bailey]

Charles Gregory wrote:

> On Thu, 5 Nov 2009, Bowie Bailey wrote:
>> I think the rules are generated from scratch each time, so there is no
>> such thing as an old rule.  The question would be if and when are
>> messages expired from the spam/ham corpus used to generate the rules.
>
> That's the essence of my question, yes. with false positives always
> being a concern, I think the retention of ham for longer periods is of
> importance.
>
>> Here is a basic description of the process:
>> http://taint.org/2007/03/05/134447a.html
>
> (nod) Yeah, I read that already. Sounds good, but leaves me with that
> lingering feeling that FP's might occur. I know I had one custom rule
> here at my server that worked very well until Micro$oft made some
> change to their mail client, and suddenly all sorts of mail was
> FP'ing.... :(

I think I remember hearing some discussion about that at one point.  I
don't think that type of thing is as big of a concern here since these
are all body rules.  I agree that you need a good corpus of ham to
prevent FP's, but I'm sure Justin is doing that.  These rules have been
working very well for quite a while now.

> Another question is why they have not been updated recently.  My
> sought rule set right now is dated Nov 1 (and is not catching much of
> anything, as I mentioned in a previous post).  Is there a problem?
>
> Hmmmm. I guess I'm not reading too closely. I saw your percentage
> figures, but overlooked any mention of them being of date....

That's because I didn't mention it last time.  :)

I didn't notice that they were not updating until after my last post.  I
decided to mention it today after noticing that they didn't update
overnight either.  I just removed the sought directories and let
sa-update download and re-create them, but it just grabbed the same file
again.

I hope these rules aren't going away.  They used to be the best rules
out there.

--
Bowie

[Dave Pooser]

> I think I remember hearing some discussion about that at one point.  I
> don't think that type of thing is as big of a concern here since these
> are all body rules.  I agree that you need a good corpus of ham to
> prevent FP's, but I'm sure Justin is doing that.

I'm sure he's working hard on it, but his ability is naturally going to be
limited by his ham corpus. I just saw a whole bunch of legit AmEx corporate
card updates get thrown into the quarantine bin due to hitting SOUGHT. It
happens sometimes; I've found that when I send him a sample of the mistagged
email he gets it fixed pretty quickly.
--
Dave Pooser
Cat-Herder-in-Chief, Pooserville.com
"...Life is not a journey to the grave with the intention of arriving
safely in one pretty and well-preserved piece, but to slide across the
finish line broadside, thoroughly used up, worn out, leaking oil, and
shouting GERONIMO!!!" -- Bill McKenna

[John Hardin]

On Thu, 5 Nov 2009, Dave Pooser wrote:

>> I think I remember hearing some discussion about that at one point.  I
>> don't think that type of thing is as big of a concern here since these
>> are all body rules.  I agree that you need a good corpus of ham to
>> prevent FP's, but I'm sure Justin is doing that.
>
> I'm sure he's working hard on it, but his ability is naturally going to
> be limited by his ham corpus. I just saw a whole bunch of legit AmEx
> corporate card updates get thrown into the quarantine bin due to hitting
> SOUGHT. It happens sometimes; I've found that when I send him a sample
> of the mistagged email he gets it fixed pretty quickly.

I wonder if he's considered running the ham exclusion against the
complete nightly masscheck ham corpora...?

--
  John Hardin KA7OHZ                    http://www.impsec.org/~jhardin/
  jhardin@...    FALaholic #11174     pgpk -a jhardin@...
  key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
   Men by their constitutions are naturally divided in to two parties:
   1. Those who fear and distrust the people and wish to draw all
   powers from them into the hands of the higher classes. 2. Those who
   identify themselves with the people, have confidence in them,
   cherish and consider them as the most honest and safe, although not
   the most wise, depository of the public interests.
           -- Thomas Jefferson
-----------------------------------------------------------------------
  6 days until Veterans Day

[Justin Mason]

I need the "full" mails to do that -- but with the uploaded mail, yes,
I should do that!
good point.

Right now, SOUGHT appears to be broken.  I need to get to where the server is
currently and fix it -- I don't have remote login to it at the mo :(

On Thu, Nov 5, 2009 at 18:02, John Hardin <jhardin@...> wrote:

> On Thu, 5 Nov 2009, Dave Pooser wrote:
>
>>> I think I remember hearing some discussion about that at one point.  I
>>> don't think that type of thing is as big of a concern here since these are
>>> all body rules.  I agree that you need a good corpus of ham to prevent FP's,
>>> but I'm sure Justin is doing that.
>>
>> I'm sure he's working hard on it, but his ability is naturally going to be
>> limited by his ham corpus. I just saw a whole bunch of legit AmEx corporate
>> card updates get thrown into the quarantine bin due to hitting SOUGHT. It
>> happens sometimes; I've found that when I send him a sample of the mistagged
>> email he gets it fixed pretty quickly.
>
> I wonder if he's considered running the ham exclusion against the complete
> nightly masscheck ham corpora...?
>
> --
>  John Hardin KA7OHZ                    http://www.impsec.org/~jhardin/
>  jhardin@...    FALaholic #11174     pgpk -a jhardin@...
>  key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
> -----------------------------------------------------------------------
>  Men by their constitutions are naturally divided in to two parties:
>  1. Those who fear and distrust the people and wish to draw all
>  powers from them into the hands of the higher classes. 2. Those who
>  identify themselves with the people, have confidence in them,
>  cherish and consider them as the most honest and safe, although not
>  the most wise, depository of the public interests.
>                                                  -- Thomas Jefferson
> -----------------------------------------------------------------------
>  6 days until Veterans Day
>
>

--
--j.

[John Hardin]

On Thu, 5 Nov 2009, Justin Mason wrote:

> I need the "full" mails to do that -- but with the uploaded mail, yes, I
> should do that! good point.

Glad to help.

> Right now, SOUGHT appears to be broken.  I need to get to where the
> server is currently and fix it -- I don't have remote login to it at the
> mo :(

ruleqa/nightly masscheck seems to have fallen apart, too. :(

--
  John Hardin KA7OHZ                    http://www.impsec.org/~jhardin/
  jhardin@...    FALaholic #11174     pgpk -a jhardin@...
  key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
  6 days until Veterans Day

[Justin Mason]

On Fri, Nov 6, 2009 at 00:00, John Hardin <jhardin@...> wrote:

> On Thu, 5 Nov 2009, Justin Mason wrote:
>
>> I need the "full" mails to do that -- but with the uploaded mail, yes, I
>> should do that! good point.
>
> Glad to help.
>
>> Right now, SOUGHT appears to be broken.  I need to get to where the server
>> is currently and fix it -- I don't have remote login to it at the mo :(
>
> ruleqa/nightly masscheck seems to have fallen apart, too. :(

I think I may have just fixed that.... fingers crossed.

--
--j.

[Anatoly Pugachev-5]

On 04.11.2009 / 09:20:16 -0500, Bowie Bailey wrote:

> polloxx wrote:
> > Hi,
> >
> > Is the spamassassin development dead?
> > On the website there's: 2008-06-12: SpamAssassin 3.2.5 has been released.
> >  
>
> Not quite.  If you look at svn, you'll see this:
>
> spamassassin_20091103151200.tar.gz    03-Nov-2009 15:12    2.1M
>
> Doesn't look dead to me!  :)

Hello!
Can you please post a full URL to this archive?
Since http://svn.apache.org/snapshots/spamassassin/ doesn't have it.

[Matt Kettler-3]

Anatoly Pugachev wrote:

> On 04.11.2009 / 09:20:16 -0500, Bowie Bailey wrote:
>  
>> polloxx wrote:
>>    
>>> Hi,
>>>
>>> Is the spamassassin development dead?
>>> On the website there's: 2008-06-12: SpamAssassin 3.2.5 has been released.
>>>  
>>>      
>> Not quite.  If you look at svn, you'll see this:
>>
>> spamassassin_20091103151200.tar.gz    03-Nov-2009 15:12    2.1M
>>
>> Doesn't look dead to me!  :)
>>    
>
> Hello!
> Can you please post a full URL to this archive?
> Since http://svn.apache.org/snapshots/spamassassin/ doesn't have it.
>
>
>  

The snapshots directory is automatically built and old versions are
purged. The November 3rd image is gone. Now we've got ones from the 10th
and 11th. By the time you look at it again, these might be gone and
newer ones may have replaced them.

[   ] spamassassin_20091110151200.tar.gz 10-Nov-2009 15:12  2.1M
[   ] spamassassin_20091110211200.tar.gz 10-Nov-2009 21:12  2.1M
[   ] spamassassin_20091111031200.tar.gz 11-Nov-2009 03:12  2.1M
[   ] spamassassin_20091111091200.tar.gz 11-Nov-2009 09:12  2.1M

However, if you're really just looking to gauge development activity, it
would be better to look at the list archives of all the SVN commits.

http://mail-archives.apache.org/mod_mbox/spamassassin-commits/

or, for the current month of November 2009, sorted by date:

http://mail-archives.apache.org/mod_mbox/spamassassin-commits/200911.mbox/date