Botnet 0.8 Plugin is available (FINALLY!!!)

[John Rudd]

Botnet 0.8 is up and available.  It took me a while (things have been
REALLY busy at work for the last 6 months), but it's there.

http://people.ucsc.edu/~jrudd/spamassassin/Botnet-0.8.tar


Botnet is a SpamAssassin plugin which attempts to identify hosts which
are likely to be spambot/virusbot hosts, using various DNS fingerprints
of the submitting relay.


What changed between 0.7 and 0.8:

     -- fixed unitialized $domain just before line 564
        (the ongoing bug that wasn't really affecting results, but did
give STDERR output every so often; was caused by a split() statement.)

     -- changed complex dsl clientword to .*dsl.*
        (too many variations of 'dsl' in hostnames, since it's word
bounded, the .*'s only match as far as the end of a word, so it's not
going to match _EVERYTHING_ under the sun, just every individual word
with dsl in it)

     -- changed ppp clienword to ppp(oe)?
        (for ppp over ethernet hosts)

     -- changed mx serverword to mx(pool)?
        (since it's word bounded, "mx" alone wasn't matching "mxpool")

     -- added bredband to clientwords
        (apparently a common foreign language clientword; I hadn't
noticed an false positives, so added it at someone's request ...
unfortunately, it was so long ago, that I don't remember who it was)


!!!!!!NOTE: BIG CHANGE ON SOHO CHECK!!!!!!

     -- allow HELO check in SOHO check

        (I used to be very against this, but I realized I already accept
SOHO for the sender's mail domain, which is just as much under the
control of the spammer as the HELO string.  I apply the same
requirements (mail-domain != hostname so HELO-string != hostname), but
if the HELO string does in fact resolve back to the IP address, I'll
grant the SOHO exemption.  This will probably mean you'll get fewer
Botnet hits, but hopefully it'll only be lowering false positives.)



There's not much in the way of major systemic changes here.  The code
for the HELO check had already been in there, just commented out.  So
that wasn't even a big change.  So I might be ready to go to 1.0 in the
not so distant future (and assuming I get time to stamp something a 1.0).


Thanks for everyone's patience.

[Kai Schaetzl]

John Rudd wrote on Sun, 05 Aug 2007 19:39:07 -0700:

> http://people.ucsc.edu/~jrudd/spamassassin/Botnet-0.8.tar

Hi John, just checking out your plugin the first time. I notice that it
just untars all files to the current location. It would be nice if you
could encapsulate it in a directory, so that untarring creates a directory
of the same name as the filename (without the .tar suffix, of course).
Most tarred up distributions do this and if one doesn't and there are a
lot of other files in the same directory it's a nuisance to "collect" your
files.

Kai

--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

[Ralf Hildebrandt]

* Kai Schaetzl <maillists@...>:
> John Rudd wrote on Sun, 05 Aug 2007 19:39:07 -0700:
>
> > http://people.ucsc.edu/~jrudd/spamassassin/Botnet-0.8.tar
>
> Hi John, just checking out your plugin the first time. I notice that it
> just untars all files to the current location. It would be nice if you
> could encapsulate it in a directory, so that untarring creates a directory
> of the same name as the filename (without the .tar suffix, of course).

I second this!

--
Ralf Hildebrandt (i.A. des IT-Zentrums)         Ralf.Hildebrandt@...
Charite - Universitätsmedizin Berlin            Tel.  [phone]
Gemeinsame Einrichtung von FU- und HU-Berlin    Fax.  [phone]
IT-Zentrum Standort CBF                    send no mail to plonk@...

[Brent Kennedy]

I don't mean to rain on the botnet plugin parade, but when I tried it, my
mail queue slowed to a crawl.  Mail became backed up pretty quickly with
processing taking upwards of 10 minutes a message.

Is there something in the config that I missed or it this plugin only meant
for small installations?

Thanks

-Brent

-----Original Message-----
From: Kai Schaetzl [mailto:maillists@...]
Sent: Monday, August 06, 2007 9:39 AM
To: users@...
Subject: Re: Botnet 0.8 Plugin is available (FINALLY!!!)

John Rudd wrote on Sun, 05 Aug 2007 19:39:07 -0700:

> http://people.ucsc.edu/~jrudd/spamassassin/Botnet-0.8.tar

Hi John, just checking out your plugin the first time. I notice that it just
untars all files to the current location. It would be nice if you could
encapsulate it in a directory, so that untarring creates a directory of the
same name as the filename (without the .tar suffix, of course).
Most tarred up distributions do this and if one doesn't and there are a lot
of other files in the same directory it's a nuisance to "collect" your
files.

Kai

--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

[Mark Martinec]

John,
> Botnet 0.8 is up and available.  It took me a while (things have been
> REALLY busy at work for the last 6 months), but it's there.
> http://people.ucsc.edu/~jrudd/spamassassin/Botnet-0.8.tar

Thanks, very nice.

> What changed between 0.7 and 0.8:

It seems a patch by Daniel J McDonald was left out.
See:
  http://marc.info/?t=118133681000003
  http://marc.info/?l=spamassassin-users&m=118194701009930

Here it is, applicable to Botnet-0.8:


--- Botnet.pm.ori Mon Aug  6 15:59:16 2007
+++ Botnet.pm Mon Aug  6 16:02:43 2007
@@ -711,5 +711,14 @@
         (defined $max) &&
         ($max =~ /^-?\d+$/) ) {
-      $resolver = Net::DNS::Resolver->new();
+      $resolver = Net::DNS::Resolver->new(
+               udp_timeout => 5,
+               tcp_timeout => 5,
+               retrans => 0,
+               retry => 1,
+               persistent_tcp => 0,
+               persistent_udp => 0,
+               dnsrch => 0,
+               defnames => 0,
+       );
       if ($query = $resolver->search($name, $type)) {
          # found matches
@@ -834,5 +843,14 @@
    my ($ip) = @_;
    my ($query, @answer, $rr);
-   my $resolver = Net::DNS::Resolver->new();
+   my $resolver = Net::DNS::Resolver->new(
+       udp_timeout => 5,
+       tcp_timeout => 5,
+       retrans => 0,
+       retry => 1,
+       persistent_tcp => 0,
+       persistent_udp => 0,
+       dnsrch => 0,
+       defnames => 0,
+       );
    my $name = "";
 



Mark

[Vernon Webb]

I added this and everything came back labled as SPAM

---------- Original Message -----------
From: John Rudd <jrudd@...>
To: SpamAssassin Users <users@...>
Sent: Sun, 05 Aug 2007 19:39:07 -0700
Subject: Botnet 0.8 Plugin is available (FINALLY!!!)

> Botnet 0.8 is up and available.  It took me a while (things have been
> REALLY busy at work for the last 6 months), but it's there.
>
> http://people.ucsc.edu/~jrudd/spamassassin/Botnet-0.8.tar
>
> Botnet is a SpamAssassin plugin which attempts to identify hosts which
> are likely to be spambot/virusbot hosts, using various DNS fingerprints
> of the submitting relay.
>
> What changed between 0.7 and 0.8:
>
>     -- fixed unitialized $domain just before line 564
>        (the ongoing bug that wasn't really affecting results, but did
> give STDERR output every so often; was caused by a split() statement.)
>
>     -- changed complex dsl clientword to .*dsl.*
>        (too many variations of 'dsl' in hostnames, since it's word
> bounded, the .*'s only match as far as the end of a word, so it's not
> going to match _EVERYTHING_ under the sun, just every individual word
> with dsl in it)
>
>     -- changed ppp clienword to ppp(oe)?
>        (for ppp over ethernet hosts)
>
>     -- changed mx serverword to mx(pool)?
>        (since it's word bounded, "mx" alone wasn't matching "mxpool")
>
>     -- added bredband to clientwords
>        (apparently a common foreign language clientword; I hadn't
> noticed an false positives, so added it at someone's request ...
> unfortunately, it was so long ago, that I don't remember who it was)
>
> !!!!!!NOTE: BIG CHANGE ON SOHO CHECK!!!!!!
>
>     -- allow HELO check in SOHO check
>
>        (I used to be very against this, but I realized I already accept
> SOHO for the sender's mail domain, which is just as much under the
> control of the spammer as the HELO string.  I apply the same
> requirements (mail-domain != hostname so HELO-string != hostname), but
> if the HELO string does in fact resolve back to the IP address, I'll
> grant the SOHO exemption.  This will probably mean you'll get fewer
> Botnet hits, but hopefully it'll only be lowering false positives.)
>
> There's not much in the way of major systemic changes here.  The code
> for the HELO check had already been in there, just commented out.  So
> that wasn't even a big change.  So I might be ready to go to 1.0 in the
> not so distant future (and assuming I get time to stamp something a 1.0).
>
> Thanks for everyone's patience.
>
> --
> This message has been scanned for viruses and
> dangerous content and is believed to be clean.

------- End of Original Message -------

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

[Loren Wilton]

> I added this and everything came back labled as SPAM

Did you configure it too, or just add it?

        Loren

[John Rudd]

I use it on a large installation (up to a 2 million messages per day).

Do you have at least a caching name server running on the hosts where
you do the scanning?  It does do quite a few DNS checks, which
could/would affect latency.


Brent Kennedy wrote:

> I don't mean to rain on the botnet plugin parade, but when I tried it, my
> mail queue slowed to a crawl.  Mail became backed up pretty quickly with
> processing taking upwards of 10 minutes a message.
>
> Is there something in the config that I missed or it this plugin only meant
> for small installations?
>
> Thanks
>
> -Brent
>
> -----Original Message-----
> From: Kai Schaetzl [mailto:maillists@...]
> Sent: Monday, August 06, 2007 9:39 AM
> To: users@...
> Subject: Re: Botnet 0.8 Plugin is available (FINALLY!!!)
>
> John Rudd wrote on Sun, 05 Aug 2007 19:39:07 -0700:
>
>> http://people.ucsc.edu/~jrudd/spamassassin/Botnet-0.8.tar
>
> Hi John, just checking out your plugin the first time. I notice that it just
> untars all files to the current location. It would be nice if you could
> encapsulate it in a directory, so that untarring creates a directory of the
> same name as the filename (without the .tar suffix, of course).
> Most tarred up distributions do this and if one doesn't and there are a lot
> of other files in the same directory it's a nuisance to "collect" your
> files.
>
> Kai
>
> --
> Kai Schätzl, Berlin, Germany
> Get your web at Conactive Internet Services: http://www.conactive.com
>
>
>
>

[John Rudd]

Kai Schaetzl wrote:

> John Rudd wrote on Sun, 05 Aug 2007 19:39:07 -0700:
>
>> http://people.ucsc.edu/~jrudd/spamassassin/Botnet-0.8.tar
>
> Hi John, just checking out your plugin the first time. I notice that it
> just untars all files to the current location. It would be nice if you
> could encapsulate it in a directory, so that untarring creates a directory
> of the same name as the filename (without the .tar suffix, of course).
> Most tarred up distributions do this and if one doesn't and there are a
> lot of other files in the same directory it's a nuisance to "collect" your
> files.

I'll look into re-working it.

[Vernon Webb]

the instruction basically said to through the files into the spamassassin folder and restart

---------- Original Message -----------
From: "Loren Wilton" <lwilton@...>
To: <users@...>
Sent: Mon, 6 Aug 2007 09:28:45 -0700
Subject: Re: Botnet 0.8 Plugin is available (FINALLY!!!)

> > I added this and everything came back labled as SPAM
>
> Did you configure it too, or just add it?
>
>        Loren

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

[Brent Kennedy]

Yes, I use a local DNS server.. Which does cache.

It was a little while ago, so I don't know if I still have mail logs that I
can look at.

-Brent

-----Original Message-----
From: John Rudd [mailto:jrudd@...]
Sent: Monday, August 06, 2007 1:17 PM
To: Brent Kennedy
Cc: users@...
Subject: Re: Botnet 0.8 Plugin is available (FINALLY!!!)


I use it on a large installation (up to a 2 million messages per day).

Do you have at least a caching name server running on the hosts where you do
the scanning?  It does do quite a few DNS checks, which could/would affect
latency.


Brent Kennedy wrote:

> I don't mean to rain on the botnet plugin parade, but when I tried it,
> my mail queue slowed to a crawl.  Mail became backed up pretty quickly
> with processing taking upwards of 10 minutes a message.
>
> Is there something in the config that I missed or it this plugin only
> meant for small installations?
>
> Thanks
>
> -Brent
>
> -----Original Message-----
> From: Kai Schaetzl [mailto:maillists@...]
> Sent: Monday, August 06, 2007 9:39 AM
> To: users@...
> Subject: Re: Botnet 0.8 Plugin is available (FINALLY!!!)
>
> John Rudd wrote on Sun, 05 Aug 2007 19:39:07 -0700:
>
>> http://people.ucsc.edu/~jrudd/spamassassin/Botnet-0.8.tar
>
> Hi John, just checking out your plugin the first time. I notice that
> it just untars all files to the current location. It would be nice if
> you could encapsulate it in a directory, so that untarring creates a
> directory of the same name as the filename (without the .tar suffix, of

course).

> Most tarred up distributions do this and if one doesn't and there are
> a lot of other files in the same directory it's a nuisance to
> "collect" your files.
>
> Kai
>
> --
> Kai Schätzl, Berlin, Germany
> Get your web at Conactive Internet Services: http://www.conactive.com
>
>
>
>

[Kai Schaetzl]

John Rudd wrote on Mon, 06 Aug 2007 10:18:30 -0700:

> I'll look into re-working it.

Thanks! You just need to tar it up including the enclosing directory.

Kai

--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

[John Rudd]

Kai Schaetzl wrote:
> John Rudd wrote on Mon, 06 Aug 2007 10:18:30 -0700:
>
>> I'll look into re-working it.
>
> Thanks! You just need to tar it up including the enclosing directory.
>

No, that's not all I have to do.

THAT part is the trivial part.

But in order to get there, I have to alter my development environment
for making spamassassin plugins.  That's the part I'll look in to.

[René Berber-2]

John Rudd wrote:

> Kai Schaetzl wrote:
>> John Rudd wrote on Mon, 06 Aug 2007 10:18:30 -0700:
>>
>>> I'll look into re-working it.
>>
>> Thanks! You just need to tar it up including the enclosing directory.
>>
>
> No, that's not all I have to do.
>
> THAT part is the trivial part.
>
> But in order to get there, I have to alter my development environment
> for making spamassassin plugins.  That's the part I'll look in to.

I think you misunderstood, Kai is asking for the source to be inside a directory
with the version in the name, not to use that directory with spamassassin, the
installation is the same (copying the 2 files) you just don't overwrite any
previous source you have.
--
René Berber

[John Rudd]

René Berber wrote:

> John Rudd wrote:
>
>> Kai Schaetzl wrote:
>>> John Rudd wrote on Mon, 06 Aug 2007 10:18:30 -0700:
>>>
>>>> I'll look into re-working it.
>>> Thanks! You just need to tar it up including the enclosing directory.
>>>
>> No, that's not all I have to do.
>>
>> THAT part is the trivial part.
>>
>> But in order to get there, I have to alter my development environment
>> for making spamassassin plugins.  That's the part I'll look in to.
>
> I think you misunderstood, Kai is asking for the source to be inside a directory
> with the version in the name, not to use that directory with spamassassin, the
> installation is the same (copying the 2 files) you just don't overwrite any
> previous source you have.

What I understood is that he'd like the contents of the tar file to be:


./Botnet-X.Y/Botnet.pm
./Botnet-X.Y/Botnet.cf
etc

instead of

./Botnet.pm
./Botnet.cf


And that's going to require a re-working of how/where I do the
development of the plugin.

[Kai Schaetzl]

John Rudd wrote on Mon, 06 Aug 2007 15:05:31 -0700:

> And that's going to require a re-working of how/where I do the
> development of the plugin.

No, it just requires to copy those 7 files (I didn't count again) to a
location where you collect and store your release files. For instance to
/whatever/spammassassin/plugins/distribution/Botnet-0.7
/whatever/spammassassin/plugins/distribution/Botnet-0.8
etc. and then tar Botnet-0.8 up instead of tarring only the content of
Botnet-0.8. I didn't imagine that might be so difficult.

Kai

--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

[John Hardin]

On Mon, 6 Aug 2007, John Rudd wrote:

> What I understood is that he'd like the contents of the tar file to be:
>
>
> ./Botnet-X.Y/Botnet.pm
> ./Botnet-X.Y/Botnet.cf
> etc
>
> instead of
>
> ./Botnet.pm
> ./Botnet.cf

Correct.

> And that's going to require a re-working of how/where I do the
> development of the plugin.

Why? Why not just script the release process to copy the relevant
files to a temp directory and tar that?

--
 John Hardin KA7OHZ                    http://www.impsec.org/~jhardin/
 jhardin@...    FALaholic #11174     pgpk -a jhardin@...
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
  A sword is never a killer, it is but a tool in the killer's hands.
                          -- Lucius Annaeus Seneca (Martial) 4BC-65AD
-----------------------------------------------------------------------
 Today: The 62nd anniversary of the Hiroshima bombing

[Kai Schaetzl]

John D. Hardin wrote on Mon, 6 Aug 2007 16:45:45 -0700 (PPT):

> Why not just script the release process to copy the relevant
> files to a temp directory and tar that?

or that ;-)

Kai

--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

[Joe Zitnik]

John,

>>> On 8/6/2007 at 1:18 PM, John Rudd <jrudd@...> wrote:

Kai Schaetzl wrote:

> John Rudd wrote on Sun, 05 Aug 2007 19:39:07 -0700:
>
>> http://people.ucsc.edu/~jrudd/spamassassin/Botnet-0.8.tar
>
> Hi John, just checking out your plugin the first time. I notice that it
> just untars all files to the current location. It would be nice if you
> could encapsulate it in a directory, so that untarring creates a directory
> of the same name as the filename (without the .tar suffix, of course).
> Most tarred up distributions do this and if one doesn't and there are a
> lot of other files in the same directory it's a nuisance to "collect" your
> files.

I'll look into re-working it.

 

Thanks for all your hard work on this.  Don't know if that was mentioned anywhere yet.

[Michael Schaap-2]

Hi John,

Thanks for maintaining this excellent plugin!

But I'm just wondering if ...

On 6-Aug-2007 04:39, John Rudd wrote:
>     -- changed complex dsl clientword to .*dsl.*
>        (too many variations of 'dsl' in hostnames, since it's word
> bounded, the .*'s only match as far as the end of a word, so it's not
> going to match _EVERYTHING_ under the sun, just every individual word
> with dsl in it)
>
... isn't a bit too aggressive.

$ grep -i dsl /usr/share/dict/words
Ardsley
Audsley
Beardsley
bondslave
coldslaw
Gadslid
handsled
Hildebrandslied
landsleit
landslid
landslidden
landslide
landslided
landslides
landsliding
landslip
landslips
Lindsley
Maudslay
mudslide
mudsling
mudslinger
mudslingers
mudslinging
NDSL
needsly
sudsless
swordslipper
windslab

Someone might call their mail server "mudslinger", and they'd be treated
as a DSL client.

Best regards,

 - Michael