Botnet 0.8 Plugin is available (FINALLY!!!)

[John Rudd]

Henrik Krohns wrote:

> On Wed, Aug 08, 2007 at 10:51:05PM +0200, Michael Schaap wrote:
>> Hi John,
>>
>> Thanks for maintaining this excellent plugin!
>>
>> But I'm just wondering if ...
>>
>> On 6-Aug-2007 04:39, John Rudd wrote:
>>>     -- changed complex dsl clientword to .*dsl.*
>>>        (too many variations of 'dsl' in hostnames, since it's word
>>> bounded, the .*'s only match as far as the end of a word, so it's not
>>> going to match _EVERYTHING_ under the sun, just every individual word with
>>> dsl in it)
>>>
>> ... isn't a bit too aggressive.
>>
>> $ grep -i dsl /usr/share/dict/words
>
>> Someone might call their mail server "mudslinger", and they'd be treated as
>> a DSL client.
>
> While that's unlikely, judging my small sample of logs the dsl string is
> always in the front or end. So "[a-z]?dsl.*? .*?dsl" should work just as
> fine. Maybe someone can grep a bigger log.
>

I had a very complex expression for it at some point, but I kept
noticing off variation, where "dsl" was in the middle of the word, or in
a few cases it was at the start of the word (dsla I think was the one I
saw).  So I decided to just go to what I have now, instead of trying to
conceive of every possible variation.

[hanz]

> Botnet 0.8 is up and available.  It took me a while (things have been
> REALLY busy at work for the last 6 months), but it's there.
> http://people.ucsc.edu/~jrudd/spamassassin/Botnet-0.8.tar

 ooking at the debug code, I notice that botnet,pm version 0.8 is only checking the last server IP and not all  IPs in the path.

example path the mail went thru:
[32635] dbg: dns: IPs found: full-external: 128.6.72.72, 127.0.0.1, 127.0.0.1, 128.6.31.86, 128.6.72.254, 127.0.0.1, 127.0.0.1, 128.6.31.85, 59.144.126.12, 59.144.126.12 untrusted: 128.6.72.72, 128.6.31.86, 128.6.72.254, 128.6.31.85, 59.144.126.12 originating:

example debug code
[32635] dbg: Botnet: starting
[32635] dbg: Botnet: no trusted relays
[32635] dbg: Botnet: get_relay good RDNS
[32635] dbg: Botnet: IP is '128.6.72.72'
[32635] dbg: Botnet: RDNS is 'gehenna.rutgers.edu'
[32635] dbg: Botnet: HELO is 'gehenna10.rutgers.edu'
[32635] dbg: Botnet: sender ''
[32635] dbg: Botnet: miss (none)


I believe if botnet.pm is checking all the path  the mail went thru like how dnsbl is used, botnet will get more accurate. I could be wrong on this but for the shake of fighting spam,I hope I am right and you could find a way to get this to work.

Here is a sample of the bad email which may or may not be from  botnet source.
       
        http://www.cs.rutgers.edu/~makmur/forjrudd.txt


Hope I give enough details.

Thanks again for making fighting spam email easier.

Hanz

[Jason Bertoch [Electronet]]

On Friday, September 28, 2007 4:06 PM hanz wrote:

>
> looking at the debug code, I notice that botnet,pm version 0.8 is only
> checking the last server IP and not all IPs in the path.
>

A botnet sends mail directly from the infected source, rather than relay it via
the ISP's mail server.  Any previous received headers would be forged so there's
no point in checking them.


Jason

[hanz]

Thanks for confirming how botnet works.  This is exactly the problem!

Botnet.pm is only checking the LAST IP and not the FIRST in the example email.

The first IP in the list is a definite botnet source but botnet.pm does not detect this as a botnet email.

hanz

Jason Bertoch [Electronet] wrote:

On Friday, September 28, 2007 4:06 PM hanz wrote:

>
> looking at the debug code, I notice that botnet,pm version 0.8 is only
> checking the last server IP and not all IPs in the path.
>

A botnet sends mail directly from the infected source, rather than relay it via
the ISP's mail server.  Any previous received headers would be forged so there's
no point in checking them.


Jason

[James Pratt-2]

>> -----Original Message-----
>> From: hanz [mailto:makmur@...]
>> Sent: Friday, September 28, 2007 4:31 PM
>> To: users@...
>> Subject: RE: Botnet 0.8 Plugin is available (FINALLY!!!)
>>
>>
>> Thanks for confirming how botnet works.  This is exactly the problem!
>>
>> Botnet.pm is only checking the LAST IP and not the FIRST in the
>> example
>> email.
>>
>> The first IP in the list is a definite botnet source but botnet.pm
>> does not
>> detect this as a botnet email.
>>
>> hanz
>>
>>
>> Jason Bertoch [Electronet] wrote:
>> >
>> > On Friday, September 28, 2007 4:06 PM hanz wrote:
>> >
>> >>
>> >> looking at the debug code, I notice that botnet,pm version 0.8 is
>> only
>> >> checking the last server IP and not all IPs in the path.
>> >>
>> >
>> > A botnet sends mail directly from the infected source, rather than
>> relay
>> > it via
>> > the ISP's mail server.  Any previous received headers would be
>> forged so
>> > there's
>> > no point in checking them.
>> >
>> >
>> > Jason
>> >
>> >
>> >
>>
>> --
>> View this message in context:

http://www.nabble.com/Botnet-0.8-Plugin-
>> is-available-%28FINALLY%21%21%21%29-tf4221965.html#a12948014
>> Sent from the SpamAssassin - Users mailing list archive at
Nabble.com.

Yes, but in most cases, it is the LAST ip that is part of the botnet
(ie, it connected to your server LAST.) - checking all of the IP's I
believe would be counterproductive and just add to false-positives. Btw
- it appears you are using botnet in the wrong place if this email only
traversed Rutgers.edu servers, minus the first bot-net IP - it should be
running on your internet-facing relay, not internal relays... that's
just weird IMO...

Regards,
jamie  

[René Berber-2]

Hanz wrote:

> [L]ooking at the debug code, I notice that botnet,pm version 0.8 is only
> checking the last server IP and not all  IPs in the path.

Which is correct, if you checked the originating address in the list then all
messages sent from home (through your ISP or work) would be marked as dynamic,
and therefore possible bot, which is not true.

[snip]
> I believe if botnet.pm is checking all the path  the mail went thru like how
> dnsbl is used,

If a dnsbl does that, it is wrong and worthless.  What dnsbl checks mail
messages anyway? they respond to queries about IPs, not message routes.

> botnet will get more accurate. I could be wrong on this but
> for the shake of fighting spam,I hope I am right and you could find a way to
> get this to work.

You are wrong.
--
René Berber

[John Rudd]

hanz wrote:

>
> I believe if botnet.pm is checking all the path  the mail went thru like how
> dnsbl is used, botnet will get more accurate.

No, it would throw a lot more false-positives.  Every end user
(corporate, home, etc.) on a dynamic IP address would suddenly get their
email flagged by botnet, because the originating host matches the botnet
conditions.


Consider this senario:

    a) user on dynamic IP sends email to their ISP's mail server
    b) ISP's mail server submits message to your mail server

In your suggested processing, this would generate a false positive: the
message would be marked as a potential botnet even though the message
was handled in a legitimate manner (message went out through the ISP's
mail server instead of coming _directly_ from the dynamic host).

Botnet specifically only tries to look at the host that submitted the
message to  your environment because of this.


So you might ask "what about ISPs that aren't policing their network, to
keep botnets from relaying through them?"   Those can much more easily
be targeted by DSBLs than trying to DSBL every little dynamic host
(though, pbl.spamhaus.org seems to be trying to do that).  In one way,
Botnet tries to encourage a bottle-neck of mail traffic through each
provider's mail server, partially to make it easier to manage all of the
end points recipient postmasters have to deal with.


So, basically, I wont be changing botnet to do what you're asking for.
I consider it to be a rather bad idea.  Though, you could fork the code,
call it something else, and make your own that behaves however you want.

[Jerry Durand]

At 02:31 PM 9/28/2007, John Rudd wrote:
>Consider this senario:
>
>    a) user on dynamic IP sends email to their ISP's mail server
>    b) ISP's mail server submits message to your mail server
>
>In your suggested processing, this would generate a false positive:
>the message would be marked as a potential botnet even though the
>message was handled in a legitimate manner (message went out through
>the ISP's mail server instead of coming _directly_ from the dynamic host).

Our mail server is on a dynamic business line, so we send through our
ISPs AUTH port (and have this listed in SPF).  We still get bounced
mail from some servers that are scanning all the headers against
things like the Zen list.  For a while, Internic was bouncing mailing
list digests that had posts from anyone with a dynamic address, seems
they were scanning the body of the message, too!


--
Jerry Durand, Durand Interstellar, Inc.  www.interstellar.com
tel: [phone], USA toll free: 1 866 356-3886
Skype:  jerrydurand

[Jari Fredriksson]

> Thanks for confirming how botnet works.  This is exactly
> the problem!
>
> Botnet.pm is only checking the LAST IP and not the FIRST
> in the example email.
>
> The first IP in the list is a definite botnet source but
> botnet.pm does not detect this as a botnet email.
>
> hanz
>
>

As far as I have understood it Botnet checks the first IP not being in your "trusted networks".

Works for me fine.

[Matus UHLAR - fantomas]

> At 02:31 PM 9/28/2007, John Rudd wrote:
> >Consider this senario:
> >
> >   a) user on dynamic IP sends email to their ISP's mail server
> >   b) ISP's mail server submits message to your mail server
> >
> >In your suggested processing, this would generate a false positive:
> >the message would be marked as a potential botnet even though the
> >message was handled in a legitimate manner (message went out through
> >the ISP's mail server instead of coming _directly_ from the dynamic host).

On 28.09.07 14:52, Jerry Durand wrote:
> Our mail server is on a dynamic business line, so we send through our
> ISPs AUTH port (and have this listed in SPF).  We still get bounced
> mail from some servers that are scanning all the headers against
> things like the Zen list.  For a while, Internic was bouncing mailing
> list digests that had posts from anyone with a dynamic address, seems
> they were scanning the body of the message, too!

Does your provider puth AUTH information into message headers? If so, those
servers are certainly broken. ZEN containt IPs like dynamic that are not
suppoded to send mail directly, but through their SMTP server. (they are in
PBL which is subset of ZEN). The header check should stop at such headers.
SA does do that

--
Matus UHLAR - fantomas, uhlar@... ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
- Holmes, what kind of school did you study to be a detective?
- Elementary, Watson.

[Matus UHLAR - fantomas]

> > Thanks for confirming how botnet works.  This is exactly
> > the problem!
> >
> > Botnet.pm is only checking the LAST IP and not the FIRST
> > in the example email.
> >
> > The first IP in the list is a definite botnet source but
> > botnet.pm does not detect this as a botnet email.

On 29.09.07 02:31, Jari Fredriksson wrote:
> As far as I have understood it Botnet checks the first IP not being in
> your "trusted networks".

botnet probably does such checks based on trusted_networks and
internal_networks settings: doesn't check IP in trusted_networks, but
continues on next IP when current one is in internal_networks
(where you should put your own mail forwarders and backups, altogether with
trusted_networks)
--
Matus UHLAR - fantomas, uhlar@... ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
- Have you got anything without Spam in it?
- Well, there's Spam egg sausage and Spam, that's not got much Spam in it.

[Loren Wilton]

>> As far as I have understood it Botnet checks the first IP not being in
>> your "trusted networks".
>
> botnet probably does such checks based on trusted_networks and
> internal_networks settings: doesn't check IP in trusted_networks, but
> continues on next IP when current one is in internal_networks
> (where you should put your own mail forwarders and backups, altogether
> with
> trusted_networks)

Probably uses first_untrusted.  So indeed, if one does not have
trusted_networks set up right, then Botnet will probably draw the wrong
conclusions.

        Loren

[Jerry Durand]

On Mon, 2007-10-01 at 10:44 +0200, Matus UHLAR - fantomas wrote:

> Does your provider puth AUTH information into message headers? If so,
> those
> servers are certainly broken. ZEN containt IPs like dynamic that are
> not
> suppoded to send mail directly, but through their SMTP server. (they
> are in
> PBL which is subset of ZEN). The header check should stop at such
> headers.
> SA does do that
>

It should, but check the headers on this message (since you'll get a
private copy from the reply-to-all).

If I have something set wrong, let me know as I have to move the server
over to our new Ubuntu system and that would be a good time to fix it.

--
Jerry Durand, Durand Interstellar, Inc.
Los Gatos, California, USA, www.interstellar.com
tel:  [phone], Skype:  jerrydurand

[Jerry Durand]

Well that didn't totally work, I received a 550 from fantomas.sk.  If anyone is willing to check my headers off-list, contact me with a private e-mail.

I'd like to make sure I have the new system set up right before I add some more domains to it.

Thanks.


On Mon, 2007-10-01 at 10:44 +0200, Matus UHLAR - fantomas wrote:

> Does your provider puth AUTH information into message headers? If so,
> those
> servers are certainly broken. ZEN containt IPs like dynamic that are
> not
> suppoded to send mail directly, but through their SMTP server. (they
> are in
> PBL which is subset of ZEN). The header check should stop at such
> headers.
> SA does do that
>

It should, but check the headers on this message (since you'll get a
private copy from the reply-to-all).

If I have something set wrong, let me know as I have to move the server
over to our new Ubuntu system and that would be a good time to fix it.

--
Jerry Durand, Durand Interstellar, Inc.
Los Gatos, California, USA, www.interstellar.com
tel:  [phone], Skype:  jerrydurand

[Daryl C. W. O'Shea]

Jerry Durand wrote:

> On Mon, 2007-10-01 at 10:44 +0200, Matus UHLAR - fantomas wrote:
>
>> Does your provider puth AUTH information into message headers? If so,
>> those
>> servers are certainly broken. ZEN containt IPs like dynamic that are
>> not
>> suppoded to send mail directly, but through their SMTP server. (they
>> are in
>> PBL which is subset of ZEN). The header check should stop at such
>> headers.
>> SA does do that
>>
>
> It should, but check the headers on this message (since you'll get a
> private copy from the reply-to-all).
>
> If I have something set wrong, let me know as I have to move the server
> over to our new Ubuntu system and that would be a good time to fix it.

Your provider does include an auth token... ESMTPA.

Daryl


Received: from smtp.interstellar.com ([71.116.65.33])
  by vms044.mailsrvcs.net (Sun Java System Messaging Server 6.2-6.01
(built Apr
  3 2006)) with ESMTPA id <0JP800G9VTMD4SC0@...> for
  users@...; Mon, 01 Oct 2007 12:23:03 -0500 (CDT)

[hanz]

Thanks for the explanation and quick replies from everyone. I was definitely wrong in my assumption on how botnet works.

I think I understand the issue now and my problem can easily be fixed by skipping the IPs or my internal forwarders.

That is  adding the following to botnet.cf fixed it.

botnet_skip_ip ^128\.6\.72\.254$
botnet_skip_ip ^128\.6\.72\.72$
botnet_skip_ip ^128\.6\.31\.85$
botnet_skip_ip ^128\.6\.31\.86$

Hanz

John Rudd wrote:

hanz wrote:

>
> I believe if botnet.pm is checking all the path  the mail went thru like how
> dnsbl is used, botnet will get more accurate.

No, it would throw a lot more false-positives.  Every end user
(corporate, home, etc.) on a dynamic IP address would suddenly get their
email flagged by botnet, because the originating host matches the botnet
conditions.


Consider this senario:

    a) user on dynamic IP sends email to their ISP's mail server
    b) ISP's mail server submits message to your mail server

In your suggested processing, this would generate a false positive: the
message would be marked as a potential botnet even though the message
was handled in a legitimate manner (message went out through the ISP's
mail server instead of coming _directly_ from the dynamic host).

Botnet specifically only tries to look at the host that submitted the
message to  your environment because of this.


So you might ask "what about ISPs that aren't policing their network, to
keep botnets from relaying through them?"   Those can much more easily
be targeted by DSBLs than trying to DSBL every little dynamic host
(though, pbl.spamhaus.org seems to be trying to do that).  In one way,
Botnet tries to encourage a bottle-neck of mail traffic through each
provider's mail server, partially to make it easier to manage all of the
end points recipient postmasters have to deal with.


So, basically, I wont be changing botnet to do what you're asking for.
I consider it to be a rather bad idea.  Though, you could fork the code,
call it something else, and make your own that behaves however you want.

[Daryl C. W. O'Shea]

hanz wrote:

> Thanks for the explanation and quick replies from everyone. I was definitely
> wrong in my assumption on how botnet works.
>
> I think I understand the issue now and my problem can easily be fixed by
> skipping the IPs or my internal forwarders.
>
> That is  adding the following to botnet.cf fixed it.
>
> botnet_skip_ip ^128\.6\.72\.254$
> botnet_skip_ip ^128\.6\.72\.72$
> botnet_skip_ip ^128\.6\.31\.85$
> botnet_skip_ip ^128\.6\.31\.86$

It sounds like you haven't configured SpamAssassin for use on your
network if the above config is necessary to make the Botnet plugin work
(assuming the Botnet plugin DTRT in regards to what IPs it checks).

You should have the IPs of your internal forwarders included in your
trusted and internal network of your SpamAssassin config, along with any
other appropriate IPs.


Daryl

[John Rudd]

Loren Wilton wrote:

>>> As far as I have understood it Botnet checks the first IP not being in
>>> your "trusted networks".
>>
>> botnet probably does such checks based on trusted_networks and
>> internal_networks settings: doesn't check IP in trusted_networks, but
>> continues on next IP when current one is in internal_networks
>> (where you should put your own mail forwarders and backups, altogether
>> with
>> trusted_networks)
>
> Probably uses first_untrusted.  So indeed, if one does not have
> trusted_networks set up right, then Botnet will probably draw the wrong
> conclusions.

Actually, as currently written, Botnet does a few things of its own to
figure out which entry is the one to check.  And it has a few config
options to modify that behavior.

I might change that at some point (to use first_untrusted by default
might be best).  But, for now, it does its own thing in trying to figure
it out.

[hanz]

Actually, I have them set correctly. I receive email straight from the internet and most mail do not go thru internal forwarders. I have no trusted machines outside of my department as many machines outside the department is not under my control and a few of them is known to have generated spam and became botnet. Putting these forwarders as  trusted in spamassassin is not a good idea. Skipping them for botnet checking makes more sense as it will make botnet checking more accurate.

The issue is started as some other departmental servers stopped checking spam for internally forwarded email to reduce their load and some of my users forward their email from other departmental servers.  

Hanz

Daryl C. W. O'Shea wrote:

hanz wrote:
> Thanks for the explanation and quick replies from everyone. I was definitely
> wrong in my assumption on how botnet works.
>
> I think I understand the issue now and my problem can easily be fixed by
> skipping the IPs or my internal forwarders.
>
> That is  adding the following to botnet.cf fixed it.
>
> botnet_skip_ip ^128\.6\.72\.254$
> botnet_skip_ip ^128\.6\.72\.72$
> botnet_skip_ip ^128\.6\.31\.85$
> botnet_skip_ip ^128\.6\.31\.86$

It sounds like you haven't configured SpamAssassin for use on your
network if the above config is necessary to make the Botnet plugin work
(assuming the Botnet plugin DTRT in regards to what IPs it checks).

You should have the IPs of your internal forwarders included in your
trusted and internal network of your SpamAssassin config, along with any
other appropriate IPs.


Daryl