Extending XBL to all untrusted

[RW-15]

I think it might be worth having 2 XBL tests, a high scoring test on
last-external and a lower-scoring test that goes back through the
untrusted headers.

I understand that Spamhaus doesn't recommend this, because dynamic IP
addresses can be reassigned from a spambot to another user, but I added
my own rule it does seem to work. In my mail it hits about 9% of my
spam, with zero false-positives. I suspect that part of this is down to
UK dynamic addresses being very sticky, but I ran my mailing lists
through SA for a few weeks and got 3 FPs out of ~2400.

I think it's probably worth a point or so, and essentially it's free
- all of the zen lookups get done for SBL.

[Tony Finch]

On Fri, 3 Jul 2009, RW wrote:
>
> I understand that Spamhaus doesn't recommend this, because dynamic IP
> addresses can be reassigned from a spambot to another user, but I added
> my own rule it does seem to work. In my mail it hits about 9% of my
> spam, with zero false-positives.

You will get false positives from senders that are using remote message
submission, and from some webmail users if their server puts the webmail
client IP address in the message headers.

Tony.
--
f.anthony.n.finch  <dot@...>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

[Matus UHLAR - fantomas]

> On Fri, 3 Jul 2009, RW wrote:
> > I understand that Spamhaus doesn't recommend this, because dynamic IP
> > addresses can be reassigned from a spambot to another user, but I added
> > my own rule it does seem to work. In my mail it hits about 9% of my
> > spam, with zero false-positives.

On 13.07.09 14:22, Tony Finch wrote:
> You will get false positives from senders that are using remote message
> submission, and from some webmail users if their server puts the webmail
> client IP address in the message headers.

agreed, although, some kind of authentication should be done in either case,
which should prevent the rules from hitting, but many ISPs and ESPs don';t
push auth informations to Received: headers...

--
Matus UHLAR - fantomas, uhlar@... ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Quantum mechanics: The dreams stuff is made of.

On Mon, 2009-07-13 at 17:19 +0200, Matus UHLAR - fantomas wrote:

> > On Fri, 3 Jul 2009, RW wrote:
> > > I understand that Spamhaus doesn't recommend this, because dynamic IP
> > > addresses can be reassigned from a spambot to another user, but I added
> > > my own rule it does seem to work. In my mail it hits about 9% of my
> > > spam, with zero false-positives.
>
> On 13.07.09 14:22, Tony Finch wrote:
> > You will get false positives from senders that are using remote message
> > submission, and from some webmail users if their server puts the webmail
> > client IP address in the message headers.
>
> agreed, although, some kind of authentication should be done in either case,
> which should prevent the rules from hitting, but many ISPs and ESPs don';t
> push auth informations to Received: headers...
>

Do the RFC's state that they need to?

[Ned Slider]

RW wrote:

> I think it might be worth having 2 XBL tests, a high scoring test on
> last-external and a lower-scoring test that goes back through the
> untrusted headers.
>
> I understand that Spamhaus doesn't recommend this, because dynamic IP
> addresses can be reassigned from a spambot to another user, but I added
> my own rule it does seem to work. In my mail it hits about 9% of my
> spam, with zero false-positives. I suspect that part of this is down to
> UK dynamic addresses being very sticky, but I ran my mailing lists
> through SA for a few weeks and got 3 FPs out of ~2400.
>

I do a very similar thing and see very similar results to yours.

I use zen.spamhaus to block at the smtp level and then run all headers
through sbl-xbl for a further few points. As already mentioned elsewhere
in this thread, it will occasionally fire against ham but I've only
noticed that from senders to mailing lists who originate from extremely
spammy ISPs (ie, they hit plenty of other DNSBLs too).

Where I find it particularly useful is for mail accounts forwarding from
ISP email addresses where checking of the last external IP would be
inappropriate.

> I think it's probably worth a point or so, and essentially it's free
> - all of the zen lookups get done for SBL.
>
>

[Matus UHLAR - fantomas]

> On Mon, 2009-07-13 at 17:19 +0200, Matus UHLAR - fantomas wrote:
> > > On Fri, 3 Jul 2009, RW wrote:
> > > > I understand that Spamhaus doesn't recommend this, because dynamic IP
> > > > addresses can be reassigned from a spambot to another user, but I added
> > > > my own rule it does seem to work. In my mail it hits about 9% of my
> > > > spam, with zero false-positives.
> >
> > On 13.07.09 14:22, Tony Finch wrote:
> > > You will get false positives from senders that are using remote message
> > > submission, and from some webmail users if their server puts the webmail
> > > client IP address in the message headers.
> >
> > agreed, although, some kind of authentication should be done in either case,
> > which should prevent the rules from hitting, but many ISPs and ESPs don';t
> > push auth informations to Received: headers...

On 13.07.09 16:26, richard@... wrote:
> Do the RFC's state that they need to?

yes, RFC4954 in section 7 does

--
Matus UHLAR - fantomas, uhlar@... ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Spam is for losers who can't get business any other way.

On Mon, 2009-07-13 at 18:28 +0200, Matus UHLAR - fantomas wrote:

> > On Mon, 2009-07-13 at 17:19 +0200, Matus UHLAR - fantomas wrote:
> > > > On Fri, 3 Jul 2009, RW wrote:
> > > > > I understand that Spamhaus doesn't recommend this, because dynamic IP
> > > > > addresses can be reassigned from a spambot to another user, but I added
> > > > > my own rule it does seem to work. In my mail it hits about 9% of my
> > > > > spam, with zero false-positives.
> > >
> > > On 13.07.09 14:22, Tony Finch wrote:
> > > > You will get false positives from senders that are using remote message
> > > > submission, and from some webmail users if their server puts the webmail
> > > > client IP address in the message headers.
> > >
> > > agreed, although, some kind of authentication should be done in either case,
> > > which should prevent the rules from hitting, but many ISPs and ESPs don';t
> > > push auth informations to Received: headers...
>
> On 13.07.09 16:26, richard@... wrote:
> > Do the RFC's state that they need to?
>
> yes, RFC4954 in section 7 does
>

Where - I don't see it say it needs to "push auth informations to
Recieved: Headers";


7.  Additional Requirements on Servers


   As described in Section 4.4 of [SMTP], an SMTP server that receives a
   message for delivery or further processing MUST insert the
   "Received:" header field at the beginning of the message content.
   This document places additional requirements on the content of a
   generated "Received:" header field.  Upon successful authentication,
   a server SHOULD use the "ESMTPA" or the "ESMTPSA" [SMTP-TT] (when
   appropriate) keyword in the "with" clause of the Received header
   field.

Am I missing what you are saying here?

On Mon, 2009-07-13 at 17:38 +0100, richard@... wrote:

> On Mon, 2009-07-13 at 18:28 +0200, Matus UHLAR - fantomas wrote:
> > > On Mon, 2009-07-13 at 17:19 +0200, Matus UHLAR - fantomas wrote:
> > > > > On Fri, 3 Jul 2009, RW wrote:
> > > > > > I understand that Spamhaus doesn't recommend this, because dynamic IP
> > > > > > addresses can be reassigned from a spambot to another user, but I added
> > > > > > my own rule it does seem to work. In my mail it hits about 9% of my
> > > > > > spam, with zero false-positives.
> > > >
> > > > On 13.07.09 14:22, Tony Finch wrote:
> > > > > You will get false positives from senders that are using remote message
> > > > > submission, and from some webmail users if their server puts the webmail
> > > > > client IP address in the message headers.
> > > >
> > > > agreed, although, some kind of authentication should be done in either case,
> > > > which should prevent the rules from hitting, but many ISPs and ESPs don';t
> > > > push auth informations to Received: headers...
> >
> > On 13.07.09 16:26, richard@... wrote:
> > > Do the RFC's state that they need to?
> >
> > yes, RFC4954 in section 7 does
> >
> Where - I don't see it say it needs to "push auth informations to
> Recieved: Headers";
>
>
> 7.  Additional Requirements on Servers
>
>
>    As described in Section 4.4 of [SMTP], an SMTP server that receives a
>    message for delivery or further processing MUST insert the
>    "Received:" header field at the beginning of the message content.
>    This document places additional requirements on the content of a
>    generated "Received:" header field.  Upon successful authentication,
>    a server SHOULD use the "ESMTPA" or the "ESMTPSA" [SMTP-TT] (when
>    appropriate) keyword in the "with" clause of the Received header
>    field.
>
> Am I missing what you are saying here?
>

Got it! Now I understand where you are coming from;
Received: from [192.168.1.56] (rubiks [192.168.1.56]) by
 mail1.buzzhost.co.uk (XmasTree)

AND HERE IT COMES.....

with ESMTPA

....
id E0C42AC0BE for

Now it makes sense.

[Justin Mason]

On Fri, Jul 3, 2009 at 22:43, RW<rwmaillists@...> wrote:

>
> I think it might be worth having 2 XBL tests, a high scoring test on
> last-external and a lower-scoring test that goes back through the
> untrusted headers.
>
> I understand that Spamhaus doesn't recommend this, because dynamic IP
> addresses can be reassigned from a spambot to another user, but I added
> my own rule it does seem to work. In my mail it hits about 9% of my
> spam, with zero false-positives. I suspect that part of this is down to
> UK dynamic addresses being very sticky, but I ran my mailing lists
> through SA for a few weeks and got 3 FPs out of ~2400.
>
> I think it's probably worth a point or so, and essentially it's free
> - all of the zen lookups get done for SBL.

we used to do it this way, but the FPs are (surprisingly) high due to
dynamic-address-pool churn.

compare:
OVERALL%   SPAM%     HAM%     S/O    RANK   SCORE  NAME
 5.100  10.1740   0.0200    0.998   0.65    0.01  T_RCVD_IN_XBL  (with
trusted-networks)
 5.417  10.6074   0.2203    0.980   0.18    0.00  RCVD_IN_XBL  (with all nets)

I'll forward on the old mail for hysterical raisins.

--j.

[McDonald, Dan]

On Mon, 2009-07-13 at 17:38 +0100, richard@... wrote:

> On Mon, 2009-07-13 at 18:28 +0200, Matus UHLAR - fantomas wrote:
> > On 13.07.09 16:26, richard@... wrote:
> > > Do the RFC's state that they need to?
> >
> > yes, RFC4954 in section 7 does
> >
> Where - I don't see it say it needs to "push auth informations to
> Recieved: Headers";
>
>
> 7.  Additional Requirements on Servers
>
> Upon successful authentication,
>    a server SHOULD use the "ESMTPA" or the "ESMTPSA" [SMTP-TT] (when
>    appropriate) keyword in the "with" clause of the Received header
>    field.

It's a SHOULD, not a MUST, but the intent is clear.


--
Daniel J McDonald, CCIE # 2495, CISSP # 78281, CNX
www.austinenergy.com

[Rob McEwen]

I agree so strongly about not checking against all IPs in the header
that I'll probably turn down business from large anti-spam vendors who
cannot guarantee in writing that ivmSIP and ivmSIP/24 will ONLY be
checked against the actual sending IP. If this means I lose 4-5 figures
in annual revenue from future vendors, so be it. (and I don't think any
of my current largest subscribers are doing this.)

There is a better system. Work to find ways to better know which headers
are forwarders, ignore them, and grab the original sender's 'mta' IP
from THAT received header. (not IP the workstation which originated the
e-mail, but the mail server IP that officially sent the message on
behalf of the sender, but before any other forwarding).

This "surgeon's scalpel" approach is not always as easy as the
alternative sledgehammer approach, but it is worth the effort. Certain
large anti-spam appliance vendors have no excuse for not making this
extra effort... and I've seen some egregious FPs (for example...
hand-typed messages from an attorney to their client, sent from an IP
which doesn't ever send spam) recently caused by such appliances which
check all IPs in the header against blacklists.

--
Rob McEwen
http://dnsbl.invaluement.com/
rob@...
[phone]

[RW-15]

On Mon, 13 Jul 2009 17:21:36 +0100
Ned Slider <ned@...> wrote:

> I do a very similar thing and see very similar results to yours.
>
> I use zen.spamhaus to block at the smtp level and then run all
> headers through sbl-xbl for a further few points. As already
> mentioned elsewhere in this thread, it will occasionally fire against
> ham but I've only noticed that from senders to mailing lists who
> originate from extremely spammy ISPs (ie, they hit plenty of other
> DNSBLs too).
>

On Mon, 13 Jul 2009 17:48:17 +0100
Justin Mason <jm@...> wrote:

> we used to do it this way, but the FPs are (surprisingly) high due to
> dynamic-address-pool churn.

That kind of thing doesn't happen much with UK DSL.

I notice Ned Slider has a .co.uk address, so I think it probably does
matter where your mail comes from.


> compare:
> OVERALL%   SPAM%     HAM%     S/O    RANK   SCORE  NAME
>  5.100  10.1740   0.0200    0.998   0.65    0.01  T_RCVD_IN_XBL  (with
> trusted-networks)
>  5.417  10.6074   0.2203    0.980   0.18    0.00  RCVD_IN_XBL  (with
> all nets)


Even on those figures, I still think it's worth scoring at the 0.5 to
1 point level.