blocking on failure of reverse_host_lookup

[Ian P. Christian]

Hi all,

I though I'd seen people post on the list about blocking outright on
reverse_host_lookup - however having implemented this yesterday, it
seems like it quite possibly did block the occasional email.  Before
yesterday, it was just used as part of a scoring system.

What are peoples thoughts on this?

--
Blog: http://pookey.co.uk/blog
Follow me on twitter: http://twitter.com/ipchristian

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Ian P. Christian wrote:

> I though I'd seen people post on the list about blocking outright on
> reverse_host_lookup

What you most likely saw was the same single person repeatedly spouting
it as the solution to all spam problems in just about every thread
vaguely related to spam or dns.

> - however having implemented this yesterday, it
> seems like it quite possibly did block the occasional email.  Before
> yesterday, it was just used as part of a scoring system.
>
> What are peoples thoughts on this?

You'll lose legitimate email if you outright block on it. It *is* an
indicator of spaminess though so it's worth scoring on. The default
score that is given by SpamAssassin for RDNS_NONE is a measily 0.1, but
then they *do* run automated tests against large amounts of ham and spam
in order to come up with carefully calculated scores, so I guess there's
something in that low score. I personally pushed the scoring up from 0.1
to 0.7 though.


--
Mike Cardwell - IT Consultant and LAMP developer
Cardwell IT Ltd. (UK Reg'd Company #06920226) http://cardwellit.com/

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[vitas1]

...

> What are peoples thoughts on this?
>
...

Well I'm using the configuration that someone clever has suggested in this
list :-)

At Exim level I check the following things for connecting host: does it
have valid rDNS, does it offer valid HELO, is it listed in DNSBL (you may
also check for SPF etc).

If one of these checks is failed then message is greylisted. If it then
passes greylist it will be checked by SpamAssassin.

Otherwise host is assumed to be "valid" and a message is accepted.

So now I have a severely limited amount of spam from "valid" hosts (almost
always they are connecting from the same static IP's so it is easy to
blacklist them locally). But I've greatly decreased the amount of FP's!

WBR,

Vitas.
--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Joan juvanteny]

2009/7/2 Mike Cardwell <exim-users@...>:

> Ian P. Christian wrote:
>
>> I though I'd seen people post on the list about blocking outright on
>> reverse_host_lookup
>
> What you most likely saw was the same single person repeatedly spouting
> it as the solution to all spam problems in just about every thread
> vaguely related to spam or dns.
>
>> - however having implemented this yesterday, it
>> seems like it quite possibly did block the occasional email.  Before
>> yesterday, it was just used as part of a scoring system.
>>
>> What are peoples thoughts on this?
>

No solution is perfect, but rdns is quite helpful, this is more or
less what I have:
I set up rdns checks enabled by default, the ip addresses that doesn't
have a valid rdns get a 450 temporary error.
I created a big whitelist for the neighbour countries grouping the ips
in big ranges.
For the unproperly configured domains, I created a web page with
information on how to get whitelisted, just sending a mail to
postmaster.
I see that it isn't the perfect solution, but in the time I've had it
implemented there hasn't been many problems (at least that we are
aware of)

Another usage of the rdns would be the one explained in this thread,
applying harder spam controls to the sources that doesn't have a
properly configured rdns.
http://www.gossamer-threads.com/lists/exim/users/83353

> ## Exim details at http://www.exim.org/
> ## Please use the Wiki with this list - http://wiki.exim.org/
>

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Joan wrote:

> I set up rdns checks enabled by default, the ip addresses that doesn't
> have a valid rdns get a 450 temporary error.

Can I ask why you respond with a temporary error rather than a permanent
  error message? By doing that it just seems that when you block a
legitimate email, the sender doesn't find out until much later... At
least with an immediate permanent rejection the sender knows that their
email didn't get through as they should get a bounce straight away...

> I created a big whitelist for the neighbour countries grouping the ips
> in big ranges.

FYI, if you want to apply "country" based rules to incoming connections,
there are dnsbl's that map IPs to countries. Eg, to accept mail from
hosts in the UK and France:

accept dnslists = uk.countries.nerd.dk : fr.countries.nerd.dk

countries.nerd.dk also let you download their full data set in rbldns
format (and other formats I think) so you can create your own local dnsbl.

--
Mike Cardwell - IT Consultant and LAMP developer
Cardwell IT Ltd. (UK Reg'd Company #06920226) http://cardwellit.com/

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Andrew-240]

Mike Cardwell wrote:
> Joan wrote:
>

--snip--


> FYI, if you want to apply "country" based rules to incoming connections,
> there are dnsbl's that map IPs to countries. Eg, to accept mail from
> hosts in the UK and France:
>
> accept dnslists = uk.countries.nerd.dk : fr.countries.nerd.dk
>
> countries.nerd.dk also let you download their full data set in rbldns
> format (and other formats I think) so you can create your own local dnsbl.
>


There is also another option using geoip tools using the maxmind.com
database.
http://www.maxmind.com/app/geoip_country
Completely free. one download a month, so no thousands of DNS queries
and you no longer have to maintain a list of country IPs.

I am using the perl API and have been for at least 4 years with no
issues as all :)

HTH
cya
Andrew

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Joan juvanteny]

>> FYI, if you want to apply "country" based rules to incoming connections,
>> there are dnsbl's that map IPs to countries. Eg, to accept mail from hosts
>> in the UK and France:
>>
>> accept dnslists = uk.countries.nerd.dk : fr.countries.nerd.dk
>>
>> countries.nerd.dk also let you download their full data set in rbldns
>> format (and other formats I think) so you can create your own local dnsbl.
>>
>
>
> There is also another option using geoip tools using the maxmind.com
> database.
> http://www.maxmind.com/app/geoip_country
> Completely free. one download a month, so no thousands of DNS queries and
> you no longer have to maintain a list of country IPs.
>
> I am using the perl API and have been for at least 4 years with no issues as
> all :)

I wrote some little script that extracts the data from
http://phpweby.com/software/ip2country and groups it into /24 or
bigger dumping it into a file.

I assumed that reading a file would be the fastest way to have this
list, am I wrong? I'd get a better performance + skip another API that
might fail.


> Can I ask why you respond with a temporary error rather than a permanent
> error message? By doing that it just seems that when you block a
> legitimate email, the sender doesn't find out until much later... At
> least with an immediate permanent rejection the sender knows that their
> email didn't get through as they should get a bounce straight away...
The 450 was just in case something went wrong not to rejecte the mail,
my intention was to test it for some more time, and afterwards do:
.- change it into a 550
.- change the retry/reject times for the mail to something faster


The solution is not ideal, but it works quite well, and after seeing
that some big domains are doing strict rdns checking (wanadoo.fr for
example), I just go with them.

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[W B Hacker]

Ian P. Christian wrote:
> Hi all,
>
> I though I'd seen people post on the list about blocking outright on
> reverse_host_lookup - however having implemented this yesterday, it
> seems like it quite possibly did block the occasional email.  Before
> yesterday, it was just used as part of a scoring system.
>
> What are peoples thoughts on this?
>

Having a PTR RR for a public-facing server (smtp or otherwise) has been an RFC
requirement for a Very Long Time. Finding, reading, and understanding that very
public information is easy.

But in line with being 'generous with what we accept' most of us did not insist
on it.

Zombots strain even the most generous among us, so we've had to 'work to rule'.

Up until about 12 - 18 months ago, there were a few false positives - very few.

But some of them were 'important' false-positives.

For example - the odd major provider whose 'outbound pool' servers did not have
appropriate DNS credentials that associated them with that provider's identity
and inbound servers for DSN or normal traffic. NetSol's hired third-party hosts
for low-cost and 'vanity' domains were at one time among these.

For a time, those which actually presented traffic to a given server required
whitelisting.

But most that we once had to whitelist have since seen the advantage of strictly
applying the relevant RFC's, have corrected their faux pas, and no longer need
to be whitelisted, so that list (here)) is down to fewer than two-dozen entries
globally, of which less than half seem to be still in need.

Likewise, more and more major providers want to find PTR RR for inbound traffic
to their servers - something distinctly hard for a Zombot to acquire, and
'risky' for professional spammers to register, as it leaves at bit more of a
backtrace trail than most dare deal with.

Criminals defending the 'rights' of Zombot's will tell you otherwise, of course.
They have serious income streams to defend.

There are also grumblings from the odd amateur hobbyist who would like to run an
MTA on dynamic or similar IP where the records cannot be set up. The honest
among those recognize the rules and ask for exemptions or wish they were
changed. The dishonest pretend they do not exist, sometimes rather stridently.

Do your own reading.

But if zombification of smtp is to be reduced, those folks will either have to
use a their provider's smarthost, or rent at least a US$5 / month virtual host
that DOES have proper credentials. No shortage of either.

I sympathize with their plight, but not enough to once again open the door to
millions of zombots.

YMMV, so a 'point score' may be easier to admin for a while yet in some
environments.

Bill

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Ian P. Christian]

2009/7/2 Ian P. Christian <pookey@...>:
> Hi all,
>
> I though I'd seen people post on the list about blocking outright on
> reverse_host_lookup - however having implemented this yesterday, it
> seems like it quite possibly did block the occasional email.  Before
> yesterday, it was just used as part of a scoring system.

Whislt on the topic... how can I check that a host has a reverse
record, without doing the full forward checking that
reverse_host_lookup does?  I can at least block outright hosts that
don't have a reverse...

--
Blog: http://pookey.co.uk/blog
Follow me on twitter: http://twitter.com/ipchristian

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Joan juvanteny]

2009/7/2 Ian P. Christian <pookey@...>:

> 2009/7/2 Ian P. Christian <pookey@...>:
>> Hi all,
>>
>> I though I'd seen people post on the list about blocking outright on
>> reverse_host_lookup - however having implemented this yesterday, it
>> seems like it quite possibly did block the occasional email.  Before
>> yesterday, it was just used as part of a scoring system.
>
> Whislt on the topic... how can I check that a host has a reverse
> record, without doing the full forward checking that
> reverse_host_lookup does?  I can at least block outright hosts that
> don't have a reverse...
>

Quoting from: http://www.gossamer-threads.com/lists/exim/users/83353

If you want to only check whether reverse DNS exists at all, with no
regard to the subsequent forward-DNS check, you can use something along
the lines of this:

defer
!condition = ${lookup dnsdb{defer_never,ptr=$sender_host_address}{yes}}
log_message = REVERSE DNS: no PTR found for $sender_host_address
message = Sender's ISP has no reverse DNS for $sender_host_address.


> --
> Blog: http://pookey.co.uk/blog
> Follow me on twitter: http://twitter.com/ipchristian
>
> --
> ## List details at http://lists.exim.org/mailman/listinfo/exim-users
> ## Exim details at http://www.exim.org/
> ## Please use the Wiki with this list - http://wiki.exim.org/
>

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Ian P. Christian]

2009/7/2 Joan <aseques@...>:
> If you want to only check whether reverse DNS exists at all, with no
> regard to the subsequent forward-DNS check, you can use something along
> the lines of this:

Perfect, thanks!

--
Blog: http://pookey.co.uk/blog
Follow me on twitter: http://twitter.com/ipchristian

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[W B Hacker]

Ian P. Christian wrote:

> 2009/7/2 Ian P. Christian <pookey@...>:
>> Hi all,
>>
>> I though I'd seen people post on the list about blocking outright on
>> reverse_host_lookup - however having implemented this yesterday, it
>> seems like it quite possibly did block the occasional email.  Before
>> yesterday, it was just used as part of a scoring system.
>
> Whislt on the topic... how can I check that a host has a reverse
> record, without doing the full forward checking that
> reverse_host_lookup does?  I can at least block outright hosts that
> don't have a reverse...
>

reverse_host_lookup does that 'first' then bails-out when no record at all is
found. it goes no further into building the list and logic of 'possibles' to
find out if a PTR RR is part of a proper DNS record set for mail service, or is
merely 'generic'.

See ~/src/host.c

So no further resources are wasted when a PTR RR is absent altogether.

Downside is that the only 'visible' difference between no PTR RR at all, and a
PTR RR that doesn't match to anything useful for smtp is in what Exim offers as
a log entry. Both conditions otherwise return a '0' eventually.

I've posted, and am running, an experimental one-line change to source that
makes the difference more readily ussable in an acl.

CAVEAT: All it really accomplishes is to cut one line per rejection of log
verbosity, so I'm not touting it.

The 'stock' Exim is just fine as-is, needing neither the above nor dnsdb, as it
already permits accepting with no, present but-generic, or full and 'proper' DNS
records - 'scored' by weighted point-value if one wishes to use such.

Bill



--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/