greylisting

[captainhastings]

I have grey listing running and it seems to work well stopping the bulk
of spam. But I had one chap complaining last week that some one was
emailing him and it was failing to get through.
A quick log check showed

2009-06-04 12:29:10 [76405] H=vscano-b2.ucl.ac.uk [128.40.105.157]:47833
I=[94.76.221.176]:25 F=<Egyptian@...> temporarily
rejected RCPT <Christine@...>: greylisted host
128.40.105.157

2009-06-04 12:29:10 [76405] H=vscano-b2.ucl.ac.uk [128.40.105.157]:47833
I=[94.76.221.176]:25 incomplete transaction (QUIT) from
<Egyptian@...>

2009-06-05 16:19:50 [48875] H=vscano-a2.ucl.ac.uk [144.82.100.153]:64384
I=[94.76.221.176]:25 F=<Egyptian@...> temporarily
rejected RCPT <Christine@...>: greylisted host
144.82.100.153

2009-06-05 16:19:50 [48875] H=vscano-a2.ucl.ac.uk [144.82.100.153]:64384
I=[94.76.221.176]:25 incomplete transaction (QUIT) from
<Egyptian@...>

So I guess there server just never retried after being grey listed at
least not from the same host. Other than not greylisting I take it
theres no way of avoiding the odd incident like this
There email got returned with a 550 unable to relay error. I am no
expert at this but I do try and get things right and avoid causing other
people problems.
I have since whitelisted them ( using postgrey by the way ) .

Thanks
Terry

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Ian P. Christian]

2009/6/6 Terry <terry@...>:
> Thanks for the reply do you use grey listing ? I would be very reluctant to
> drop it

I use it selectively... if a host looks naughty, then I do.  no rDNS,
silly HELO's,  host in XBL.. that kinda thing. I keep an ACL variable
for a score, then only greylist based on if that score is greater than
X.

I think greylisting should be avoided where possible, as it will lead
to legitimate mail being bounced, lost, or delayed - and no one wants
any of that :)


--
Blog: http://pookey.co.uk/blog
Follow me on twitter: http://twitter.com/ipchristian

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[W B Hacker]

Terry wrote:

> I have grey listing running and it seems to work well stopping the bulk
> of spam. But I had one chap complaining last week that some one was
> emailing him and it was failing to get through.
> A quick log check showed
>
> 2009-06-04 12:29:10 [76405] H=vscano-b2.ucl.ac.uk [128.40.105.157]:47833
> I=[94.76.221.176]:25 F=<Egyptian@...> temporarily
> rejected RCPT <Christine@...>: greylisted host
> 128.40.105.157
>
> 2009-06-04 12:29:10 [76405] H=vscano-b2.ucl.ac.uk [128.40.105.157]:47833
> I=[94.76.221.176]:25 incomplete transaction (QUIT) from
> <Egyptian@...>
>
> 2009-06-05 16:19:50 [48875] H=vscano-a2.ucl.ac.uk [144.82.100.153]:64384
> I=[94.76.221.176]:25 F=<Egyptian@...> temporarily
> rejected RCPT <Christine@...>: greylisted host
> 144.82.100.153
>
> 2009-06-05 16:19:50 [48875] H=vscano-a2.ucl.ac.uk [144.82.100.153]:64384
> I=[94.76.221.176]:25 incomplete transaction (QUIT) from
> <Egyptian@...>
>
> So I guess there server just never retried after being grey listed at
> least not from the same host.

There is the key.

A SWAG says it looks as if they make two back-to-back attempts, then drop any
failures onto a backup host for delayed retry, and that it also does two
back-to-back then hard fail. .. Where 'back to back' may mean a RST and go again
on the same connection. Your logs can show that if made more verbose.

Hard to fault their approach, BTW.

Compared to typical retry configurations it will make far queue-manips, fewer
*total* retries, gives the opportunity of avoiding IP or route-specific
problems, and lets the sender know much earlier that there *is* a problem.

OTOH - greylisting-friendly it is not...

> Other than not greylisting I take it
> theres no way of avoiding the odd incident like this

You could do - if you choose to ignore the source IP and go only on the
envelope-from and recpt-to coupling.

OTOH, unless you have ALSO pre-qualified the source IP (reverse_host_lookup)
ignoring the IP might make GL far less effective for OTHER arrivals.

...it will also make GL trigger less often, 'coz zombots rarely pass rDNS
checks. Enough 'less often' that we were able to scrap GL altogether.. YMMV.

> There email got returned with a 550 unable to relay error. I am no
> expert at this but I do try and get things right and avoid causing other
> people problems.
> I have since whitelisted them ( using postgrey by the way ) .
>
> Thanks
> Terry
>

So long as your user-base doesn't go berserk while discovering the hard way what
*needs* WL, you probably won't have to WL more than a dozen or so (ever), as the
above multi-IP pattern WITH a mere 4 total tries is not all that common.



Bill

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[David Woodhouse]

On Sat, 2009-06-06 at 17:19 +0100, Terry wrote:
> So I guess there server just

Where server?

> never retried after being grey listed at least not from the same host.

Why would you require them to retry from the _same_ host?

That seems unnecessary and silly -- many sites will punt 'problematic'
mails to another host to be retried periodically, to keep the mail queue
on the main servers clear.

>  Other than not greylisting I take it theres no way of avoiding the
> odd incident like this

Er, how about greylisting sensibly, instead of badly? :)

> ( using postgrey by the way ) .

Ah, that explains it. Postgrey is one of the worst implementations of
greylisting I've ever encountered. Why aren't you just using Exim's
native capabilities?

I'd suggest reading http://wiki.exim.org/SimpleGreylisting -- the prose
sets out some things that you may want to think about regardless of
which greylisting implementation you use, and then there's an example
Exim configuration which shouldn't suffer most of the stupid problems
that postgrey does.

There are still trade-offs which are fundamental to the nature of
greylisting, of course, but most of them can be significantly minimised.
Especially in comparison with postgrey.

--
dwmw2


--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Ian P. Christian]

2009/6/8 David Woodhouse <dwmw2@...>:
> I'd suggest reading http://wiki.exim.org/SimpleGreylisting -- the prose
> sets out some things that you may want to think about regardless of
> which greylisting implementation you use, and then there's an example
> Exim configuration which shouldn't suffer most of the stupid problems
> that postgrey does.

There's actually a flaw in this implementation here.

  # Generate a hashed 'identity' for the mail, as described above.
  warn set acl_m_greyident =
${hash{20}{62}{$sender_address$recipients$h_message-id:}}

Because it's common at the moment to get a mail to someone sent from
their own address without a message ID, hash clashes occour.

I'm currently not sure of the best way to d eal with this - perahps
adding the Subject line into the hash...

Perhaps I should just block mail sent from someone, to themselves,
with a null message ID.

--
Blog: http://pookey.co.uk/blog
Follow me on twitter: http://twitter.com/ipchristian

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[W B Hacker]

Ian P. Christian wrote:

> 2009/6/8 David Woodhouse <dwmw2@...>:
>> I'd suggest reading http://wiki.exim.org/SimpleGreylisting -- the prose
>> sets out some things that you may want to think about regardless of
>> which greylisting implementation you use, and then there's an example
>> Exim configuration which shouldn't suffer most of the stupid problems
>> that postgrey does.
>
> There's actually a flaw in this implementation here.
>
>   # Generate a hashed 'identity' for the mail, as described above.
>   warn set acl_m_greyident =
> ${hash{20}{62}{$sender_address$recipients$h_message-id:}}
>
> Because it's common at the moment to get a mail to someone sent from
> their own address without a message ID, hash clashes occour.
>
> I'm currently not sure of the best way to d eal with this - perahps
> adding the Subject line into the hash...
>
> Perhaps I should just block mail sent from someone, to themselves,
> with a null message ID.
>

Got me confused....

- If the message is 'outbound' it has not (and will not) see *your* greylisting,

- but 'should' get a message_ID generated and added by the MTA if there was none
(to self or otherwise). Simple option, 'owed' to our fellow MTA admins anyway,
and 'kinder' to Em Mess Dose hag-ridden et al than blocking.

.. or at least less hassle than arguing with 'em about 'upgrading' or 'standards'.

;-)

Bill



--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Ian P. Christian]

2009/6/11 W B Hacker <wbh@...>:
> Got me confused....

turns out I'm probably confused.  I'm getting duplicate insertion
errors from something.. I thought that explained it, but actually, it
later turned out it didn't.

There's something wrong with the implementation, or my implementation
of that implementation ;)

I'll past back when (if) I figure out what it is, assuming it's not
just me being stupid!

--
Blog: http://pookey.co.uk/blog
Follow me on twitter: http://twitter.com/ipchristian

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[W B Hacker]

Ian P. Christian wrote:

> 2009/6/11 W B Hacker <wbh@...>:
>> Got me confused....
>
> turns out I'm probably confused.  I'm getting duplicate insertion
> errors from something.. I thought that explained it, but actually, it
> later turned out it didn't.
>
> There's something wrong with the implementation, or my implementation
> of that implementation ;)
>
> I'll past back when (if) I figure out what it is, assuming it's not
> just me being stupid!
>

LOL!

Another of Exim's great strengths .... allowing us to ... ah

'express our creativity'.

BTDTGTTSWBH

Bill

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Eduardo M KALINOWSKI-4]

On Seg, 08 Jun 2009, David Woodhouse wrote:
> I'd suggest reading http://wiki.exim.org/SimpleGreylisting -- the prose
> sets out some things that you may want to think about regardless of
> which greylisting implementation you use, and then there's an example
> Exim configuration which shouldn't suffer most of the stupid problems
> that postgrey does.

Thanks for the link. I already had selective greylisting implemented  
in my exim (using another database recipe I found somewhere), but this  
implementation is simpler and more straightforward. I've just  
implemented it.

One thing that struck me is: once a greylisted message is seen again  
and accepted (because the delay is over), couldn't its entry be  
removed from the greylist table? It will eventually be removed 7 days  
later from the cron script, but couldn't it be removed with something  
like this:

   # The message was listed but it's been more than five minutes.  
Accept it now and whitelist
   # the _original_ sending host by its { IP, HELO } so that we don't  
delay its mail again.
   warn set acl_m_orighost = ${lookup sqlite {GREYDB SELECT host FROM  
greylist \
                                 WHERE  
id='${quote_sqlite:$acl_m_greyident}';}{$value}}
        set acl_m_orighelo = ${lookup sqlite {GREYDB SELECT helo FROM  
greylist \
                                 WHERE  
id='${quote_sqlite:$acl_m_greyident}';}{$value}}
        set acl_m_dontcare = ${lookup sqlite {GREYDB INSERT INTO resenders \
                                 VALUES ( '$acl_m_orighost', \
                                          '${quote_sqlite:$acl_m_orighelo}', \
                                          '$tod_epoch' ); }}
        set acl_m_dontcare = ${lookup sqlite {GREYDB DELETE FROM greylist \
                                 WHERE id='${quote_sqlite:$acl_m_greyident}';}}
        logwrite = Added host $acl_m_orighost with HELO  
'$acl_m_orighelo' to known resenders

     What do you guys think? Is it worth it? Or it is better to leave  
old entries (retried or not) to be bulk deleted from cron?


--
Eduardo M KALINOWSKI
eduardo@...


--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[David Woodhouse]

On Fri, 2009-06-12 at 10:38 -0300, Eduardo M KALINOWSKI wrote:
> One thing that struck me is: once a greylisted message is seen again  
> and accepted (because the delay is over), couldn't its entry be  
> removed from the greylist table?
 ...
>      What do you guys think? Is it worth it? Or it is better to leave  
> old entries (retried or not) to be bulk deleted from cron?

Please don't drop me from Cc when replying to me. It's quite impolite.

I did think about removing entries when we accept mail -- but since we
still need to run the cron job _anyway_ to clean up after the messages
which _weren't_ retried, I didn't really see the point.

It's just more complexity without any real benefit -- the size of the
greylist db isn't really much of an issue in practice, is it?

--
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@...                              Intel Corporation


--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[David Woodhouse]

On Thu, 2009-06-11 at 13:41 +0100, Ian P. Christian wrote:
> 2009/6/8 David Woodhouse <dwmw2@...>:
> > I'd suggest reading http://wiki.exim.org/SimpleGreylisting -- the prose
> > sets out some things that you may want to think about regardless of
> > which greylisting implementation you use, and then there's an example
> > Exim configuration which shouldn't suffer most of the stupid problems
> > that postgrey does.
>
> There's actually a flaw in this implementation here.

Er, thanks for dropping me from Cc when you criticize my work...! :)

>   # Generate a hashed 'identity' for the mail, as described above.
>   warn set acl_m_greyident =
> ${hash{20}{62}{$sender_address$recipients$h_message-id:}}
>
> Because it's common at the moment to get a mail to someone sent from
> their own address without a message ID, hash clashes occour.

Yeah, at the time I first implemented this I was just rejecting all mail
without a Message-Id, so it wasn't much of an issue.

> I'm currently not sure of the best way to deal with this - perahps
> adding the Subject line into the hash...

That seems like it would be a reasonable thing to do. Is it enough,
though? A lot of spam messages have the same subject line too.

What else could we include -- bearing in mind that we have to be sure
that it _won't_ get changed by the sending MTA between retry attempts. I
suppose we could use the full From:, To: and Cc: headers -- and maybe
also the Date: header?

> Perhaps I should just block mail sent from someone, to themselves,
> with a null message ID.

You could use PRVS and just reject _all_ mail which is faked to appear
as if it's from your own addresses, surely?

--
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@...                              Intel Corporation


--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Dmitry Samersoff]

David,

I think SQL for graylist is totally overkill.

Check this implementation:
http://www.beastsoft.net/cgi-bin/hg/hgwebdir.cgi/greyd/

-Dmitry

David Woodhouse wrote:

> On Thu, 2009-06-11 at 13:41 +0100, Ian P. Christian wrote:
>> 2009/6/8 David Woodhouse <dwmw2@...>:
>>> I'd suggest reading http://wiki.exim.org/SimpleGreylisting -- the prose
>>> sets out some things that you may want to think about regardless of
>>> which greylisting implementation you use, and then there's an example
>>> Exim configuration which shouldn't suffer most of the stupid problems
>>> that postgrey does.
>> There's actually a flaw in this implementation here.
>
> Er, thanks for dropping me from Cc when you criticize my work...! :)
>
>>   # Generate a hashed 'identity' for the mail, as described above.
>>   warn set acl_m_greyident =
>> ${hash{20}{62}{$sender_address$recipients$h_message-id:}}
>>
>> Because it's common at the moment to get a mail to someone sent from
>> their own address without a message ID, hash clashes occour.
>
> Yeah, at the time I first implemented this I was just rejecting all mail
> without a Message-Id, so it wasn't much of an issue.
>
>> I'm currently not sure of the best way to deal with this - perahps
>> adding the Subject line into the hash...
>
> That seems like it would be a reasonable thing to do. Is it enough,
> though? A lot of spam messages have the same subject line too.
>
> What else could we include -- bearing in mind that we have to be sure
> that it _won't_ get changed by the sending MTA between retry attempts. I
> suppose we could use the full From:, To: and Cc: headers -- and maybe
> also the Date: header?
>
>> Perhaps I should just block mail sent from someone, to themselves,
>> with a null message ID.
>
> You could use PRVS and just reject _all_ mail which is faked to appear
> as if it's from your own addresses, surely?
>

--
Dmitry Samersoff
dms@..., http://devnull.samersoff.net
* There will come soft rains ...


--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[David Woodhouse]

On Sat, 2009-06-27 at 17:28 +0400, Dmitry Samersoff wrote:
> I think SQL for graylist is totally overkill.

It's only sqlite -- it doesn't require a separate database server; it's
purely within Exim. You've got to have _some_ kind of database, and this
is more efficient than just doing it with text files (as my original
implementation did.

> Check this implementation:
> http://www.beastsoft.net/cgi-bin/hg/hgwebdir.cgi/greyd/

Ew, Mercurial and C++... not the best first impression.

A separate dæmon written in C++ with a 'thread pool' implementation and
weird OS 'abstraction' layers to handle signals... that's not overkill?

You also don't seem to be passing it anything other than $sender_address
and $sender_host_address -- and you're even assuming the latter is
Legacy IP, afaict.

--
dwmw2


--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Dmitry Samersoff]

David,

David Woodhouse wrote:
> On Sat, 2009-06-27 at 17:28 +0400, Dmitry Samersoff wrote:
>> I think SQL for graylist is totally overkill.
>
> It's only sqlite -- it doesn't require a separate database server; it's
> purely within Exim.

It doesn't make it faster and doesn't excuse SQL parser, transactions
and bunch of other staff not needed in this case.

> You've got to have _some_ kind of database, and this
> is more efficient than just doing it with text files (as my original
> implementation did.

sqlite is nice product (good step back to 1994) but people tend to
consider it as a cure-all-diseases magic pile. We need a record manager
here but not a database - i.e. Berkeley DB, not sqlite.

>> Check this implementation:
>> http://www.beastsoft.net/cgi-bin/hg/hgwebdir.cgi/greyd/
>
> Ew, Mercurial and C++... not the best first impression.
>
> A separate dæmon written in C++ with a 'thread pool' implementation and
> weird OS 'abstraction' layers to handle signals... that's not overkill?

It's really fast and scalable (actually what it was written for - one of
mid size ISP asked me for help). Also it couldn't cause email loss -
i.e. if something goes wrong e-mail just passed in.

> You also don't seem to be passing it anything other than $sender_address
> and $sender_host_address -- and you're even assuming the latter is
> Legacy IP, afaict.

I'm checking sender host address and sender from address, e.g:
209.85.218.168:*@gmail.com

This combination is sufficient enough but not perfect. What else you
suggest to check?


--
Dmitry Samersoff
dms@..., http://devnull.samersoff.net
* There will come soft rains ...


--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[David Woodhouse]

On Sat, 2009-06-27 at 19:49 +0400, Dmitry Samersoff wrote:

> David,
>
> David Woodhouse wrote:
> > On Sat, 2009-06-27 at 17:28 +0400, Dmitry Samersoff wrote:
> >> I think SQL for graylist is totally overkill.
> >
> > It's only sqlite -- it doesn't require a separate database server; it's
> > purely within Exim.
>
> It doesn't make it faster and doesn't excuse SQL parser, transactions
> and bunch of other staff not needed in this case.
>
> > You've got to have _some_ kind of database, and this
> > is more efficient than just doing it with text files (as my original
> > implementation did.
>
> sqlite is nice product (good step back to 1994) but people tend to
> consider it as a cure-all-diseases magic pile. We need a record manager
> here but not a database - i.e. Berkeley DB, not sqlite.

True -- it would be nice if we could use Berkeley DB that way from Exim,
but it's read-only.

> >> Check this implementation:
> >> http://www.beastsoft.net/cgi-bin/hg/hgwebdir.cgi/greyd/
> >
> > Ew, Mercurial and C++... not the best first impression.
> >
> > A separate dæmon written in C++ with a 'thread pool' implementation and
> > weird OS 'abstraction' layers to handle signals... that's not overkill?
>
> It's really fast and scalable (actually what it was written for - one of
> mid size ISP asked me for help). Also it couldn't cause email loss -
> i.e. if something goes wrong e-mail just passed in.

Sounds like it's being used too much. Ideally, I believe greylisting
should only be invoked for mails which look suspicious in some way, if
they come from a host which hasn't previously been observed to queue and
retry.

> > You also don't seem to be passing it anything other than $sender_address
> > and $sender_host_address -- and you're even assuming the latter is
> > Legacy IP, afaict.
>
> I'm checking sender host address and sender from address, e.g:
> 209.85.218.168:*@gmail.com

How's it going to cope with what I get on your incoming mail:
        2001:4830:2446:ff00:214:51ff:fe65:c65c:dms@...

> This combination is sufficient enough but not perfect. What else you
> suggest to check?

There is some discussion of that on the wiki page to which I referred.

--
dwmw2


--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Dmitry Samersoff]

David,

>>> A separate dæmon written in C++ with a 'thread pool' implementation and
>>> weird OS 'abstraction' layers to handle signals... that's not overkill?
>> It's really fast and scalable (actually what it was written for - one of
>> mid size ISP asked me for help). Also it couldn't cause email loss -
>> i.e. if something goes wrong e-mail just passed in.
>
> Sounds like it's being used too much. Ideally, I believe greylisting
> should only be invoked for mails which look suspicious in some way, if
> they come from a host which hasn't previously been observed to queue and
> retry.

I try to greylist all incoming mail as early as possible to minimize
relay load and (less important) traffic.

>>> You also don't seem to be passing it anything other than $sender_address
>>> and $sender_host_address -- and you're even assuming the latter is
>>> Legacy IP, afaict.
>> I'm checking sender host address and sender from address, e.g:
>> 209.85.218.168:*@gmail.com
>
> How's it going to cope with what I get on your incoming mail:
> 2001:4830:2446:ff00:214:51ff:fe65:c65c:dms@...

What is it? MSG ID?  My outgoing e-mail not graylisted - I use
login/pass & SSL.

> There is some discussion of that on the wiki page to which I referred.

Some comments on wiki page:

I.
"One problem is that some "genuine" mail servers might be broken in the
  same way as we expect the spam bots to be."

Much more often problem is clusters: lots of SMTP clusters can resend
e-mail from another IP address. Fortunately most of them is not
spammers.  I think a) the problem should be mentioned in this chapter b)
it's better to whitelist such clusters rather then exclude sender IP
from hash calculation.

II.

"Firstly, there's the database of "known resenders", which lists the
hosts that are known to retry sending mail."

In the NAT world I'm in doubt that such list have a sense.

--
Dmitry Samersoff
dms@..., http://devnull.samersoff.net
* There will come soft rains ...


--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[list-40]

Le samedi 27 juin 2009 à 15:28:34, Dmitry Samersoff, vous avez écrit :
> David,
>
> I think SQL for graylist is totally overkill.
[...]

I used 'greylistd' python script ( using Exim socket ), then decided to
reimplement its in purely within Exim ${dlfunc.

In RCPT or DATA :

 defer condition = ${if ...}{1}{0}}
       set acl_m_gris  = ${dlfunc{/usr/lib/exim/exim-gris.so}{gris}}
       condition          = ${if >{$acl_m_gris}{0}}
       message          = Greylisting, please try again later
                                ($acl_m_gris secondes)

This ${dlfunc use Exim functions for accessing Exim's hints database. This
module does not contain code for reading DBM  files, all the relevant
functions are called as Exim macros (exim-4.69/src/dbstuff.h,
exim-4.69/src/dbfunctions.h). The DBM file has stored in Exim
spool_directory/db/greylisting

For maintaining greylisting database, this function auto remove obsolete data
(macro LAST_UPDATE 60*15).

To dump out the contents of the database:
% exim -be '${dlfunc{/usr/lib/exim/exim-gris.so}{dump}}'
#
# Greylisting Sat Jun 27 18:45:57 2009
#
{
key  = "131.111.8/exim-users@.../recipient@..."
        flag  = W
        last  = Thu Jun 25 13:29:04 2009
        first = Thu Jun 25 13:19:54 2009
        count = 1
}
[...]
# Total: 83 (White: 57|Grey: 25)

% gcc -O2 -Wall -Werror -shared -fPIC -g \
        -I./exim-4.69/build-Linux-i386/ \
        -L/usr/lib \
        -DLAST_UPDATE=60*15 \
        -DMAX_GREY=60*60*8 \
        -DRETRY_MAX=60*60*24*36 \
        -DRETRY_MIN=60*5 \
        -o /usr/lib/exim/exim-gris.so exim-gris.c \
        && strip /usr/lib/exim/exim-gris.so


-- I'm French. Sorry for my poor English.

--
Serge Demonchaux

[exim-gris.c]

---

/*
 * Copyright (C) 2009 by Serge Demonchaux
 *  <serge(DOT)d3(AT)free(DOT)fr>
 * $Id: exim-gris.c, v0.0.1 Jun 2009
 */
/**
- Build :
 % gcc -O2 -Wall -Werror -shared -fPIC -g \
        -I./exim-4.69/build-* \
        -L/usr/lib \
        -DLAST_UPDATE=60*15 \
        -DMAX_GREY=60*60*8 \
        -DRETRY_MAX=60*60*24*36 \
        -DRETRY_MIN=60*5 \
        -o /usr/lib/exim/exim-gris.so exim-gris.c \
        && strip /usr/lib/exim/exim-gris.so


- In RCPT or DATA :

 defer condition       = ${if ...}{1}{0}}
       set acl_m_gris  = ${dlfunc{/usr/lib/exim/exim-gris.so}{gris}}
       condition       = ${if >{$acl_m_gris}{0}}
       message         = Greylisting, please try again later
                                ($acl_m_gris secondes)


- To dump out the contents of the database :
 % exim -be '${dlfunc{/usr/lib/exim/exim-gris.so}{dump}}'

- Or :
 % perl -e 'use DB_File;tie my %h, "DB_File", $ARGV[0], 0, 0, $DB_HASH;print "total: ".keys(%h)."\n";untie %h;' /var/spool/exim/db/greylisting

*/

#include <string.h>
#include <errno.h>
#include <time.h>
#include <fcntl.h>
/* Exim */
#include "local_scan.h"
#include "dbstuff.h"
#include "dbfunctions.h"


#define K_LAST_UPDATE "LAST_UPDATE" /* Nom de la clé du prochain */
                                      /* temps de nétoyage */
#ifndef LAST_UPDATE
#  define LAST_UPDATE 60*15      /* Interval de l'auto nétoyage */
#endif
#ifndef MAX_GREY
#  define MAX_GREY 60*60*8      /* Temps max à l'état Grey avant */
#endif      /* expiration */
#ifndef RETRY_MAX
#  define RETRY_MAX 60*60*24*36   /* Temps max à l'état White avant */
#endif      /* expiration */
#ifndef RETRY_MIN
#  define RETRY_MIN 60*3      /* Temps iniciale du mécanisme */
#endif      /* de grelistage */

#define DEBUG(x) if ((debug_selector & (x)) != 0)

/* Structure des données enregistées
 * dans la base de données.
 * ( dans l'ordre ).
 * Cette structure reprend les consignes sur l'utilisation
 * des db avec Exim -> dbfunctions.h + dbstuff.h
 */
typedef struct {
    time_t       time_stamp;  /* timestamp dernier passage */
  /*************/
    time_t       first; /* timestamp enregistrement */
    unsigned int flag;  /* drapeau 'W' ou 'G' ( le drapeu
                         * 'A' est réservé pour les clés
                         * d'administration du programme */
    unsigned int count; /* Compteur des passages */
} GREY_CTX;


/* --- global --- */

/* temps des différents états */
static int retry_max = RETRY_MAX;
static int retry_min = RETRY_MIN;
static int max_grey  = MAX_GREY;
/* db */
open_db dbblock, *dbm;
GREY_CTX *dbd;

int __greylisting_fn( unsigned char *recipient, char *relay_ip );
int __traverse_fn( int dump );


/* dump le contenu de la db dans un format humain
 * de cette manière:
 * exim -be '${dlfunc{/usr/lib/exim/exim-gris.so}{dump}}' > dump.txt
 */
int dump(void) {


    if ( ( dbm = dbfn_open(
        US"greylisting", O_RDWR, &dbblock, TRUE ) ) == NULL ) {
        DEBUG(D_local_scan)
            fprintf( stderr, "greylisting database not available!\n");
        return -1;
    };

    /* Traverse la db et dump sur STDERR */
    if (  __traverse_fn( 1 ) != 0 )
        goto err;

    (void) dbfn_close( dbm );

    return 0;

    err: (void) dbfn_close( dbm );
    return -1;
}

/* defer condition = ${if ...}{1}{0}}
 *       set acl_m_gris = ${dlfunc{/usr/lib/exim/exim-gris.so}{gris}}
 *       condition      = ${if >{$acl_m_gris}{0}}
 *       message        = Greylisting, please try again later \
 * ($acl_m_gris secondes)
 */
int gris( unsigned char **yield, int argc, unsigned char *argv[] ) {

    char ipaddr[16];
    char *mask;
    int  delay = 0;


   /* ------ greylisting ------- */
    if ( ( dbm = dbfn_open(
        US"greylisting", O_RDWR, &dbblock, TRUE ) ) == NULL ) {
        DEBUG(D_local_scan)
            debug_printf("greylisting database not available!\n");
        return OK;
    };

    DEBUG(D_local_scan) debug_printf("Using %s\n", EXIM_DBTYPE);

   /* --- construction de la requête ---
    * Elle prend la forme:
    * ip/sender/recipient.
    * L'ip est elle tronquée au dernier point,
    * par exemple : 192.168.0.10 est transformée en 192.168.0
    */
    strncpy( ipaddr, (sender_host_address)?
        (char*)sender_host_address : "127.0.0.1", sizeof( ipaddr ) );

    mask = ( strrchr( ipaddr, '.') ) ?
             strrchr( ipaddr, '.') :
                strrchr( ipaddr, ':');
    ipaddr[(int)(mask - ipaddr)] = '\0';

    /* acl_smtp_data */
    if ( expand_string(US"${if def:recipients}") != NULL ) {
        int i, ret;
        for ( i = 0; i < recipients_count; i++ ) {
            ret = __greylisting_fn( recipients_list[i].address, ipaddr );
            if ( ret > 0 )
                delay = ret;
        };
    } else {/* acl_smtp_rcpt */
        unsigned char *recipient;
        recipient = expand_string(US"$local_part@$domain");
        delay = __greylisting_fn( recipient, ipaddr );
    };

    (void) __traverse_fn( 0 );
    (void) dbfn_close( dbm );
    *yield = string_sprintf( "%d", delay );

    return OK;
}

/* clean db */
int __traverse_fn( int dump ) {

    EXIM_CURSOR *cursor;
    unsigned int total = 0, deleted = 0, white = 0, grey = 0;
    unsigned char *key;
    time_t now;
    int r;


    now = time( (time_t)NULL );
    key = (unsigned char*)K_LAST_UPDATE;

    if ( ( dbd = dbfn_read( dbm, key ) ) == NULL ) {
        dbd = store_get( sizeof( GREY_CTX ) );
        dbd->first = dbd->time_stamp = now;
        dbd->count = 0;
        dbd->flag  = 'A';
    };

    if ( dump == 0 && dbd->time_stamp + LAST_UPDATE > now )
        return 0;

    dbd->count += 1;
    if ( dbfn_write( dbm, key, dbd, sizeof(GREY_CTX) ) != 0 ) {
        DEBUG(D_local_scan)
            debug_printf("Failed: %s\n", strerror(errno));
        return -1;
    };

    if ( dump == 1 )
        fprintf( stderr, "#\n# Greylisting %s#\n", ctime(&now) );

    key = dbfn_scan( dbm, TRUE, &cursor);
    while ( key != NULL ) {

        dbd = dbfn_read( dbm, key );

        switch ( dbd->flag ) {
            case 'G': /* Grey */
                r = ( dbd->time_stamp + max_grey < now )? 1 : 0;
                break;
             case 'W': /* White */
                r = ( dbd->time_stamp + retry_max < now )? 1 : 0;
                break;
             case 'A': /* Admin key */
                r = 0;
                break;
             default: /* Inconnu */
                r = 1;
        };

        if ( r != 0 ) {
            (void) dbfn_delete( dbm, key );
            ++deleted;
        };
        if ( dump == 1 && r == 0 ) {
            white += (dbd->flag == 'W')? 1 : 0;
            grey  += (dbd->flag == 'G')? 1 : 0;
            fprintf( stderr, "{\nkey  = \"%s\"\n", key );
            fprintf( stderr, "\tflag  = %c\n", dbd->flag );
            fprintf( stderr, "\tlast  = %s", ctime(&dbd->time_stamp));
            fprintf( stderr, "\tfirst = %s", ctime(&dbd->first));
            fprintf( stderr, "\tcount = %u\n}\n", dbd->count );
        };
        ++total;
        key = dbfn_scan( dbm, FALSE, &cursor );
    };

    DEBUG(D_local_scan)
        debug_printf("total: %u deleted %u/%u  (White: %u|Grey: %u)\n", total-deleted, deleted, total, white, grey );

    if ( dump == 1 )
        fprintf( stderr, "\n# Total: %u (White: %u|Grey: %u)", total-deleted,  white, grey);

    return 0;
}

int __greylisting_fn( unsigned char *recipient, char *relay_ip ) {

    unsigned char query[256];
    time_t now;
    int delay;
    int r;


    now = time( (time_t)NULL );

    /* key: 'ip/sender/recipient' */
    if ( sender_address != NULL ) {
        (void) snprintf( (char*)query, sizeof( query ),
            "%s/%s/%s",
            relay_ip,
            sender_address,
            recipient );
    } else {
        (void) snprintf( (char*)query, sizeof( query ),
            "%s/MAILER-DAEMON@%s/%s",
            relay_ip,
            sender_host_name,
            recipient );
    };

    dbd = dbfn_read( dbm, query );
    /*  Validation des informations récupérées directement
     *  dans la structure GREY_CTX de 'g'
     */
    if ( dbd == NULL ) {/* initializing new key's data */
        dbd = store_get( sizeof( GREY_CTX ) );
        r = -1;
    } else if ( dbd->flag == 'W' ) {
        r = (dbd->time_stamp + retry_max < now)? -1 : 0;
    } else {//if ( dbd->flag == 'G' ) {
         r = (dbd->first + retry_min > now)? 1 ://encore grey
             (dbd->time_stamp + max_grey < now)? -1 : 0;//expiré
    };

    switch ( r ) {
        case 0:
            dbd->flag  = 'W';
            delay = 0;
            break;
        case 1:
            dbd->flag  = 'G';
            delay = (dbd->first + retry_min) - now;
            break;
        default:
            dbd->time_stamp = dbd->first = now;
            dbd->flag  = 'G';
            dbd->count = 0;
            delay = retry_min;
            DEBUG(D_local_scan)
                debug_printf("greylisting initializing new key's\n");
    };
    /* compteur */
    dbd->count += 1;

    if ( dbfn_write( dbm, query, dbd, sizeof(GREY_CTX)) != 0 ) {
        DEBUG(D_local_scan)
            debug_printf("Failed: %s\n", strerror(errno));
    } else {
        DEBUG(D_local_scan)
            debug_printf("greylisting db updated\n");
    };


    if ( delay > 0 )
        log_write(0, LOG_MAIN, "${dlfunc greylisting "
            "relay_ip=%s sender=<%s> recipient=<%s> delay=%dsecs",
            relay_ip,
            (sender_address)? (char*)sender_address : "MAILER-DAEMON",
            recipient, delay );

    return delay;
};

---

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[David Woodhouse]

On Sat, 2009-06-27 at 21:00 +0400, Dmitry Samersoff wrote:

> David,
>
> >>> A separate dæmon written in C++ with a 'thread pool' implementation and
> >>> weird OS 'abstraction' layers to handle signals... that's not overkill?
> >> It's really fast and scalable (actually what it was written for - one of
> >> mid size ISP asked me for help). Also it couldn't cause email loss -
> >> i.e. if something goes wrong e-mail just passed in.
> >
> > Sounds like it's being used too much. Ideally, I believe greylisting
> > should only be invoked for mails which look suspicious in some way, if
> > they come from a host which hasn't previously been observed to queue and
> > retry.
>
> I try to greylist all incoming mail as early as possible to minimize
> relay load and (less important) traffic.
>
> >>> You also don't seem to be passing it anything other than $sender_address
> >>> and $sender_host_address -- and you're even assuming the latter is
> >>> Legacy IP, afaict.
> >> I'm checking sender host address and sender from address, e.g:
> >> 209.85.218.168:*@gmail.com
> >
> > How's it going to cope with what I get on your incoming mail:
> > 2001:4830:2446:ff00:214:51ff:fe65:c65c:dms@...
>
> What is it? MSG ID?  My outgoing e-mail not graylisted - I use
> login/pass & SSL.

That's just $sender_host_address:$sender_address for the mail that you
sent, on one of its hops on the way to me.

I was asking how your code would cope with it, if I was using your code.

> > There is some discussion of that on the wiki page to which I referred.
>
> Some comments on wiki page:
>
> I.
> "One problem is that some "genuine" mail servers might be broken in the
>   same way as we expect the spam bots to be."
>
> Much more often problem is clusters: lots of SMTP clusters can resend
> e-mail from another IP address. Fortunately most of them is not
> spammers.  I think a) the problem should be mentioned in this chapter b)
> it's better to whitelist such clusters rather then exclude sender IP
> from hash calculation.

That would be nice if we know where they are. If they're on the dnswl,
they _will_ be skipped by the configuration I suggest.

But that's still not sufficient for the hosts that aren't -- the problem
is that if you include the sender IP in the hash, you'll settle into a
mode where you _always_ defer the mail from the first MTA, and always
accept it from the backup, when it's deferred. You're forcing your mail
to _always_ go through a slower path than it needs to.

> II.
>
> "Firstly, there's the database of "known resenders", which lists the
> hosts that are known to retry sending mail."
>
> In the NAT world I'm in doubt that such list have a sense.

That's why we use $sender_helo_name _and_ $sender_host_address, rather
than just $sender_host_address.

It's not perfect, but the alternative is that you just keep on deferring
mail from the same hosts over and over again, even though you gain no
benefit by doing so. You can of course do that if you wish, but many
people won't want to.

--
dwmw2


--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Heiko Schlittermann]

Dmitry Samersoff <dms@...> (Sa 27 Jun 2009 19:00:13 CEST):
...
>
> Much more often problem is clusters: lots of SMTP clusters can resend
> e-mail from another IP address. Fortunately most of them is not
> spammers.  I think a) the problem should be mentioned in this chapter b)
> it's better to whitelist such clusters rather then exclude sender IP
> from hash calculation.

I'm greylisting on '$sender_address|$local_part@$domain'. It works with
cluster based senders (web.de, I think).

It's perl based as exim "plugin", but limited to a single host.

    http://www.schlittermann.de/doc/grey.shtml

    Best regards from Dresden/Germany
    Viele Grüße aus Dresden
    Heiko Schlittermann
--
 SCHLITTERMANN.de ---------------------------- internet & unix support -
 Heiko Schlittermann HS12-RIPE -----------------------------------------
 gnupg encrypted messages are welcome - key ID: 48D0359B ---------------
 gnupg fingerprint: 3061 CFBF 2D88 F034 E8D2  7E92 EE4E AC98 48D0 359B -


--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Chris Wilson-30]

Hi all,

On Sat, 27 Jun 2009, Heiko Schlittermann wrote:

> Dmitry Samersoff <dms@...> (Sa 27 Jun 2009 19:00:13 CEST):
>>
>> Much more often problem is clusters: lots of SMTP clusters can resend
>> e-mail from another IP address. Fortunately most of them is not
>> spammers.  I think a) the problem should be mentioned in this chapter
>> b) it's better to whitelist such clusters rather then exclude sender IP
>> from hash calculation.

It's not exactly a solution, but Jaco Kroon in South Africa is running an
RBL which advertises hosts that pass greylisting. The big server farms are
all on it, all of their servers, and new ones get added pretty quickly.
We're using it effectively share our greylist database with him, by
contributing our database to his RBL as well.

http://jkroon.blogs.uls.co.za/it/email/spam/getting-a-grip-on-greylisting

> It's perl based as exim "plugin", but limited to a single host.

I do greylisting in pure exim + mysql. You can see how in my spam
presentation at www.exim-new-users.co.uk, although the site seems to be
down at the moment so I can't give you a link. I've also updated my config
a bit since then, especially to integrate with what Jaco is doing.

Cheers, Chris.
--
_____ __     _
\  __/ / ,__(_)_  | Chris Wilson <0000 at qwirx.com> - Cambs UK |
/ (_/ ,\/ _/ /_ \ | Security/C/C++/Java/Ruby/Perl/SQL Developer |
\__/_/_/_//_/___/ | We are GNU : free your mind & your software |

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/