Here's how the milter library works in sendmail

[Ale2008]

Hi all,
there has been some discussion some time ago about the possibility
to implement a milter-compatible library for courier. At the time, I
didn't know how milter actually works. Now I have one more reason to
be happy for using courier!

-------- Original Message --------
Subject: RE: Abort data transfer?
Date: Thu, 22 Oct 2009 11:02:45 -0700
From: Murray S. Kucherawy <msk@...>
To: David MacQuigg <macquigg@...>,
  IETF SMTP list <ietf-smtp@...>


> If the receiver is accepting data so fast, that it runs a few seconds
> ahead of the milter process, then a few seconds of data might
> accumulate
> *beyond* where the milter says REJECT.  That's certainly less of a
> problem than receiving data forever.

This isn't how Sendmail works.  The entire message is cached to the
queue before milter is told anything about the headers or body.
There's no "a few seconds ahead", it's all the way ahead.  Milter
has no opportunity to say REJECT in the middle of the SMTP DATA
phase because the filter doesn't even know that's where the MTA is.

Read the code in srvrsmtp.c and note that collect() is called before
milter_data(); the former collects the entire message from the
incoming stream, and the latter is what sends header and body
information down to the filters.

> I just did an experiment with Sendmail using telnet as the client.  If
> I
> send one line of data, then pause, Sendmail waits 10 minutes then
> disconnects with no message to telnet.  There is no timeout in telnet,
> so it just sits there until I force a disconnect on that end.

That's an MTA timeout and has nothing to do with milter.  Upon
reaching this timeout, the filter has received the envelope data but
none of the header or body, and will be given an "abort" instruction.


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
courier-users mailing list
courier-users@...
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users

[Alexander Erameh]

You know you hit the Nail right on the head. Courier is simply fantastic.
Big Kudos to the developers.

Alexander

-----Original Message-----
From: Alessandro Vesely [mailto:vesely@...]
Sent: Friday, October 23, 2009 7:56 AM
To: courier-users@...
Subject: [courier-users] Here's how the milter library works in sendmail

Hi all,
there has been some discussion some time ago about the possibility
to implement a milter-compatible library for courier. At the time, I
didn't know how milter actually works. Now I have one more reason to
be happy for using courier!

-------- Original Message --------
Subject: RE: Abort data transfer?
Date: Thu, 22 Oct 2009 11:02:45 -0700
From: Murray S. Kucherawy <msk@...>
To: David MacQuigg <macquigg@...>,
  IETF SMTP list <ietf-smtp@...>


> If the receiver is accepting data so fast, that it runs a few seconds
> ahead of the milter process, then a few seconds of data might
> accumulate
> *beyond* where the milter says REJECT.  That's certainly less of a
> problem than receiving data forever.

This isn't how Sendmail works.  The entire message is cached to the
queue before milter is told anything about the headers or body.
There's no "a few seconds ahead", it's all the way ahead.  Milter
has no opportunity to say REJECT in the middle of the SMTP DATA
phase because the filter doesn't even know that's where the MTA is.

Read the code in srvrsmtp.c and note that collect() is called before
milter_data(); the former collects the entire message from the
incoming stream, and the latter is what sends header and body
information down to the filters.

> I just did an experiment with Sendmail using telnet as the client.  If
> I
> send one line of data, then pause, Sendmail waits 10 minutes then
> disconnects with no message to telnet.  There is no timeout in telnet,
> so it just sits there until I force a disconnect on that end.

That's an MTA timeout and has nothing to do with milter.  Upon
reaching this timeout, the filter has received the envelope data but
none of the header or body, and will be given an "abort" instruction.


----------------------------------------------------------------------------
--
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
courier-users mailing list
courier-users@...
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
courier-users mailing list
courier-users@...
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users

[Matus UHLAR - fantomas]

On 23.10.09 08:55, Alessandro Vesely wrote:
> there has been some discussion some time ago about the possibility
> to implement a milter-compatible library for courier. At the time, I
> didn't know how milter actually works. Now I have one more reason to
> be happy for using courier!

I somehow don't understand what you mean.

I would invite milter interface for courier. The present filtering
capability can't modify the mail (or can it already?) which would be nice
for integrating e.g. spamassassin, DCC, DKIM and probably other software and
techniques.

And how is this related to the mail cited below?

> -------- Original Message --------
> Subject: RE: Abort data transfer?
> Date: Thu, 22 Oct 2009 11:02:45 -0700
> From: Murray S. Kucherawy <msk@...>
> To: David MacQuigg <macquigg@...>,
>   IETF SMTP list <ietf-smtp@...>
>
>
> > If the receiver is accepting data so fast, that it runs a few seconds
> > ahead of the milter process, then a few seconds of data might
> > accumulate
> > *beyond* where the milter says REJECT.  That's certainly less of a
> > problem than receiving data forever.
>
> This isn't how Sendmail works.  The entire message is cached to the
> queue before milter is told anything about the headers or body.
> There's no "a few seconds ahead", it's all the way ahead.  Milter
> has no opportunity to say REJECT in the middle of the SMTP DATA
> phase because the filter doesn't even know that's where the MTA is.
>
> Read the code in srvrsmtp.c and note that collect() is called before
> milter_data(); the former collects the entire message from the
> incoming stream, and the latter is what sends header and body
> information down to the filters.
>
> > I just did an experiment with Sendmail using telnet as the client.  If
> > I
> > send one line of data, then pause, Sendmail waits 10 minutes then
> > disconnects with no message to telnet.  There is no timeout in telnet,
> > so it just sits there until I force a disconnect on that end.
>
> That's an MTA timeout and has nothing to do with milter.  Upon
> reaching this timeout, the filter has received the envelope data but
> none of the header or body, and will be given an "abort" instruction.
>
>
> ------------------------------------------------------------------------------
> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> is the only developer event you need to attend this year. Jumpstart your
> developing skills, take BlackBerry mobile applications to market and stay
> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> http://p.sf.net/sfu/devconference
> _______________________________________________
> courier-users mailing list
> courier-users@...
> Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users

--
Matus UHLAR - fantomas, uhlar@... ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I feel like I'm diagonally parked in a parallel universe.

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
courier-users mailing list
courier-users@...
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users

[Gordon Messmer]

On 10/29/2009 07:07 AM, Matus UHLAR - fantomas wrote:

> On 23.10.09 08:55, Alessandro Vesely wrote:
>    
>> there has been some discussion some time ago about the possibility
>> to implement a milter-compatible library for courier. At the time, I
>> didn't know how milter actually works. Now I have one more reason to
>> be happy for using courier!
>>      
> I somehow don't understand what you mean.
>
> I would invite milter interface for courier. The present filtering
> capability can't modify the mail (or can it already?) which would be nice
> for integrating e.g. spamassassin, DCC, DKIM and probably other software and
> techniques.
>    

It can.  Pythonfilter has a framework for filters that will modify
messages.  Even when Courier didn't support it directly, it was possible
to inject a new message with the required modifications and mark the
original complete (pythonfilter could do that, too).

I've contemplated adding a milter server interface to pythonfilter so
that Courier users could take advantage of existing milter software.  As
far as I know, the only interesting uses would be commercial spam/virus
scanners, though.  I don't think I've ever heard any real interest in
it, so it's way down on my priority list.

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
courier-users mailing list
courier-users@...
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users

[Ale2008]

Matus UHLAR - fantomas wrote:
> On 23.10.09 08:55, Alessandro Vesely wrote:
>> there has been some discussion some time ago about the possibility
>> to implement a milter-compatible library for courier. At the time, I
>> didn't know how milter actually works. Now I have one more reason to
>> be happy for using courier!
>
> I somehow don't understand what you mean.
> [...]
> And how is this related to the mail cited below?

I never actually used libmilter. When I looked at its reference I had
been fooled by the calls' names --xxfi_header, xxfi_eoh-- into
believing they were occurring "live" while receiving the body. The
message cited said how sendmail works, correcting my wrong opinion.

Although conceptually easier than I thought, a milter compatibility
layer still requires a good deal of coding, though. It would result in
duplicating many existing functionalities, e.g. header parsing...

>> -------- Original Message --------
>> From: Murray S. Kucherawy <msk@...>
>>
>> This isn't how Sendmail works.  The entire message is cached to the
>> queue before milter is told anything about the headers or body.

For example, if one wanted to add DKIM support in C, it would seem
easier and safer to write a courier global filter using libopendkim
than write a milter compatibility layer for using the opendkim milter
module.





























------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
courier-users mailing list
courier-users@...
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users

[Matus UHLAR - fantomas]

> > On 23.10.09 08:55, Alessandro Vesely wrote:
> >> there has been some discussion some time ago about the possibility
> >> to implement a milter-compatible library for courier. At the time, I
> >> didn't know how milter actually works. Now I have one more reason to
> >> be happy for using courier!

> Matus UHLAR - fantomas wrote:
> > I somehow don't understand what you mean.
> > [...]
> > And how is this related to the mail cited below?

On 30.10.09 14:19, Alessandro Vesely wrote:
> I never actually used libmilter. When I looked at its reference I had
> been fooled by the calls' names --xxfi_header, xxfi_eoh-- into
> believing they were occurring "live" while receiving the body. The
> message cited said how sendmail works, correcting my wrong opinion.

Aha, I understand now :-)

> Although conceptually easier than I thought, a milter compatibility
> layer still requires a good deal of coding, though. It would result in
> duplicating many existing functionalities, e.g. header parsing...

functionalities existing where?

incorporating milter would ease much of work since milter interface is
available for sendmail and postfix and there are many milters available. I
haven't checked if there are already available alternatives for courier but
there are many milters available already...

> >> -------- Original Message --------
> >> From: Murray S. Kucherawy <msk@...>
> >>
> >> This isn't how Sendmail works.  The entire message is cached to the
> >> queue before milter is told anything about the headers or body.
>
> For example, if one wanted to add DKIM support in C, it would seem
> easier and safer to write a courier global filter using libopendkim
> than write a milter compatibility layer for using the opendkim milter
> module.

does the same apply for multiple filters like spamassassin etc? All they may
need modifying of the e-mail.

--
Matus UHLAR - fantomas, uhlar@... ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Eagles may soar, but weasels don't get sucked into jet engines.

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
courier-users mailing list
courier-users@...
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users

[Ale2008]

Matus UHLAR - fantomas wrote:
>> Although conceptually easier than I thought, a milter compatibility
>> layer still requires a good deal of coding, though. It would result in
>> duplicating many existing functionalities, e.g. header parsing...
>
> functionalities existing where?

For headers, each filter, as well as the MTA software itself, has
its own functions. In addition, there may be a number of assumptions
at both linkage and command execution layers, e.g. DNS lookups and
the sendmail executable. That is to say, it should be checked that
those milters don't rely on further peculiarities of sendmail than
the milter interface proper.

> incorporating milter would ease much of work since milter interface is
> available for sendmail and postfix and there are many milters available. I
> haven't checked if there are already available alternatives for courier but
> there are many milters available already...

I haven't actually tried to run the configure script of a milter. I
could only try to guess where one could find a break-even between
porting each of N given milters versus porting the milter interface.
For N <= 2, I'd still guess it's easier to port each milter...

>> For example, if one wanted to add DKIM support in C, it would seem
>> easier and safer to write a courier global filter using libopendkim
>> than write a milter compatibility layer for using the opendkim milter
>> module.
>
> does the same apply for multiple filters like spamassassin etc? All they may
> need modifying of the e-mail.

SpamAssassin is a different thing. It is commonly run from maildrop
recipes for a number of reasons, including using personal Bayes data
for each recipient, and concealing filtering results from the sender.

At any rate, to close files before invoking global filters, so as to
allow them to alter the message, was done in October 2007. See
http://markmail.org/message/7clmkipmcvaw4ini






























------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
courier-users mailing list
courier-users@...
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users

[Matus UHLAR - fantomas]

> >> Although conceptually easier than I thought, a milter compatibility
> >> layer still requires a good deal of coding, though. It would result in
> >> duplicating many existing functionalities, e.g. header parsing...

> Matus UHLAR - fantomas wrote:
> > functionalities existing where?

On 01.11.09 15:53, Alessandro Vesely wrote:
> For headers, each filter, as well as the MTA software itself, has
> its own functions. In addition, there may be a number of assumptions
> at both linkage and command execution layers, e.g. DNS lookups and
> the sendmail executable. That is to say, it should be checked that
> those milters don't rely on further peculiarities of sendmail than
> the milter interface proper.

we could check if those milters work with postfix (some do) and if they have
changes towards postfix compatibility in ChangeLog ,,,
 
> > incorporating milter would ease much of work since milter interface is
> > available for sendmail and postfix and there are many milters available. I
> > haven't checked if there are already available alternatives for courier but
> > there are many milters available already...
>
> I haven't actually tried to run the configure script of a milter. I
> could only try to guess where one could find a break-even between
> porting each of N given milters versus porting the milter interface.
> For N <= 2, I'd still guess it's easier to port each milter...

for me, the N is currently 3-4 :-) (SA, clamav, DKIM, maybe DCC greylist)

> >> For example, if one wanted to add DKIM support in C, it would seem
> >> easier and safer to write a courier global filter using libopendkim
> >> than write a milter compatibility layer for using the opendkim milter
> >> module.
> >
> > does the same apply for multiple filters like spamassassin etc? All they may
> > need modifying of the e-mail.
>
> SpamAssassin is a different thing. It is commonly run from maildrop
> recipes for a number of reasons, including using personal Bayes data
> for each recipient, and concealing filtering results from the sender.
>
> At any rate, to close files before invoking global filters, so as to
> allow them to alter the message, was done in October 2007. See
> http://markmail.org/message/7clmkipmcvaw4ini

If there's only one recipient, SA can use his rules too. If there are more
of them, a special recipient with safe options can be defined and mail can
be re-scored in users' maildrop/procmail. Rejecting spam at SMTP level is
better than saving to spam folder...

--
Matus UHLAR - fantomas, uhlar@... ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I intend to live forever - so far so good.

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
courier-users mailing list
courier-users@...
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users

[Ale2008]

Matus UHLAR - fantomas wrote:
>> That is to say, it should be checked that
>> those milters don't rely on further peculiarities of sendmail than
>> the milter interface proper.
>
> we could check if those milters work with postfix (some do) and if they have
> changes towards postfix compatibility in ChangeLog ,,,

I only looked at opendkim. I found no ChangeLog. However, that milter
uses functions such as smfi_getsymval to look up symbols like
"{daemon_name}" and "{auth_type}", in order to establish whether a
message has been submitted in a MSA-like fashion and hence has to be
signed. Although libmilter is presented in a MTA-neutral fashion, the
doc at https://www.milter.org/developers/api/smfi_getsymval defines
that function as

    char* smfi_getsymval(SMFICTX *ctx, char *symname);

    *Get the value of a sendmail macro.*

http://www.postfix.org/MILTER_README.html#macros also documents this
use. It requires either a full blown sendmail emulation layer, or a
thorough analysis of the milter to determine how to make it work with
minimal efforts. For example, the xxfi_envrcpt callback is only used
to check per-recipient signing options; as it always returns
SMFIS_CONTINUE, it may more easily be called from a global filter with
prerecorded values. Such slovenly global filter might then also work
for spamd, but not for greylisting.

































------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
courier-users mailing list
courier-users@...
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users