temporary errors (LDAP failures) while routing

Hello!

Consider a scenario where Exim is relaying mail for _one_ domain to
multiple internal systems with queries to LDAP directories
deciding where mail for each local part goes to.
(Suppose that it's not possible to regularly create a local
version of this mapping, as debatable this restriction may be.)

If one implements this the straight forward way with multiple routers
like

dest_a:
  driver = manualroute
  domains = example.com
  condition = ${lookup ldapm{..URL..}{yes}{no}}
  route_list = * host_a bydns
  transport = internal_smtp

dest_b:
  ...


the following problem is not trivial to solve it seems:

As "9.5 Temporary errors in lookups" of the spec states
  Lookup functions can return temporary error codes if the lookup cannot be
  completed. For example, an SQL or LDAP database might be unavailable. [...]
  When a lookup cannot be completed in a router or transport, delivery of the
  message (to the relevant address) is deferred, as for any other temporary
  error. In other circumstances Exim may assume the lookup has failed, or may
  give up altogether. -
if _one_ destination system is not available, messages to _every_ user in this
domain will be deferred (even those at other destinations).

For two destinations it's a workaround to have two pair of routers with
order swapped, and
  condition = ${if and {{first_delivery}{${lookup ...}}}}
in the first pair (!first_delivery in the second), additionally tuning retry
rules.

Any idea for an Exim internal solution (no queryprogram router, run or perl)
for more than two destinations? (Using $message_age to extend the concept is
not reliable enough, I think.)

Thanks,
  Lutz


--
Lutz Preßler  <Lutz.Pressler@...>    http://www.SerNet.DE/
SerNet Service Network GmbH, Bahnhofsallee 1b, D-37081 Göttingen
Tel.: [phone],      FAX: [phone]
AG Göttingen, HRB 2816,      GF: Dr. Johannes Loxen

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Phil Pennock-9]

On 2009-07-14 at 17:25 +0200, Lutz Preßler wrote:

> dest_a:
>   driver = manualroute
>   domains = example.com
>   condition = ${lookup ldapm{..URL..}{yes}{no}}
>   route_list = * host_a bydns
>   transport = internal_smtp
>
> dest_b:
>   ...
>
>
> the following problem is not trivial to solve it seems:
>
> As "9.5 Temporary errors in lookups" of the spec states
>   Lookup functions can return temporary error codes if the lookup cannot be
>   completed. For example, an SQL or LDAP database might be unavailable. [...]
>   When a lookup cannot be completed in a router or transport, delivery of the
>   message (to the relevant address) is deferred, as for any other temporary
>   error. In other circumstances Exim may assume the lookup has failed, or may
>   give up altogether. -
> if _one_ destination system is not available, messages to _every_ user in this
> domain will be deferred (even those at other destinations).

I believe that this conclusion is false.

It's for precisely the problem of not knowing how other LHSs will be
routed that the dnslookup and manualroute Routers have a
"same_domain_copy_routing" option, which defaults to false.

As long as you don't set same_domain_copy_routing then you should be
fine.

-Phil

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Hello Phil,
On Tue, 14 Jul 2009, Phil Pennock wrote:

> On 2009-07-14 at 17:25 +0200, Lutz Preßler wrote:
> > dest_a:
> >   driver = manualroute
> >   domains = example.com
> >   condition = ${lookup ldapm{..URL..}{yes}{no}}
> >   route_list = * host_a bydns
> >   transport = internal_smtp
> >
> > dest_b:
> >   ...
> >
> >
> > the following problem is not trivial to solve it seems:
> >
> > As "9.5 Temporary errors in lookups" of the spec states
> >   Lookup functions can return temporary error codes if the lookup cannot be

[...]
> > if _one_ destination system is not available, messages to _every_ user in this
> > domain will be deferred (even those at other destinations).
[...]
> It's for precisely the problem of not knowing how other LHSs will be
> routed that the dnslookup and manualroute Routers have a
> "same_domain_copy_routing" option, which defaults to false.
That's why multiple routers can route the same destination domain - which
works, of course. But if the lookup times out in the first router
while figuring out if it should handle this local part, the message
is deferred immediately without doing the lookups in the other router(s).

Lutz

--
Lutz Preßler  <Lutz.Pressler@...>    http://www.SerNet.DE/
SerNet Service Network GmbH, Bahnhofsallee 1b, D-37081 Göttingen
Tel.: [phone],      FAX: [phone]
AG Göttingen, HRB 2816,      GF: Dr. Johannes Loxen

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

[Phil Pennock-9]

On 2009-07-15 at 07:53 +0200, Lutz Preßler wrote:
> That's why multiple routers can route the same destination domain - which
> works, of course. But if the lookup times out in the first router
> while figuring out if it should handle this local part, the message
> is deferred immediately without doing the lookups in the other router(s).

Okay, that's a different issue from what I understood you to be saying
in the original post.

Yes, the general idea is that mail routing should be deterministic and
if you can't figure out if "X" should handle a mail, it's not safe to
just carry on.

Generally, if things have scaled to where you want to use LDAP then
hopefully you have more than one LDAP server anyway, using some
read-only replicas, perhaps even one local to the mail-server.  So you
specify multiple servers in ldap_default_server and various types of
failure (documented in the LDAP stuff in The Exim Specification,
"spec.txt" and so on) will cause fail-through to the next server.

If all your LDAP servers go down, then good design means that will have
to be some kind of network or DNS issue and in either of those cases,
holding the mail is the right thing to do anyway.

-Phil

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

On Wed, 15 Jul 2009, Phil Pennock wrote:
> On 2009-07-15 at 07:53 +0200, Lutz Preßler wrote:
[...]
> > works, of course. But if the lookup times out in the first router
> > while figuring out if it should handle this local part, the message
> > is deferred immediately without doing the lookups in the other router(s).
[...]
> Yes, the general idea is that mail routing should be deterministic and
> if you can't figure out if "X" should handle a mail, it's not safe to
> just carry on.
Yes, sure. This special case is different though as potentially conflicting
local parts at the destination systems are dealt with administratively
(or at least seen as a smaller problem than deferral of all incoming mail).
>
> Generally, if things have scaled to where you want to use LDAP then
> hopefully you have more than one LDAP server anyway, using some
> read-only replicas, perhaps even one local to the mail-server.
(Not possible in practice.)
[...]
> If all your LDAP servers go down, then good design means that will have
> to be some kind of network or DNS issue and in either of those cases,
> holding the mail is the right thing to do anyway.
It's not good design at all, but part of helping to deal with technical
(and organizational) restrictions (including network structure) hopefully
leading to a well designed overall system later.
That's why I am not proposing to change lookup errori handling in
Exim (even though a way to disable automatic deferral and e.g. a
variable to access the last lookup's status would have been
nice), but just hoped that someone would come up with a clever trick
I haven't seen. I'll probably go the embedded perl route.
 
Lutz

--
Lutz Preßler  <Lutz.Pressler@...>    http://www.SerNet.DE/
SerNet Service Network GmbH, Bahnhofsallee 1b, D-37081 Göttingen
Tel.: [phone],      FAX: [phone]
AG Göttingen, HRB 2816,      GF: Dr. Johannes Loxen

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users 
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/