[Bug 736] New: exim 4.69 fails to use all MX records before failing

[David Favor]

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736
           Summary: exim 4.69 fails to use all MX records before failing
           Product: Exim
           Version: 4.69
          Platform: x86
        OS/Version: Linux
            Status: NEW
          Severity: bug
          Priority: high
         Component: Delivery in general
        AssignedTo: nigel@...
        ReportedBy: david@...
                CC: exim-dev@...


I'm testing exim and MX record processing.

I have two MX records - net1.coolsurf.com & net2.coolsurf.com where
net1 has an SMTP server running and net2 has none.

At least in version 4.69 it appears only net2 is tried. The MX priority
is net1 == 10 and net2 == 20.

As I understand the RFC, MX records should be tried in progressive order
and it appears here they are being tried backwards, as net1 is an async
version of qpsmtpd which will handle millions of connections and is up
during these errors.

Also, even if net1 & net2 both time out, in the case of the error below
no retry period was honored. The email was generated in a midnight cron
job (note the time stamp of 2008-07-14 00:00:17) and delivery appears to
only have been tried once, then returned as an error.

So there appear to be either 1 or 2 errors:

1) MX priority fails to be honored

2) SMTP mail delivery error is generated on first send attempt

_______


main.log:2008-07-14 00:00:17 1KII2j-00049Q-1U <= root@... U=root
P=local S=18464 T="MediaPlayer.swf - 14 July 2008 (Mon) -
http://www.jeroenwijering.com/?item=JW_FLV_Media_Player" from
<root@...> for david@...
main.log:2008-07-14 00:00:17 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc
1KII2j-00049Q-1U
main.log:2008-07-14 00:00:17 1KII2j-00049Q-1U net2.coolsurf.com [69.94.137.128]
Connection refused
main.log:2008-07-14 00:00:17 1KII2j-00049Q-1U == david@...
R=dnslookup T=remote_smtp defer (111): Connection refused
main.log:2008-07-14 00:00:17 1KII2j-00049Q-1U ** david@...: retry
timeout exceeded
main.log:2008-07-14 00:00:17 cwd=/var/spool/exim 7 args: /usr/sbin/exim -t -oem
-oi -f <> -E1KII2j-00049Q-1U
main.log:2008-07-14 00:00:17 1KII2j-00049T-3z <= <> R=1KII2j-00049Q-1U U=exim
P=local S=19259 T="Mail delivery failed: returning message to sender" from <>
for root@...
main.log:2008-07-14 00:00:17 1KII2j-00049Q-1U Completed


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736




--- Comment #1 from Nigel Metheringham <nigel@...>  2008-07-14 14:16:31 ---
Sounds suspiciously as though there is explicit routing in place rather than
MX.

Suggest you add the output of:-
    exim -d -bt david@...

as run by root (to enable debugging) on the affected machine.  Details of the
configuration would also be useful.

Might be better to discuss this on the exim-users list as turn around is
likely to be better than working through a bugzilla entry - however any
conclusions should be flagged here.


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

[David Favor]

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736




--- Comment #2 from David Favor <david@...>  2008-07-14 15:04:07 ---
Config is:
_______

#
# Research this statement
#
# You want to ensure that Routing is done ahead of time; -odqs will do
# this, or two-stage queueing with -qq (thanks Graeme for the reminder)
# or using queue_smtp_domains.

primary_hostname  = coolsurf.com
qualify_domain    = coolsurf.com
qualify_recipient = coolsurf.com

#domainlist local_domains = coolsurf.com : davidfavor.com : radicalhealth.com :
livefeast.com
domainlist local_domains =
domainlist relay_to_domains =
hostlist relay_from_hosts = 127.0.0.1 : 192.168.0.0/16

freeze_tell = postmaster
ignore_bounce_errors_after = 2d
timeout_frozen_after = 7d

split_spool_directory = true

log_selector = +address_rewrite +all_parents +arguments +connection_reject
+delay_delivery +delivery_size +dnslist_defer +incoming_interface
+incoming_port +lost_incoming_connection +queue_run +received_sender
+received_recipients +retry_defer +sender_on_delivery +size_reject
+skip_delivery +smtp_confirmation +smtp_connection +smtp_protocol_error
+smtp_syntax_error +subject +tls_cipher +tls_peerdn

### BACKSCATTER PROCESSING ###
### http://psg.com/~brian/software/authbounce/configure-authbounce.txt

# A unique key for your company, e.g. a domain you own and a serial number
# to allow the key to be changed in future
BOUNCE_ID = coolsurf.com-1

# A genuinely random key. e.g. from
#   head -c16 /dev/urandom | md5
# If running across multiple mailservers they will all need to share the
# same key, and you will need to keep it secret from anyone else.
BOUNCE_SECRET = f11825b7da99c1f6798f8d950dd98953

begin routers

# all messages go to their MX defined server
dnslookup:
  driver    = dnslookup
  self      = send
  transport = remote_smtp

begin transports

#  max_rcpt = 1
remote_smtp:
  driver = smtp
  serialize_hosts = *
  headers_add = X-ID:
${lc:$local_part@$domain};${length_8:${md5:${lc:$local_part@$domain};BOUNCE_SECRET}};${length_8:BOUNCE_SECRET}

begin retry

# Address or Domain    Error       Retries
# -----------------    -----       -------
*                      *           F,2h,15m; G,16h,1h,1.5; F,4d,6h
_______

net1#exim -d -bt david@...
Exim version 4.69 uid=0 gid=0 pid=20800 D=fbb95cfd
Berkeley DB: Berkeley DB 4.6.21: (October 11, 2007)
Support for: crypteq iconv() IPv6 PAM Perl Expand_dlfunc TCPwrappers OpenSSL
Content_Scanning Old_Demime
Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmnz
dnsdb dsearch ldap ldapdn ldapm nis nis0 nisplus passwd sqlite
Authenticators: cram_md5 cyrus_sasl dovecot plaintext spa
Routers: accept dnslookup ipliteral manualroute queryprogram redirect
Transports: appendfile/maildir/mailstore/mbx autoreply lmtp pipe smtp
Fixed never_users: 0
Size of off_t: 4
changed uid/gid: forcing real = effective
  uid=0 gid=0 pid=20800
  auxiliary group list: <none>
configuration file is /etc/exim/exim.conf
log selectors = 0000dfff 006bb86b
Reset TZ to NULL: time is 2008-07-14 06:26:21
Loading lookup modules from /usr/lib/exim/4.69-5.fc8/lookups
Loaded 0 lookup modules
Total 17 lookups
trusted user
admin user
originator: uid=0 gid=0 login=root name=root
sender address = root@...
Address testing: uid=0 gid=93 euid=0 egid=93
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Testing david@...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Considering david@...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
routing david@...
--------> dnslookup router <--------
local_part=david domain=davidfavor.com
calling dnslookup router
dnslookup router called for david@...
  domain = davidfavor.com
DNS lookup of davidfavor.com (MX) succeeded
DNS lookup of net1.coolsurf.com (AAAA) gave NO_DATA
returning DNS_NODATA
DNS lookup of net1.coolsurf.com (A) succeeded
DNS lookup of net2.coolsurf.com (AAAA) gave NO_DATA
returning DNS_NODATA
DNS lookup of net2.coolsurf.com (A) succeeded
local host has lowest MX
fully qualified name = davidfavor.com
host_find_bydns yield = HOST_FOUND_LOCAL (3); returned hosts:
  net1.coolsurf.com 69.94.137.124 MX=10
  net2.coolsurf.com 69.94.137.128 MX=20
lowest numbered MX record points to local host: davidfavor.com: configured to
try delivery anyway
set transport remote_smtp
queued for remote_smtp transport: local_part = david
domain = davidfavor.com
  errors_to=NULL
  domain_data=NULL localpart_data=NULL
routed by dnslookup router
  envelope to: david@...
  transport: remote_smtp
  host net1.coolsurf.com [69.94.137.124] MX=10
  host net2.coolsurf.com [69.94.137.128] MX=20
david@...
  router = dnslookup, transport = remote_smtp
  host net1.coolsurf.com [69.94.137.124] MX=10
  host net2.coolsurf.com [69.94.137.128] MX=20
search_tidyup called
>>>>>>>>>>>>>>>> Exim pid=20800 terminating with rc=0 >>>>>>>>>>>>>>>>


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

On Mon, 14 Jul 2008, Nigel Metheringham wrote:

> Might be better to discuss this on the exim-users list as turn around is
> likely to be better than working through a bugzilla entry - however any
> conclusions should be flagged here.

Have you checked hosts_max_try? See section 30.5.

--
Philip Hazel

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736




--- Comment #3 from Philip Hazel <ph10@...>  2008-07-14 15:40:51 ---
On Mon, 14 Jul 2008, Nigel Metheringham wrote:

> Might be better to discuss this on the exim-users list as turn around is
> likely to be better than working through a bugzilla entry - however any
> conclusions should be flagged here.

Have you checked hosts_max_try? See section 30.5.


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

[David Favor]

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736




--- Comment #4 from David Favor <david@...>  2008-07-14 18:44:28 ---
It's unclear how this might effect my case or what value to set here.

I've set hosts_max_try = 100 as a test.

Any other suggestions are welcome, as I now see this same situation
occurring with several domains. These domains are on multi-homed machines
and they are the only domain being hit, so all I can figure is maybe the
retry count is being exceeded by previously queued messages which are
effecting the retry count.

I'm new to exim, so I'm just guessing.


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

On Mon, 14 Jul 2008, David Favor wrote:

> I've set hosts_max_try = 100 as a test.
>
> Any other suggestions are welcome, as I now see this same situation
> occurring with several domains. These domains are on multi-homed machines
> and they are the only domain being hit, so all I can figure is maybe the
> retry count is being exceeded by previously queued messages which are
> effecting the retry count.
>
> I'm new to exim, so I'm just guessing.

Check the retry times (see the exinext utility). Maybe that's the clue
that you need. If there are 2 MX records, for hosts A and B, but host A
failed recently, Exim will not try it again until its retry time comes -
even for a new message.

Try a delivery with retry time override: put a message on the queue
using -odq and then force a delivery using -M (check these options out
in the manual).


--
Philip Hazel

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736




--- Comment #5 from Philip Hazel <ph10@...>  2008-07-15 09:17:32 ---
On Mon, 14 Jul 2008, David Favor wrote:

> I've set hosts_max_try = 100 as a test.
>
> Any other suggestions are welcome, as I now see this same situation
> occurring with several domains. These domains are on multi-homed machines
> and they are the only domain being hit, so all I can figure is maybe the
> retry count is being exceeded by previously queued messages which are
> effecting the retry count.
>
> I'm new to exim, so I'm just guessing.

Check the retry times (see the exinext utility). Maybe that's the clue
that you need. If there are 2 MX records, for hosts A and B, but host A
failed recently, Exim will not try it again until its retry time comes -
even for a new message.

Try a delivery with retry time override: put a message on the queue
using -odq and then force a delivery using -M (check these options out
in the manual).


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

[David Favor]

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736




--- Comment #6 from David Favor <david@...>  2008-07-15 22:51:27 ---
Even with hosts_max_try = 100 the same error persists.

Using 'exinext david@...' yields an error for net2.

There is no error listed for the net1 host, so I'm still unclear
why net1 with a higher MX priority, able to tolerate 100,000s of
message per minute, in the middle of the night (midnight) is being
passed over as the first MX record to use.

Suggestions appreciated. Thanks.


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736

Phil Pennock <exim-dev@...> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |exim-dev@...




--- Comment #7 from Phil Pennock <exim-dev@...>  2008-07-17 00:40:56 ---
You have serialize_hosts set on the transport to match everything, so there can
only be one connection at a time to a given host, controlled by hints.

Is there another exim process running (exiwhat should tell you this)?

Did the system reboot and did you set up the DB hints nuking recommended in the
documentation of serialize_hosts ?

"exim_dumpdb /var/spool/exim misc" might be informative.

Note that serialize_hosts is a rather rare option to see (in my experience)
since most people seem happy to have a few concurrent connections per host so
that if one stalls (eg, antivirus on the remote side gets unhappy at a message)
it doesn't stop the entire flow.  You have an option here explicitly designed
to prevent delivery to the remote host.

I'm also curious about hint lifetime and if you had any stale hints from before
you added "self = send" to the configuration to permit the host to send to
itself, as the primary MX.

It would be useful to see a debugging run for the actual delivery of a mail,
"exim -d -odf -t" and supply a mail on stdin; that should show the debugging
for the use of the transport and would let you/us see why the preferred MX is
skipped.  Perhaps -d+transport+hints_lookup+host_lookup to get more data if the
initial test doesn't provide enough information.


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

[David Favor]

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736




--- Comment #8 from David Favor <david@...>  2008-07-17 01:07:07 ---
It escapes me why I put in serialize hosts. I've removed it.

'exim_dumpdb /var/spool/exim misc' yields no output.

I'll try the manual mail delivery in a bit.


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736




--- Comment #9 from Phil Pennock <exim-dev@...>  2008-08-13 05:24:23 ---
David, has this been resolved to your satisfaction or are you still
experiencing problems?

Thanks,
-Phil


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

[David Favor]

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736




--- Comment #10 from David Favor <david@...>  2008-08-13 13:28:20 ---
Phil Pennock wrote:

> ------- You are receiving this mail because: -------
> You reported the bug.
>
> http://bugs.exim.org/show_bug.cgi?id=736
>
>
>
>
> --- Comment #9 from Phil Pennock <exim-dev@...>  2008-08-13 05:24:23 ---
> David, has this been resolved to your satisfaction or are you still
> experiencing problems?
>
> Thanks,
> -Phil
>
>

This problem persists. Every few days I'll get sometimes 2-3 messages
which bounce because of this.


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

[Tony Finch]

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736

Tony Finch <dot@...> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dot@...




--- Comment #11 from Tony Finch <dot@...>  2008-08-13 14:06:34 ---
Could you please try getting debugging output from one of these failed
deliveries?

I suspect the problem is related to the fact that the lowest numbered MX points
to the local host, though you have the self=send option.


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

[David Favor]

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736




--- Comment #12 from David Favor <david@...>  2008-08-13 14:27:33 ---
Tony Finch wrote:

> ------- You are receiving this mail because: -------
> You reported the bug.
>
> http://bugs.exim.org/show_bug.cgi?id=736
>
> Tony Finch <dot@...> changed:
>
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |dot@...
>
>
>
>
> --- Comment #11 from Tony Finch <dot@...>  2008-08-13 14:06:34 ---
> Could you please try getting debugging output from one of these failed
> deliveries?
>
> I suspect the problem is related to the fact that the lowest numbered MX points
> to the local host, though you have the self=send option.

I do have self=send set.

And I agree. This only seems to happen in a cron job, sending email on
the localhost to a user on the localhost.

Since this only occurs at night when cron is pretty much the only thing
running, it appears to be something other than a load dependency.

If you can tell me exactly how to setup logging to assist resolving this
problem, I'll set it up and send you the logs next time it happens.


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736




--- Comment #13 from Phil Pennock <exim-dev@...>  2008-08-14 10:29:19 ---
What were the results of the manual mail delivery per comment #7 on 2008-07-17
(and your reply to that)?


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

[Tony Finch]

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736




--- Comment #14 from Tony Finch <dot@...>  2008-08-14 11:47:14 ---
OK, try sending email as if from cron but with debugging turned on, e.g. (as
root)

echo test | exim -d+all root


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

[David Favor]

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736




--- Comment #15 from David Favor <david@...>  2008-08-14 12:15:33 ---
Answer to comment #13 is in comment #8.

This command produces no output.


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

[David Favor]

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736




--- Comment #16 from David Favor <david@...>  2008-08-14 12:16:58 ---
Answer to comment #14.

Click http://radicalhealth.com/exim/log.txt for this command's output.


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

[Tony Finch]

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=736




--- Comment #17 from Tony Finch <dot@...>  2008-08-14 12:49:11 ---
That debugging output doesn't exhibit the problem. What happens if you run
echo test | exim -d david@...
instead (using the address that failed in the original bug report).


--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##