Using pdns-recursor on a network with high latency.

[Andre Recnik]

I have been using PowerDNS and the PowerDNS recursor to host two name
servers, for the last three months, in a 30+ user environment with no
problems.

Recently my ISP has been doing some 'upgrades' which as caused very high
latency in the last few days.  This latency is between 2.1 and 5.6
seconds (as seen in a trace route to outside my network).

As a result almost all look-ups are failing - and I have had to use BIND
as my recursor (which for some reason works even with the high latency).

I'm assuming that there is a built in timeout system in pdns-recursor,
and that this is causing the lookups to fail - however I can't find a
way to change this value (at least not without source modification).

Is there a config file option that would let me change the timeout (or
number of tries)?   Should that be added as a feature to future
releases?   Or is this perhaps the result of some other bug or feature
in pdns-recursor?

Thanks,
Andre
_______________________________________________
Pdns-users mailing list
Pdns-users@...
http://mailman.powerdns.com/mailman/listinfo/pdns-users

[Sten Spans]

On Wed, 16 Sep 2009, Andre Recnik wrote:

> Recently my ISP has been doing some 'upgrades' which as caused very high
> latency in the last few days.  This latency is between 2.1 and 5.6 seconds
> (as seen in a trace route to outside my network).
>
> As a result almost all look-ups are failing - and I have had to use BIND as
> my recursor (which for some reason works even with the high latency).

This is indeed something that pdns-recursor doesn't handle very well.

> I'm assuming that there is a built in timeout system in pdns-recursor, and
> that this is causing the lookups to fail - however I can't find a way to
> change this value (at least not without source modification).
>
> Is there a config file option that would let me change the timeout (or number
> of tries)?   Should that be added as a feature to future releases?   Or is
> this perhaps the result of some other bug or feature in pdns-recursor?

As far as I know there currently is no way to improve powerdns-recursor's
behaviour, short of modifying the source. I think Bert would appreciate a
well tested patch which improves the behaviour, although only if it doesn't
hurt lookup performance. It might be reasonable to add an "high-latency"
option, or even better use the initial root-hints lookup and prehaps later
ones to auto-tune a reasonable timeout value.

--
Sten Spans

"There is a crack in everything, that's how the light gets in."
Leonard Cohen - Anthem
_______________________________________________
Pdns-users mailing list
Pdns-users@...
http://mailman.powerdns.com/mailman/listinfo/pdns-users

[bert hubert-2]

On Fri, Sep 18, 2009 at 9:16 AM, Sten Spans <sten@...> wrote:
> As far as I know there currently is no way to improve powerdns-recursor's
> behaviour, short of modifying the source. I think Bert would appreciate a
> well tested patch which improves the behaviour, although only if it doesn't
> hurt lookup performance. It might be reasonable to add an "high-latency"
> option, or even better use the initial root-hints lookup and prehaps later
> ones to auto-tune a reasonable timeout value.

Sten, Andre,

I'm currently looking in to this and I see why this has not been fixed
- the timeout mechanism is rather crude, and may in fact not even be
doing exactly what it is supposed to be doing. It is supposed to
deliver a '1000ms' timeout right now, but it appear it is only doing
that on average, sometimes doing a 0ms timeout and sometimes a 2000ms
timeout.

So I suspect this would be hard for a third party to patch. I'm
looking into this to see if there IS a bug, and if there could be a
tunable timeout somewhere.

On a related note, both BIND and djbdns decide to keep on sending
questions until they get an answer, behaviour I do not find to be
warranted for normal operations. PowerDNS effectively tries not to be
an amplifier, and is thus slightly more prone to sending timeouts. In
effect, PowerDNS delivers a lower percentage of timeouts to clients in
the 'normal case', so this is not a problem.

    Bert
_______________________________________________
Pdns-users mailing list
Pdns-users@...
http://mailman.powerdns.com/mailman/listinfo/pdns-users

[bert hubert-2]

On Fri, Sep 18, 2009 at 10:40 AM, bert hubert <bert.hubert@...> wrote:
> I'm currently looking in to this and I see why this has not been fixed
> - the timeout mechanism is rather crude, and may in fact not even be
> doing exactly what it is supposed to be doing. It is supposed to
> deliver a '1000ms' timeout right now, but it appear it is only doing
> that on average, sometimes doing a 0ms timeout and sometimes a 2000ms
> timeout.

Gentelemen,

Could you perhaps try
http://svn.powerdns.com/snapshots/pdns-recursor-3.1.8-testing.tar.bz2
?
It has a default timeout of 1.5 seconds now, and this can be tuned
using the 'network-timeout' setting (in milliseconds).

Please let me know if it solves the problem.

I might make this auto-tuning.

    Bert
_______________________________________________
Pdns-users mailing list
Pdns-users@...
http://mailman.powerdns.com/mailman/listinfo/pdns-users

[Andre Recnik]

On 09/22/2009 06:37 AM, bert hubert wrote:

> Gentelemen,
>
> Could you perhaps try
> http://svn.powerdns.com/snapshots/pdns-recursor-3.1.8-testing.tar.bz2
> ?
> It has a default timeout of 1.5 seconds now, and this can be tuned
> using the 'network-timeout' setting (in milliseconds).
>
> Please let me know if it solves the problem.
>
> I might make this auto-tuning.
>
>      Bert
>    

My network problem (the high latency) was fixed by the ISP (at least for
now), but I have tested the updated recursor with a latency simulator
(WANem).

The results are:

Works great for all sites I tested with up to 5600ms latency (provided I
set the 'network-timeout' to 6000).

I did not test beyond 5600ms of latency; I do not see much point, as
networks with that kind of latency are almost unusable anyway.

Having it auto-tuning would be nice, as that would save the problem of
needing to manually change it in a network event.

Thank you for the patch!

Andre





_______________________________________________
Pdns-users mailing list
Pdns-users@...
http://mailman.powerdns.com/mailman/listinfo/pdns-users