UDP Connection Table Exhaustion?

[Matthew Walster - Gyron]

Hey there,

I've inherited a DNS recursive resolver infrastructure based on PowerDNS recursor, and there are reports of strange DNS errors occasionally - either NXDOMAIN or just timeouts.

Looking into it, ip_conntrack seems to think there are currently 30000 connections, with a connection table size of 65535. Some users do have high figures, such as 1000 for just one IP, but on the large part, most IPs have low connection quantities.

I've tried changing the UDP timeout to 5 seconds, and this alleviated some of the issues, but didn't wholly fix it. Unfortunately, the software is currently running on CentOS 4.7, PowerDNS recursor version 3.1.4 - so a little older than I'd like, but it is only allowing queries from the CIDR ranges we have allocated to us so it shouldn't be too overloaded. Load averages are in the 0.10-0.25 range, and memory usage is 1GB of 3GB RAM. It's currently running on VMWare, but it will eventually run on a blade or similar.

Does anyone have any tips and/or tricks for running a medium-scale DNS recursive resolver appropriate to my situation? Medium being bigger than "run it off a DSL router" but smaller than "get a server farm to do it"!

Kind regards,

Matthew Walster

Network Architect
Gyron Internet Ltd

mail:   matthew.walster@...
web:    http://gyron.net/

tel:    [phone]
fax:    [phone]



This message may be private and confidential. If you have received this message in error, please notify us and remove it from your system.

Gyron may monitor email traffic data and the content of email for the purposes of security and staff training.

Gyron Internet Ltd is a limited company registered in England and Wales. Registered number: 4239332. Registered office: 3 Centro, Boundary Way, Hemel Hempsted, HP2 7SU. VAT reg no 804 2532 63. Gyron is a registered trademark.
_______________________________________________
Pdns-users mailing list
Pdns-users@...
http://mailman.powerdns.com/mailman/listinfo/pdns-users

[Sten Spans]

On Thu, 2 Jul 2009, Matthew Walster - Gyron wrote:

> Hey there,
>
> Does anyone have any tips and/or tricks for running a medium-scale DNS
> recursive resolver appropriate to my situation? Medium being bigger than
> "run it off a DSL router" but smaller than "get a server farm to do it"!

1- make sure you configure a limit for max-cache-entries
    otherwise it will keep growing and consume all memory.
    100k - 500k should cover most regular servers.
2- don't load any iptables modules if at all possible,
    the state tracking causes serious (performance) problems
    on loaded servers.
3- make sure to explicitly configure incoming and outgoing ips,
    preferably different ones. This enables later load-balancing / anycast
    schemes and makes the traffic-flow predictable. If your network
    layout is a bit more complicated (bgp, multiple gateways) then the
    linus arp_announce and arp_filter sysctls should be tweaked to
    make sure that Linux selects the correct source-ips.
4- use allow-from-file to configure the ranges that should be allowed
    to use your nameservers.

This should allow you to handle 5-10k queries on reasonable hardware
with a decent uplink (100mbit). Anything beyond that will require
compilation with a recent compiler and system specific tuning,
binding pdns to a specific cpu and the ethernet driver to another
for example. This kind of tuning should only be done with careful
measurements to test the effect of each change.

--
Sten Spans

"There is a crack in everything, that's how the light gets in."
Leonard Cohen - Anthem
_______________________________________________
Pdns-users mailing list
Pdns-users@...
http://mailman.powerdns.com/mailman/listinfo/pdns-users

[Matthew Walster - Gyron]

> -----Original Message-----
> From: Sten Spans [mailto:sten@...]
> Sent: 02 July 2009 17:04
> To: Matthew Walster - Gyron
> Cc: Pdns-users@...
> Subject: Re: [Pdns-users] UDP Connection Table Exhaustion?


> 2- don't load any iptables modules if at all possible,
>     the state tracking causes serious (performance) problems
>     on loaded servers.

I think this is the one I'm going to have to follow - the DNS server is currently on VMWare, with a 1000Mbit interface onto our core switching network, so that should be more than enough bandwidth, the state tracking is what is concerning me.

Thanks for your advice!

Matthew Walster



This message may be private and confidential. If you have received this message in error, please notify us and remove it from your system.

Gyron may monitor email traffic data and the content of email for the purposes of security and staff training.

Gyron Internet Ltd is a limited company registered in England and Wales. Registered number: 4239332. Registered office: 3 Centro, Boundary Way, Hemel Hempsted, HP2 7SU. VAT reg no 804 2532 63. Gyron is a registered trademark.
_______________________________________________
Pdns-users mailing list
Pdns-users@...
http://mailman.powerdns.com/mailman/listinfo/pdns-users

[bert hubert-2]

To nuance this a bit - on Linux, you can have great benefit from the
iptables 'NOTRACK' target, which can help you do firewalling that will
not run into problems from busy DNS traffic.

   Bert

On Fri, Jul 3, 2009 at 9:58 AM, Matthew Walster -
Gyron<matthew.walster@...> wrote:

>
>
>> -----Original Message-----
>> From: Sten Spans [mailto:sten@...]
>> Sent: 02 July 2009 17:04
>> To: Matthew Walster - Gyron
>> Cc: Pdns-users@...
>> Subject: Re: [Pdns-users] UDP Connection Table Exhaustion?
>
>
>> 2- don't load any iptables modules if at all possible,
>>     the state tracking causes serious (performance) problems
>>     on loaded servers.
>
> I think this is the one I'm going to have to follow - the DNS server is currently on VMWare, with a 1000Mbit interface onto our core switching network, so that should be more than enough bandwidth, the state tracking is what is concerning me.
>
> Thanks for your advice!
>
> Matthew Walster
>
>
>
> This message may be private and confidential. If you have received this message in error, please notify us and remove it from your system.
>
> Gyron may monitor email traffic data and the content of email for the purposes of security and staff training.
>
> Gyron Internet Ltd is a limited company registered in England and Wales. Registered number: 4239332. Registered office: 3 Centro, Boundary Way, Hemel Hempsted, HP2 7SU. VAT reg no 804 2532 63. Gyron is a registered trademark.
> _______________________________________________
> Pdns-users mailing list
> Pdns-users@...
> http://mailman.powerdns.com/mailman/listinfo/pdns-users
>
>
> !DSPAM:4a4dbaac25157472095810!
>
>

_______________________________________________
Pdns-users mailing list
Pdns-users@...
http://mailman.powerdns.com/mailman/listinfo/pdns-users

[Matthew Walster - Gyron]

> -----Original Message-----
> From: bert.hubert@... [mailto:bert.hubert@...] On Behalf Of
> bert hubert
> Sent: 03 July 2009 09:16
> To: Matthew Walster - Gyron
> Cc: Sten Spans; Pdns-users@...
> Subject: Re: [Pdns-users] UDP Connection Table Exhaustion?
>
> To nuance this a bit - on Linux, you can have great benefit from the
> iptables 'NOTRACK' target, which can help you do firewalling that will
> not run into problems from busy DNS traffic.

I'll have to look into that one, thanks. AFAICT, the only iptables rules we run are for icmp 0/3/8/11, ssh, dns, and what appears to be some kind of multicast traffic.

Thanks for the tip!

Matthew Walster

This message may be private and confidential. If you have received this message in error, please notify us and remove it from your system.

Gyron may monitor email traffic data and the content of email for the purposes of security and staff training.

Gyron Internet Ltd is a limited company registered in England and Wales. Registered number: 4239332. Registered office: 3 Centro, Boundary Way, Hemel Hempsted, HP2 7SU. VAT reg no 804 2532 63. Gyron is a registered trademark.
_______________________________________________
Pdns-users mailing list
Pdns-users@...
http://mailman.powerdns.com/mailman/listinfo/pdns-users

[Matthew Walster - Gyron]

> -----Original Message-----
> From: bert.hubert@... [mailto:bert.hubert@...] On Behalf Of
> bert hubert
> Sent: 03 July 2009 09:16
> To: Matthew Walster - Gyron
> Cc: Sten Spans; Pdns-users@...
> Subject: Re: [Pdns-users] UDP Connection Table Exhaustion?
>
> To nuance this a bit - on Linux, you can have great benefit from the
> iptables 'NOTRACK' target, which can help you do firewalling that will
> not run into problems from busy DNS traffic.

In case anyone reads the mail archives, runs CentOS, and is scared of iptables, here is the contents of /etc/sysconfig/iptables after I disabled connection tracking.

*** START ***

# Generated by iptables-save v1.3.5 on Fri Jul  3 15:16:17 2009
*raw
:PREROUTING ACCEPT [567:46949]
:OUTPUT ACCEPT [521:57257]
-A PREROUTING -p tcp -m tcp --sport 53 -j NOTRACK
-A PREROUTING -p tcp -m tcp --dport 53 -j NOTRACK
-A PREROUTING -p udp -m udp --dport 53 -j NOTRACK
-A PREROUTING -p udp -m udp --sport 53 -j NOTRACK
-A OUTPUT -p tcp -m tcp --sport 53 -j NOTRACK
-A OUTPUT -p tcp -m tcp --dport 53 -j NOTRACK
-A OUTPUT -p udp -m udp --dport 53 -j NOTRACK
-A OUTPUT -p udp -m udp --sport 53 -j NOTRACK
COMMIT
# Completed on Fri Jul  3 15:16:17 2009
# Generated by iptables-save v1.3.5 on Fri Jul  3 15:16:17 2009
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [607:67528]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p esp -j ACCEPT
-A RH-Firewall-1-INPUT -p ah -j ACCEPT
-A RH-Firewall-1-INPUT -d 224.0.0.251 -p udp -m udp --dport 5353 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp --sport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --sport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp --dport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 53 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Fri Jul  3 15:16:17 2009

*** END ***

Note that this is only for my VMWare test machine, so you'll want more/less rules depending on which services you are going to run (httpd etc) and you'll probably want to tighten a couple of the rules (such as anything with "--sport" otherwise you're letting in unnecessary traffic) but it's otherwise working.

Thanks to all who assisted with research for this!

Matthew Walster

This message may be private and confidential. If you have received this message in error, please notify us and remove it from your system.

Gyron may monitor email traffic data and the content of email for the purposes of security and staff training.

Gyron Internet Ltd is a limited company registered in England and Wales. Registered number: 4239332. Registered office: 3 Centro, Boundary Way, Hemel Hempsted, HP2 7SU. VAT reg no 804 2532 63. Gyron is a registered trademark.
_______________________________________________
Pdns-users mailing list
Pdns-users@...
http://mailman.powerdns.com/mailman/listinfo/pdns-users

[Stefan Schmidt-17]

On Thu, Jul 02, 2009 at 06:04:03PM +0200, Sten Spans wrote:
>> Hey there,
>>
>> Does anyone have any tips and/or tricks for running a medium-scale DNS  
>> recursive resolver appropriate to my situation? Medium being bigger
>> than "run it off a DSL router" but smaller than "get a server farm to
>> do it"!

0- do _not_ run it on a DSL router ;)
10- get a server farm to do it, no really if you get a chance to be
running a redundant recursive dns setup do it, today so many services
depend on this core service that the redundancy should be well worth it.
Also note that most stub resolvers (like libc) are taking their time
when failing over to the next nameserver entry in resolv.conf, some even
stall for 30s by default.

> 1- make sure you configure a limit for max-cache-entries
>    otherwise it will keep growing and consume all memory.
>    100k - 500k should cover most regular servers.

If you're already using ~1GB it's probably around 2.5 million entries
already. 1 million gives me 78% hitrate but this really depends on your
customerbase.

> 2- don't load any iptables modules if at all possible,
>    the state tracking causes serious (performance) problems
>    on loaded servers.

I do run connection tracking and am running 4 recursor processes [1] on
a Dual DC Opteron > 1000 qps each without problems with these settings:
net.ipv4.netfilter.ip_conntrack_udp_timeout=60
net.ipv4.netfilter.ip_conntrack_udp_timeout_stream=120
net.ipv4.netfilter.ip_conntrack_max=1000000
kernel.printk=4 4 1 7
net.core.netdev_max_backlog=3000
vm.min_free_kbytes=8118
net.core.rmem_max=8388608
net.core.wmem_max=8388608
net.core.rmem_default=2097152
net.core.wmem_default=2097152
net.ipv4.tcp_syncookies=0

The default for ip_conntrack_max is simply not made for nameservers, so
it needs to be tuned.

[1] This is a load balancing setup ofc. I'm using linux IPVS+keepalived.

> 3- make sure to explicitly configure incoming and outgoing ips,
>    preferably different ones. This enables later load-balancing / anycast
>    schemes and makes the traffic-flow predictable. If your network
>    layout is a bit more complicated (bgp, multiple gateways) then the
>    linus arp_announce and arp_filter sysctls should be tweaked to
>    make sure that Linux selects the correct source-ips.

yepp, totaly worth it.

> 4- use allow-from-file to configure the ranges that should be allowed
>    to use your nameservers.

That is _if_ you really have that many distinct IP-Ranges your clients
come from, i still use the 'regular' allow-from statement.

> This should allow you to handle 5-10k queries on reasonable hardware
> with a decent uplink (100mbit). Anything beyond that will require
> compilation with a recent compiler and system specific tuning,
> binding pdns to a specific cpu and the ethernet driver to another
> for example. This kind of tuning should only be done with careful
> measurements to test the effect of each change.

Also read the PERFORMANCE section of the README file from the recursors
tarball. c++ profiling for the recursor works just fine for me..

        Stefan
--
- All right, Gamma shift, time to defend the Federation against gaseous
  anomalies.
Commander Janice Rand, "Flashback.", ST-Voyager
_______________________________________________
Pdns-users mailing list
Pdns-users@...
http://mailman.powerdns.com/mailman/listinfo/pdns-users