Re: Possible DNS DOS?

(425)216-4691 Desk
(360)340-7431 Mobile
IM: serpent6877@...

I have an interesting problem that has been happening for about 2 weeks.  First a little about my setup, currently I am running the following:

Debian 5.0 (Lenny)

Pdns-server 2.9.22-1

Pdns-backend-mysql 2.9.21.2-1

Pdns-recursor 3.1.7-1

Hardware Platform is a Dell 1850 (dual processor), 8 GIG ram running a VMWARE virtualized environment.

We are hosting about 100 forwarding lookup domains and a lot of reverse delegation zones (we are an ISP with about 40,000 IP addresses we currently manage).

Our system is fairly busy but under normal traffic I very rarely see much load on the processor/network cards.

This server is the primary server and we have a few (mysql slaves) that replicate off of its database.  Under normal circumstances (4 or 5 days in a row) database queue averages 0 and spikes to 2 (so the database is keeping up just fine).

What I have been seeing recently show up in the logs is:

Jun 22 09:09:38 dns1 pdns[10948]: 5003 questions waiting for database attention. Limit is 5000, respawning

Jun 22 09:09:39 dns1 pdns[2538]: Our pdns instance exited with code 1

Jun 22 09:09:39 dns1 pdns[2538]: Respawning

Jun 22 09:09:39 dns1 kernel: [724751.668503] UDP: bad checksum. From 71.113.153.36:61250 to 208.187.180.2:53 ulen 46

Jun 22 09:09:40 dns1 pdns[10957]: Guardian is launching an instance

Jun 22 09:09:40 dns1 pdns[10957]: Reading random entropy from '/dev/urandom'

Jun 22 09:09:40 dns1 pdns[10957]: This is module gmysqlbackend.so reporting

Jun 22 09:09:40 dns1 pdns[10957]: This is a guarded instance of pdns

Jun 22 09:09:40 dns1 pdns[10957]: It is advised to bind to explicit addresses with the --local-address option

Jun 22 09:09:40 dns1 pdns[10957]: UDP server bound to 0.0.0.0:53

Jun 22 09:09:40 dns1 pdns[10957]: TCP server bound to 0.0.0.0:53

Jun 22 09:09:40 dns1 pdns[10957]: PowerDNS 2.9.22 (C) 2001-2009 PowerDNS.COM BV (Mar 22 2009, 16:58:52, gcc 4.3.2) starting up

Jun 22 09:09:40 dns1 pdns[10957]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.

Jun 22 09:09:40 dns1 pdns[10957]: DNS Proxy launched, local port 24312, remote 127.0.0.1:5300

Jun 22 09:09:40 dns1 pdns[10957]: Master/slave communicator launching

Jun 22 09:09:40 dns1 pdns[10957]: Creating backend connection for TCP

Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection succesful

Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection succesful

Jun 22 09:09:40 dns1 pdns[10957]: About to create 3 backend threads for UDP

Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection succesful

Jun 22 09:09:40 dns1 pdns[10957]: All slave domains are fresh

Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection succesful

Jun 22 09:09:40 dns1 pdns[10957]: gmysql Connection succesful

Jun 22 09:09:40 dns1 pdns[10957]: Done launching threads, ready to distribute questions

I will see this 11 to 12 times in less than 1 minute, network traffic and eth0 interrupts spike quickly during this time (feeling a little like a DNS denial of service).  After this happens about 11 times I see the following in the logs:

Jun 22 09:09:41 dns1 pdns[10957]: 5029 questions waiting for database attention. Limit is 5000, respawning

Jun 22 09:09:41 dns1 pdns[10957]: Got a signal 11, attempting to print trace:

Jun 22 09:09:41 dns1 pdns[10957]: /usr/sbin/pdns_server-instance [0x80ba397]

Jun 22 09:09:41 dns1 pdns[10957]: [0xb7f83400]

Jun 22 09:09:41 dns1 pdns[10957]: /usr/sbin/pdns_server-instance(_ZN5boost11multi_index6detail13ordered_indexINS0_13composite_keyIN11PacketCache10CacheEntryENS0_6memberIS5_SsXadL_ZNS5_5qnameEEEEENS6_IS5_tXadL_

ZNS5_5qtypeEEEEENS6_IS5_tXadL_ZNS5_5ctypeEEEEENS6_IS5_iXadL_ZNS5_6zoneIDEEEEENS6_IS5_bXadL_ZNS5_15meritsRecursionEEEEENS_6tuples9null_typeESD_SD_SD_SD_EENS0_21composite_key_compareI24CIBackwardsStringCompareSt

4lessItESI_SH_IiESH_IbESD_SD_SD_SD_SD_EENS1_9nth_layerILi1ES5_NS0_10indexed_byINS0_14ordered_uniqueISE_SL_N4mpl_2naEEENS0_9sequencedINS0_3tagISQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_EEEESQ_

SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_EESaIS5_EEENS_3mpl7vector0ISQ_EENS1_18ordered_unique_tagEE10link_pointERKNS0_20composite_key_resultISE_EERNS13_9link_infoES12_+0x286) [0x809f606]

Jun 22 09:09:41 dns1 pdns[10957]: /usr/sbin/pdns_server-instance(_ZN11PacketCache6insertERKSsRK5QTypeNS_14CacheEntryTypeES1_jib+0x103) [0x809a3c3]

Jun 22 09:09:41 dns1 pdns[10957]: /usr/sbin/pdns_server-instance(_ZN12UeberBackend11addNegCacheERKNS_8QuestionE+0x8e) [0x80c32de]

Jun 22 09:09:41 dns1 pdns[10957]: /usr/sbin/pdns_server-instance(_ZN12UeberBackend3getER17DNSResourceRecord+0x12f) [0x80c351f]

After this entry PDNS is down and stays down.

So a couple of questions for the group, I already have a wire shark up doing a long term capture (so I can see what is being sent at the server).  However is there a way PDNS can email/notify when it dies and does not come back?  Also what type of information/logging should I be enabling the system to further diagnose or troubleshoot the issue?

Any help/feedback is greatly appreciated.

Thanks

--Chris