From wessels at measurement-factory.com Tue Mar 4 13:53:11 2014 From: wessels at measurement-factory.com (Duane Wessels) Date: Tue, 4 Mar 2014 06:53:11 -0700 (MST) Subject: [dsc] CPU problem In-Reply-To: References: Message-ID: Hello Elias, I have two suggestions: 1) you can run dsc in a debugging mode. For example: $ sudo dsc -d -f /usr/local/etc/dsc/dsc.conf that might help identify the problem. 2) second option is to send the running process an ABRT signal while it is running: $ sudo kill -ABRT dsc-pid That signal should cause the dsc process to make a core dump, although sometimes you have take some extra steps to make sure the core file gets created. Check "ulimit -c" and make sure dsc's working directory is writable. When you have a core dump, and if the binary is compiled with debugging symbols, then you can use gdb to pinpoint where its getting stuck. DW On Tue, 25 Feb 2014, Elias Abacioglu wrote: > Hi all, > > I've joined this mailing list cause I have what I would assume to be an > unusual problem. > I have 5 name servers running bind 9 and with similar setup. > > DSC works on all of them but causes problem on the fifth. > It uses 100% CPU on one core. > I tried updating the entire system (apt-get update && apt-get upgrade). It > did not help. > DSC is installed using a configuration manager so installation is identical. > > On all servers it spawns two processes, these two > /usr/bin/dsc -p /var/run/dsc-statistics-collector/dsc-collector_default.cfg > \_ [dsc] > > But on that faulty server after a short while it starts eating the CPU. And > it doesn't seem to be so more DNS usage on that nameserver. > The version is 201106061022-4, which is the one found in Ubuntu 12.04 > (precise) repos. > $ cat /etc/issue > Ubuntu 12.04.3 LTS \n \l > > How should I continue with this? Can't have one of two cores running at > 100% all the time. > > Regards, > Elias > _______________________________________________ > dsc mailing list > dsc at measurement-factory.com > http://www.measurement-factory.com/mailman/listinfo/dsc > From jeremy.cohoe at ubc.ca Tue Mar 4 20:09:41 2014 From: jeremy.cohoe at ubc.ca (Cohoe, Jeremy) Date: Tue, 4 Mar 2014 20:09:41 +0000 Subject: [dsc] Busiest Client Subnets chart Message-ID: Hello, I've been using DSC for a few months now and it's really great. It's provided me with insight into our DNS servers that I've never had before... so thanks :-) On the Client Geography page, the top entry is almost always the subnet of the DNS server itself, which skews the chart significantly. Is there something that I can do to fix this? The collector's config has local_address set to the correct IP, but from reading the manual it looks like this setting is only used for detecting the direction of the traffic. Screenshot attached for reference. Thanks for any insight or advise, Jeremy Cohoe, Network Analyst, Network Management Centre Information Technology | Engage. Envision. Enable. The University of British Columbia Tel: 604.827.5707 From elias.rabi at gmail.com Wed Mar 5 08:55:29 2014 From: elias.rabi at gmail.com (Elias Abacioglu) Date: Wed, 5 Mar 2014 09:55:29 +0100 Subject: [dsc] CPU problem In-Reply-To: References: Message-ID: Hi Duane, I've attached the output from the debug run. I doubt it will show anything since it doesn't spike the CPU instantly. It does it after it's been running for some time. I just restarted DSC(and running in foreground so I get the output). When it goes to 100% CPU I will do a kill -ABRT. ulimit -c was 0 so I guess unlimited size of core file. Thanks! Elias 2014-03-04 14:53 GMT+01:00 Duane Wessels : > > Hello Elias, > > I have two suggestions: > > 1) you can run dsc in a debugging mode. For example: > > $ sudo dsc -d -f /usr/local/etc/dsc/dsc.conf > > that might help identify the problem. > > 2) second option is to send the running process an ABRT signal > while it is running: > $ sudo kill -ABRT dsc-pid > > That signal should cause the dsc process to make a core dump, although > sometimes you have take some extra steps to make sure the core file > gets created. Check "ulimit -c" and make sure dsc's working directory > is writable. > > When you have a core dump, and if the binary is compiled with debugging > symbols, then you can use gdb to pinpoint where its getting stuck. > > DW > > > > On Tue, 25 Feb 2014, Elias Abacioglu wrote: > > Hi all, >> >> I've joined this mailing list cause I have what I would assume to be an >> unusual problem. >> I have 5 name servers running bind 9 and with similar setup. >> >> DSC works on all of them but causes problem on the fifth. >> It uses 100% CPU on one core. >> I tried updating the entire system (apt-get update && apt-get upgrade). It >> did not help. >> DSC is installed using a configuration manager so installation is >> identical. >> >> On all servers it spawns two processes, these two >> /usr/bin/dsc -p /var/run/dsc-statistics-collector/dsc-collector_ >> default.cfg >> \_ [dsc] >> >> But on that faulty server after a short while it starts eating the CPU. >> And >> it doesn't seem to be so more DNS usage on that nameserver. >> The version is 201106061022-4, which is the one found in Ubuntu 12.04 >> (precise) repos. >> $ cat /etc/issue >> Ubuntu 12.04.3 LTS \n \l >> >> How should I continue with this? Can't have one of two cores running at >> 100% all the time. >> >> Regards, >> Elias >> _______________________________________________ >> dsc mailing list >> dsc at measurement-factory.com >> http://www.measurement-factory.com/mailman/listinfo/dsc >> >> -------------- next part -------------- adding local address 127.0.0.1 adding local address ::1 adding local address 10.3.128.23 adding local address fe80::709c:d6ff:fee0:e87c setting current directory to /var/lib/dsc-statistics/default minfree_bytes 5000000 PID file is: /var/run/dsc-statistics-collector/default/dsc.pid BPF program is: udp port 53 Opening interface lo Pcap_init: FD_SET 4 Opening interface eth0 Pcap_init: FD_SET 5 creating dataset qtype creating dataset rcode creating dataset opcode creating dataset rcode_vs_replylen creating dataset client_subnet creating dataset qtype_vs_qnamelen creating dataset qtype_vs_tld creating dataset certain_qnames_vs_qtype creating dataset client_subnet2 creating dataset client_addr_vs_rcode creating dataset chaos_types_and_names creating dataset idn_qname creating dataset edns_version creating dataset edns_bufsiz creating dataset do_bit creating dataset rd_bit creating dataset idn_vs_tld creating dataset ipv6_rsn_abusers creating dataset transport_vs_qtype creating dataset client_port_range creating dataset second_ld_vs_rcode creating dataset third_ld_vs_rcode creating dataset direction_vs_ipproto creating dataset dns_ip_version_vs_qtype writing PID to /var/run/dsc-statistics-collector/default/dsc.pid Running grew d2[1] of rcode_vs_replylen from 128 to 256 grew d2[0] of edns_bufsiz from 2 to 16 grew d2[1] of client_subnet2 from 2 to 4 grew d2[0] of client_subnet from 2 to 4 grew d2[0] of qtype_vs_qnamelen from 32 to 64 grew d2[0] of client_addr_vs_rcode from 2 to 4 grew d2[1] of third_ld_vs_rcode from 2 to 4 grew d2[1] of client_addr_vs_rcode from 2 to 4 grew d2[1] of rcode_vs_replylen from 256 to 1024 grew d2[1] of qtype_vs_tld from 2 to 4 grew d2[0] of client_subnet from 4 to 8 grew d2[0] of third_ld_vs_rcode from 2 to 4 grew d2[0] of second_ld_vs_rcode from 2 to 4 grew d2[0] of client_addr_vs_rcode from 4 to 8 grew d2[1] of client_subnet2 from 4 to 8 grew d2[1] of third_ld_vs_rcode from 4 to 8 grew d2[1] of client_addr_vs_rcode from 4 to 8 grew d2[0] of qtype_vs_tld from 2 to 4 grew d2[0] of client_addr_vs_rcode from 8 to 16 grew d2[4] of dns_ip_version_vs_qtype from 2 to 4 grew d2[0] of transport_vs_qtype from 2 to 4 grew d2[2] of certain_qnames_vs_qtype from 2 to 4 grew d1 of qtype_vs_tld from 2 to 4 grew d1 of qtype_vs_qnamelen from 2 to 4 grew d2[0] of qtype from 2 to 4 grew d2[0] of third_ld_vs_rcode from 4 to 16 grew d2[0] of rcode_vs_replylen from 256 to 512 grew d2[1] of qtype_vs_tld from 4 to 8 grew d2[0] of second_ld_vs_rcode from 4 to 8 grew d2[1] of client_addr_vs_rcode from 8 to 16 grew d2[1] of client_subnet2 from 8 to 16 grew d2[0] of client_subnet from 8 to 16 grew d2[0] of client_addr_vs_rcode from 16 to 32 grew d2[1] of client_addr_vs_rcode from 16 to 32 grew d2[1] of third_ld_vs_rcode from 8 to 16 grew d2[1] of second_ld_vs_rcode from 2 to 8 grew d2[0] of qtype_vs_tld from 4 to 8 grew d2[0] of client_addr_vs_rcode from 32 to 64 grew d2[1] of client_addr_vs_rcode from 32 to 64 grew d2[0] of client_subnet2 from 8 to 16 grew d2[0] of third_ld_vs_rcode from 16 to 32 grew d2[1] of third_ld_vs_rcode from 16 to 32 grew d2[1] of second_ld_vs_rcode from 8 to 16 writing to 1394009040.dscdata.xml.XXXhipm2Y renaming to 1394009040.dscdata.xml From elias.rabi at gmail.com Wed Mar 5 20:02:41 2014 From: elias.rabi at gmail.com (Elias Abacioglu) Date: Wed, 5 Mar 2014 21:02:41 +0100 Subject: [dsc] CPU problem In-Reply-To: References: Message-ID: no core file :( I guess the ubuntu package isn't built with debugging symbols 2014-03-05 9:55 GMT+01:00 Elias Abacioglu : > Hi Duane, > I've attached the output from the debug run. I doubt it will show anything > since it doesn't spike the CPU instantly. It does it after it's been > running for some time. > I just restarted DSC(and running in foreground so I get the output). When > it goes to 100% CPU I will do a kill -ABRT. > ulimit -c was 0 so I guess unlimited size of core file. > > > Thanks! > Elias > > > 2014-03-04 14:53 GMT+01:00 Duane Wessels > : > > >> Hello Elias, >> >> I have two suggestions: >> >> 1) you can run dsc in a debugging mode. For example: >> >> $ sudo dsc -d -f /usr/local/etc/dsc/dsc.conf >> >> that might help identify the problem. >> >> 2) second option is to send the running process an ABRT signal >> while it is running: >> $ sudo kill -ABRT dsc-pid >> >> That signal should cause the dsc process to make a core dump, although >> sometimes you have take some extra steps to make sure the core file >> gets created. Check "ulimit -c" and make sure dsc's working directory >> is writable. >> >> When you have a core dump, and if the binary is compiled with debugging >> symbols, then you can use gdb to pinpoint where its getting stuck. >> >> DW >> >> >> >> On Tue, 25 Feb 2014, Elias Abacioglu wrote: >> >> Hi all, >>> >>> I've joined this mailing list cause I have what I would assume to be an >>> unusual problem. >>> I have 5 name servers running bind 9 and with similar setup. >>> >>> DSC works on all of them but causes problem on the fifth. >>> It uses 100% CPU on one core. >>> I tried updating the entire system (apt-get update && apt-get upgrade). >>> It >>> did not help. >>> DSC is installed using a configuration manager so installation is >>> identical. >>> >>> On all servers it spawns two processes, these two >>> /usr/bin/dsc -p /var/run/dsc-statistics-collector/dsc-collector_ >>> default.cfg >>> \_ [dsc] >>> >>> But on that faulty server after a short while it starts eating the CPU. >>> And >>> it doesn't seem to be so more DNS usage on that nameserver. >>> The version is 201106061022-4, which is the one found in Ubuntu 12.04 >>> (precise) repos. >>> $ cat /etc/issue >>> Ubuntu 12.04.3 LTS \n \l >>> >>> How should I continue with this? Can't have one of two cores running at >>> 100% all the time. >>> >>> Regards, >>> Elias >>> _______________________________________________ >>> dsc mailing list >>> dsc at measurement-factory.com >>> http://www.measurement-factory.com/mailman/listinfo/dsc >>> >>> > From wessels at measurement-factory.com Thu Mar 6 01:35:54 2014 From: wessels at measurement-factory.com (Duane Wessels) Date: Wed, 5 Mar 2014 18:35:54 -0700 (MST) Subject: [dsc] CPU problem In-Reply-To: References: Message-ID: You probably need to increase the corefile size limit. Try 'ulimit -c unlimited' before you run dsc. DW On Wed, 5 Mar 2014, Elias Abacioglu wrote: > no core file :( I guess the ubuntu package isn't built with debugging symbols > > > 2014-03-05 9:55 GMT+01:00 Elias Abacioglu : > Hi Duane, > I've attached the output from the debug run. I doubt it will show anything since it doesn't spike the CPU instantly. It > does it after it's been running for some time. > I just restarted DSC(and running in foreground so I get the output). When it goes to 100% CPU I will do a kill -ABRT. > ulimit -c was 0 so I guess unlimited size of core file. > > > Thanks! > Elias > > > 2014-03-04 14:53 GMT+01:00 Duane Wessels : > > Hello Elias, > > I have two suggestions: > > 1) you can run dsc in a debugging mode. For example: > > $ sudo dsc -d -f /usr/local/etc/dsc/dsc.conf > > that might help identify the problem. > > 2) second option is to send the running process an ABRT signal > while it is running: > $ sudo kill -ABRT dsc-pid > > That signal should cause the dsc process to make a core dump, although > sometimes you have take some extra steps to make sure the core file > gets created. Check "ulimit -c" and make sure dsc's working directory > is writable. > > When you have a core dump, and if the binary is compiled with debugging > symbols, then you can use gdb to pinpoint where its getting stuck. > > DW > > > On Tue, 25 Feb 2014, Elias Abacioglu wrote: > > Hi all, > > I've joined this mailing list cause I have what I would assume to be an > unusual problem. > I have 5 name servers running bind 9 and with similar setup. > > DSC works on all of them but causes problem on the fifth. > It uses 100% CPU on one core. > I tried updating the entire system (apt-get update && apt-get upgrade). It > did not help. > DSC is installed using a configuration manager so installation is identical. > > On all servers it spawns two processes, these two > /usr/bin/dsc -p /var/run/dsc-statistics-collector/dsc-collector_default.cfg > \_ [dsc] > > But on that faulty server after a short while it starts eating the CPU. And > it doesn't seem to be so more DNS usage on that nameserver. > The version is 201106061022-4, which is the one found in Ubuntu 12.04 > (precise) repos. > $ cat /etc/issue > Ubuntu 12.04.3 LTS \n \l > > How should I continue with this? Can't have one of two cores running at > 100% all the time. > > Regards, > Elias > _______________________________________________ > dsc mailing list > dsc at measurement-factory.com > http://www.measurement-factory.com/mailman/listinfo/dsc > > > > > From elias.rabi at gmail.com Thu Mar 6 15:15:21 2014 From: elias.rabi at gmail.com (Elias Abacioglu) Date: Thu, 6 Mar 2014 16:15:21 +0100 Subject: [dsc] CPU problem In-Reply-To: References: Message-ID: Ok, done. A 4.6 MB core file was created. But I really don't know how to use gdb (I tried the bt command). I executed gdb like this: # gdb /usr/bin/dsc ./core GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /usr/bin/dsc...(no debugging symbols found)...done. [New LWP 27928] warning: Can't read pathname for load map: Input/output error. Core was generated by `/usr/bin/dsc -p /var/run/dsc-statistics-collector/dsc-collector_default.cfg'. Program terminated with signal 6, Aborted. #0 0x00007ff2da182a08 in poll () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) bt #0 0x00007ff2da182a08 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ff2da9785f8 in ?? () from /usr/lib/x86_64-linux-gnu/libpcap.so.0.8 #2 0x0000000000406544 in ?? () #3 0x0000000000403fba in ?? () #4 0x00007ff2da0bb76d in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000404469 in ?? () How shall I proceed? /Elias 2014-03-06 2:35 GMT+01:00 Duane Wessels : > You probably need to increase the corefile size limit. Try 'ulimit -c > unlimited' before you run dsc. > > DW > > > > On Wed, 5 Mar 2014, Elias Abacioglu wrote: > > no core file :( I guess the ubuntu package isn't built with debugging >> symbols >> >> >> 2014-03-05 9:55 GMT+01:00 Elias Abacioglu : >> Hi Duane, >> I've attached the output from the debug run. I doubt it will show >> anything since it doesn't spike the CPU instantly. It >> does it after it's been running for some time. >> I just restarted DSC(and running in foreground so I get the output). When >> it goes to 100% CPU I will do a kill -ABRT. >> ulimit -c was 0 so I guess unlimited size of core file. >> >> >> Thanks! >> Elias >> >> >> 2014-03-04 14:53 GMT+01:00 Duane Wessels > >: >> >> Hello Elias, >> >> I have two suggestions: >> >> 1) you can run dsc in a debugging mode. For example: >> >> $ sudo dsc -d -f /usr/local/etc/dsc/dsc.conf >> >> that might help identify the problem. >> >> 2) second option is to send the running process an ABRT signal >> while it is running: >> $ sudo kill -ABRT dsc-pid >> >> That signal should cause the dsc process to make a core dump, >> although >> sometimes you have take some extra steps to make sure the core >> file >> gets created. Check "ulimit -c" and make sure dsc's working >> directory >> is writable. >> >> When you have a core dump, and if the binary is compiled with >> debugging >> symbols, then you can use gdb to pinpoint where its getting stuck. >> >> DW >> >> >> On Tue, 25 Feb 2014, Elias Abacioglu wrote: >> >> Hi all, >> >> I've joined this mailing list cause I have what I would assume to >> be an >> unusual problem. >> I have 5 name servers running bind 9 and with similar setup. >> >> DSC works on all of them but causes problem on the fifth. >> It uses 100% CPU on one core. >> I tried updating the entire system (apt-get update && apt-get >> upgrade). It >> did not help. >> DSC is installed using a configuration manager so installation is >> identical. >> >> On all servers it spawns two processes, these two >> /usr/bin/dsc -p /var/run/dsc-statistics-collector/dsc-collector_ >> default.cfg >> \_ [dsc] >> >> But on that faulty server after a short while it starts eating the >> CPU. And >> it doesn't seem to be so more DNS usage on that nameserver. >> The version is 201106061022-4, which is the one found in Ubuntu >> 12.04 >> (precise) repos. >> $ cat /etc/issue >> Ubuntu 12.04.3 LTS \n \l >> >> How should I continue with this? Can't have one of two cores >> running at >> 100% all the time. >> >> Regards, >> Elias >> _______________________________________________ >> dsc mailing list >> dsc at measurement-factory.com >> http://www.measurement-factory.com/mailman/listinfo/dsc >> >> >> >> >> >>