From wessels at measurement-factory.com Sat Sep 6 08:30:30 2008 From: wessels at measurement-factory.com (Duane Wessels) Date: Sat, 6 Sep 2008 02:30:30 -0600 (MDT) Subject: [dsc] DNS Queries Trending In-Reply-To: References: Message-ID: <20080906022226.Q52150@measurement-factory.com> On Thu, 28 Aug 2008, David A. Evans wrote: > I'm looking to do some trending with the data that DSC has > collected over the last year for our servers. I'm looking a doing a 1 > year query/sec graph with a trend line for the next 6 months or a year. > Before I started digging into the data structure I want to make sure > nobody has done this or I'm not over looking something that will do this > already built into the tool. > I tried adding a 1 year time scale into the presenter and see that > its limited to 1 month. Is that due to the number of data points that > something piece can handle or is it load on the presenter that caused the > limit to be coded into the application? Hi David, In theory you can add "1 year" to the presenter config file, but in practice it doesn't work because it takes too much time to read 356 (x number of nodes) files. The apache/CGI interface probably times out. In the past I have written some scripts to read and process long time spans of data, but its always been custom "one-off" type scripts. If you're comfortable reading someone else's Perl code, you can take a look at perllib/DSC/grapher.pm and the load_data subroutine in particular. With a few tweaks that would give you something that can read the files pretty easily (although still slowly). Duane W. From Hoan.Le at webex.com Mon Sep 15 21:31:39 2008 From: Hoan.Le at webex.com (Hoan Le) Date: Mon, 15 Sep 2008 14:31:39 -0700 Subject: [dsc] No Data To Display at This Time Message-ID: Hi, I'm not able to see any graphs on my presenter server. All I'm getting is "No data to display at this time." 1. I have the following dsc-grapher.conf file server corp corp-dns1 corp-dns2 trace_windows 1hour 4hour 1day 1week accum_windows 1day 2days 3days 1week embargo 0 #anonymize_ip #no_http_header #hide_nodes domain_list all_tlds ac ad ae af ag ai al am an ao aq ar as at au aw az ba bb bd be bf domain_list all_tlds bg bh bi bj bm bn bo br bs bt bv bw by bz ca cc cd cf cg ch ci ck domain_list all_tlds cl cm cn co cr cu cv cx cy cz de dj dk dm do dz ec ee eg eh er es domain_list all_tlds et fi fj fk fm fo fr ga gd ge gf gg gh gi gl gm gn gp gq gr gs gt domain_list all_tlds gu gw gy hk hm hn hr ht hu id ie il im in io iq ir is it je jm jo domain_list all_tlds jp ke kg kh ki km kn kp kr kw ky kz la lb lc li lk lr ls lt lu lv domain_list all_tlds ly ma mc md mg mh mk ml mm mn mo mp mq mr ms mt mu mv mw mx my mz domain_list all_tlds na nc ne nf ng ni nl no np nr nu nz om pa pe pf pg ph pk pl pm pn domain_list all_tlds pr ps pt pw py qa re ro ru rw sa sb sc sd se sg sh si sj sk sl sm domain_list all_tlds sn so sr st sv sy sz tc td tf tg th tj tk tm tn to tp tr tt tv tw domain_list all_tlds tz ua ug uk um us uy uz va vc ve vg vi vn vu wf ws ye yt yu za zm domain_list all_tlds zw aero biz com coop info museum name net org pro gov edu mil int domain_list all_tlds arpa asia paris eu . valid_domains f-root all_tlds 2. In my data/corp/corp-dns2 directory I have my *.xml data file stored. #ls -lF | more total 154364 -rw-rw-r-- 1 root root 23055 Sep 11 04:28 1221132480.dscdata.xml -rw-rw-r-- 1 root root 24026 Sep 11 04:29 1221132540.dscdata.xml -rw-rw-r-- 1 root root 24158 Sep 11 04:30 1221132600.dscdata.xml -rw-rw-r-- 1 root root 22881 Sep 11 04:31 1221132660.dscdata.xml -rw-rw-r-- 1 root root 22885 Sep 11 04:32 1221132720.dscdata.xml -rw-rw-r-- 1 root root 22766 Sep 11 04:33 1221132780.dscdata.xml 3. I ran the following cronjob * * * * * /usr/bin/nice -10 /usr/local/dsc/libexec/refile-and-grok.sh 4. There are no errors in the /var/log/httpd/error_log file. Can someone let me know what might be the problem? Thanks, Hoan From mave at nic.cl Mon Sep 15 21:43:15 2008 From: mave at nic.cl (Mauricio Vergara Ereche) Date: Mon, 15 Sep 2008 17:43:15 -0400 Subject: [dsc] No Data To Display at This Time In-Reply-To: References: Message-ID: <200809151743.16211.mave@nic.cl> On Monday 15 September 2008 17:31:39 Hoan Le wrote: > Hi, > > I'm not able to see any graphs on my presenter server. All I'm getting > is "No data to display at this time." [... DSC grapher config...] > 2. In my data/corp/corp-dns2 directory I have my *.xml data file > stored. > > #ls -lF | more > > total 154364 [...] > -rw-rw-r-- 1 root root 23055 Sep 11 04:28 1221132480.dscdata.xml > > 3. I ran the following cronjob > > * * * * * /usr/bin/nice -10 /usr/local/dsc/libexec/refile-and-grok.sh Did you try to change this script at the beginning with "set +x" to enable more debug? What does it says "refile-and-grok-node.sh.stderr" and "refile-and-grok-node.sh.stdout" in your corp-dns2 directory? Do you have an "error" directory in it? > 4. There are no errors in the /var/log/httpd/error_log file. > > Can someone let me know what might be the problem? Did you upgrade from a previous version of DSC or is this the first install? If you did upgrade, did you check this ? http://dns.measurement-factory.com/tools/dsc/200801301758-upgrade.html -- Mauricio Vergara Ereche User #188365 counter.li.org DNS Admin NIC Chile mave [@] nic [.] cl Miraflores 222 piso 14, Santiago CHILE +56 2 9407710 Codigo Postal: 832-0198 http://www.nic.cl From Hoan.Le at webex.com Tue Sep 16 22:15:44 2008 From: Hoan.Le at webex.com (Hoan Le) Date: Tue, 16 Sep 2008 15:15:44 -0700 Subject: [dsc] No Data To Display at This Time In-Reply-To: References: Message-ID: Hi Mauricio, 1. This is my first install. 2. I see the following errors in /var/log/httpd/error_log [Tue Sep 16 11:02:56 2008] [error] [client 192.168.201.35] -stdin -maxrows 20000 -csmap -mapfile /usr/local/dsc/cache/ce244e7abf1023de8b50d9675d227bb3.map.new -png -o /usr/local/dsc/cache/ce244e7abf1023de8b50d9675d227bb3.png.new at /usr/lib/perl5/site_perl/5.8.5/DSC/ploticus.pm line 422., referer: http://rancid.corp.webex.com/cgi-bin/dsc-grapher.pl?window=604800&node=c orp-dns1&plot=qtype&server=corp [Tue Sep 16 11:02:56 2008] [error] [client 192.168.201.35] Premature end of script headers: dsc-grapher.pl, referer: http://rancid.corp.webex.com/cgi-bin/dsc-gr apher.pl?window=604800&node=corp-dns1&plot=qtype&server=corp I have installed ploticus (located at /usr/local/bin/pl). 3. Permission on refile-and-grok.sh is -rwxr-xr-x 1 root root 498 Aug 24 22:53 refile-and-grok.sh* 4. I have a /usr/local/dsc/var/log directory but there are no files or entries in it. 5. "refile-and-grok-node.sh.stdout" contents [root at rancid data]# more refile-and-grok.sh.stdout Mon Sep 15 13:57:23 PDT 2008 corp/corp-dns1: corp/corp-dns2: production/ns01: production/ns02: production/ns03: production/ns05: production/ns06: production/ns07: production/ns08: Mon Sep 15 14:01:42 PDT 2008 I don't have a "refile-and-grok-node.sh.stderr" file. 6. I have a "dsc-xml-extractor.out" file with no errors in it. Thanks, Hoan -----Original Message----- From: dsc-bounces at measurement-factory.com [mailto:dsc-bounces at measurement-factory.com] On Behalf Of dsc-request at measurement-factory.com Sent: Tuesday, September 16, 2008 11:00 AM To: dsc at measurement-factory.com Subject: dsc Digest, Vol 15, Issue 2 Send dsc mailing list submissions to dsc at measurement-factory.com To subscribe or unsubscribe via the World Wide Web, visit http://www.measurement-factory.com/mailman/listinfo/dsc or, via email, send a message with subject or body 'help' to dsc-request at measurement-factory.com You can reach the person managing the list at dsc-owner at measurement-factory.com When replying, please edit your Subject line so it is more specific than "Re: Contents of dsc digest..." Today's Topics: 1. No Data To Display at This Time (Hoan Le) 2. Re: No Data To Display at This Time (Mauricio Vergara Ereche) ---------------------------------------------------------------------- Message: 1 Date: Mon, 15 Sep 2008 14:31:39 -0700 From: "Hoan Le" Subject: [dsc] No Data To Display at This Time To: Message-ID: Content-Type: text/plain; charset="us-ascii" Hi, I'm not able to see any graphs on my presenter server. All I'm getting is "No data to display at this time." 1. I have the following dsc-grapher.conf file server corp corp-dns1 corp-dns2 trace_windows 1hour 4hour 1day 1week accum_windows 1day 2days 3days 1week embargo 0 #anonymize_ip #no_http_header #hide_nodes domain_list all_tlds ac ad ae af ag ai al am an ao aq ar as at au aw az ba bb bd be bf domain_list all_tlds bg bh bi bj bm bn bo br bs bt bv bw by bz ca cc cd cf cg ch ci ck domain_list all_tlds cl cm cn co cr cu cv cx cy cz de dj dk dm do dz ec ee eg eh er es domain_list all_tlds et fi fj fk fm fo fr ga gd ge gf gg gh gi gl gm gn gp gq gr gs gt domain_list all_tlds gu gw gy hk hm hn hr ht hu id ie il im in io iq ir is it je jm jo domain_list all_tlds jp ke kg kh ki km kn kp kr kw ky kz la lb lc li lk lr ls lt lu lv domain_list all_tlds ly ma mc md mg mh mk ml mm mn mo mp mq mr ms mt mu mv mw mx my mz domain_list all_tlds na nc ne nf ng ni nl no np nr nu nz om pa pe pf pg ph pk pl pm pn domain_list all_tlds pr ps pt pw py qa re ro ru rw sa sb sc sd se sg sh si sj sk sl sm domain_list all_tlds sn so sr st sv sy sz tc td tf tg th tj tk tm tn to tp tr tt tv tw domain_list all_tlds tz ua ug uk um us uy uz va vc ve vg vi vn vu wf ws ye yt yu za zm domain_list all_tlds zw aero biz com coop info museum name net org pro gov edu mil int domain_list all_tlds arpa asia paris eu . valid_domains f-root all_tlds 2. In my data/corp/corp-dns2 directory I have my *.xml data file stored. #ls -lF | more total 154364 -rw-rw-r-- 1 root root 23055 Sep 11 04:28 1221132480.dscdata.xml -rw-rw-r-- 1 root root 24026 Sep 11 04:29 1221132540.dscdata.xml -rw-rw-r-- 1 root root 24158 Sep 11 04:30 1221132600.dscdata.xml -rw-rw-r-- 1 root root 22881 Sep 11 04:31 1221132660.dscdata.xml -rw-rw-r-- 1 root root 22885 Sep 11 04:32 1221132720.dscdata.xml -rw-rw-r-- 1 root root 22766 Sep 11 04:33 1221132780.dscdata.xml 3. I ran the following cronjob * * * * * /usr/bin/nice -10 /usr/local/dsc/libexec/refile-and-grok.sh 4. There are no errors in the /var/log/httpd/error_log file. Can someone let me know what might be the problem? Thanks, Hoan ------------------------------ Message: 2 Date: Mon, 15 Sep 2008 17:43:15 -0400 From: Mauricio Vergara Ereche Subject: Re: [dsc] No Data To Display at This Time To: dsc at measurement-factory.com Message-ID: <200809151743.16211.mave at nic.cl> Content-Type: text/plain; charset="iso-8859-1" On Monday 15 September 2008 17:31:39 Hoan Le wrote: > Hi, > > I'm not able to see any graphs on my presenter server. All I'm getting > is "No data to display at this time." [... DSC grapher config...] > 2. In my data/corp/corp-dns2 directory I have my *.xml data file > stored. > > #ls -lF | more > > total 154364 [...] > -rw-rw-r-- 1 root root 23055 Sep 11 04:28 1221132480.dscdata.xml > > 3. I ran the following cronjob > > * * * * * /usr/bin/nice -10 /usr/local/dsc/libexec/refile-and-grok.sh Did you try to change this script at the beginning with "set +x" to enable more debug? What does it says "refile-and-grok-node.sh.stderr" and "refile-and-grok-node.sh.stdout" in your corp-dns2 directory? Do you have an "error" directory in it? > 4. There are no errors in the /var/log/httpd/error_log file. > > Can someone let me know what might be the problem? Did you upgrade from a previous version of DSC or is this the first install? If you did upgrade, did you check this ? http://dns.measurement-factory.com/tools/dsc/200801301758-upgrade.html -- Mauricio Vergara Ereche User #188365 counter.li.org DNS Admin NIC Chile mave [@] nic [.] cl Miraflores 222 piso 14, Santiago CHILE +56 2 9407710 Codigo Postal: 832-0198 http://www.nic.cl ------------------------------ _______________________________________________ dsc mailing list dsc at measurement-factory.com http://www.measurement-factory.com/mailman/listinfo/dsc End of dsc Digest, Vol 15, Issue 2 ********************************** From mave at nic.cl Tue Sep 16 22:58:38 2008 From: mave at nic.cl (Mauricio Vergara Ereche) Date: Tue, 16 Sep 2008 18:58:38 -0400 Subject: [dsc] No Data To Display at This Time In-Reply-To: References: Message-ID: <200809161858.38873.mave@nic.cl> Hi Hoan! On Tuesday 16 September 2008 18:15:44 Hoan Le wrote: > Hi Mauricio, > > 1. This is my first install. OK > 2. I see the following errors in /var/log/httpd/error_log [...] I think that you didn't wait for the browser to complete the process. That would be OK in that case. > I have installed ploticus (located at /usr/local/bin/pl). ok > 3. Permission on refile-and-grok.sh is -rwxr-xr-x 1 root root 498 > Aug 24 22:53 refile-and-grok.sh* ok > 4. I have a /usr/local/dsc/var/log directory but there are no files or > entries in it. Are you transferring the files with rsync or x509? If it is the second, you must see some logs in there. If you're doing the rsync, check if the line 45 is commented and line 44 is working ok... Some linux distros may fail in this. When I did try to install that, the "--remove-source" failed and we used the other method. > 5. "refile-and-grok-node.sh.stdout" contents > > [root at rancid data]# more refile-and-grok.sh.stdout > Mon Sep 15 13:57:23 PDT 2008 > corp/corp-dns1: > corp/corp-dns2: [...] > Mon Sep 15 14:01:42 PDT 2008 What about the refile-and-grok.sh.stdout that is inside of data/corp/corp-dns1 ? Do you have the incoming dir in it? Do you have an error dir maybe? > I don't have a "refile-and-grok-node.sh.stderr" file. Same as above. It must be inside of data/corp/corp-dns1 > 6. I have a "dsc-xml-extractor.out" file with no errors in it. > My first impression is that you do have some data in it, but it's very old and you can't see it because the time span. The logs will help to verify that, but I may be wrong... It's just a gut feeling :-) Regards, Mauricio -- Mauricio Vergara Ereche User #188365 counter.li.org DNS Admin NIC Chile mave [@] nic [.] cl Miraflores 222 piso 14, Santiago CHILE +56 2 9407710 Codigo Postal: 832-0198 http://www.nic.cl From wessels at measurement-factory.com Thu Sep 18 23:11:57 2008 From: wessels at measurement-factory.com (Duane Wessels) Date: Thu, 18 Sep 2008 17:11:57 -0600 (MDT) Subject: [dsc] No Data To Display at This Time In-Reply-To: References: Message-ID: <20080918170551.V71618@measurement-factory.com> On Tue, 16 Sep 2008, Hoan Le wrote: > 2. I see the following errors in /var/log/httpd/error_log > > [Tue Sep 16 11:02:56 2008] [error] [client 192.168.201.35] -stdin > -maxrows > 20000 -csmap -mapfile > /usr/local/dsc/cache/ce244e7abf1023de8b50d9675d227bb3.map.new > -png -o /usr/local/dsc/cache/ce244e7abf1023de8b50d9675d227bb3.png.new > at /usr/lib/perl5/site_perl/5.8.5/DSC/ploticus.pm line 422., referer: > http://rancid.corp.webex.com/cgi-bin/dsc-grapher.pl?window=604800&node=c > orp-dns1&plot=qtype&server=corp > > [Tue Sep 16 11:02:56 2008] [error] [client 192.168.201.35] Premature end > of script headers: dsc-grapher.pl, referer: > http://rancid.corp.webex.com/cgi-bin/dsc-gr > apher.pl?window=604800&node=corp-dns1&plot=qtype&server=corp > > > I have installed ploticus (located at /usr/local/bin/pl). Based on your error logs it really looks like the CGI script cannot find the ploticus program. Since you say that it is installed then I would first make sure that it (and all its parent directories) have permissions that allow the CGI userid to execute it. For example: $ ls -ld /usr /usr/local /usr/local/bin /usr/local/bin/pl drwxr-xr-x 18 root wheel 512 Mar 26 08:39 /usr drwxr-xr-x 23 root wheel 512 Aug 4 14:31 /usr/local drwxr-xr-x 3 root wheel 22016 Sep 15 03:25 /usr/local/bin -r-xr-xr-x 1 root wheel 371804 Apr 24 20:32 /usr/local/bin/ploticus FYI the version of ploticus that installs as 'pl' might be too old for DSC now. You should have at least ploticus version 2.33 and 2.40 is also available. If you think that permissions are correct then perhaps the next step is to add some debugging in the DSC::ploticus module as follows Index: presenter/perllib/DSC/ploticus.pm =================================================================== --- presenter/perllib/DSC/ploticus.pm (revision 10393) +++ presenter/perllib/DSC/ploticus.pm (working copy) @@ -411,6 +411,7 @@ sub ploticus_begin { die "wrong state" unless (1 == $ploticus_state); + system "ls -ld /usr/local/bin/pl /usr/local/bin/ploticus 1>&2"; my $prog = (grep {-f $_} qw(/usr/local/bin/ploticus /usr/local/bin/pl))[0]; my $cmd = join(" ", $prog, Duane W.