From jromero at rcp.pe Wed Nov 4 16:23:21 2009 From: jromero at rcp.pe (Jose Romero) Date: Wed, 4 Nov 2009 11:23:21 -0500 Subject: [dsc] Problems in DSC latest version (dsc-200910121649.tar.gz) Message-ID: <003f01ca5d6b$203b4700$60b1d500$@pe> Hi List! I find DSC an interesting and valuable tool. I have installed DSC in our DNS servers but I have some problems with it. I will appreciate your help. I have installed the collector in the DNS servers, all is fine there, the XML files are generated every 60 seconds. In other machine I have installed the presenter, it runs FreeBSD 7.2, Apache 2.2, all required Perl modules and Ploticus 2.40_1. The refile-and-grok.sh script process the XML files and generate the .dat files. I can see the graphics for the following datasets: dataset qtype dns All:null Qtype:qtype queries-only; dataset rcode dns All:null Rcode:rcode replies-only; dataset opcode dns All:null Opcode:opcode queries-only; dataset rcode_vs_replylen dns Rcode:rcode ReplyLen:msglen replies-only; dataset qtype_vs_qnamelen dns Qtype:qtype QnameLen:qnamelen queries-only; dataset qtype_vs_tld dns Qtype:qtype TLD:tld queries-only,popular-qtypes max-cells=200; dataset idn_qname dns All:null IDNQname:idn_qname queries-only; dataset edns_version dns All:null EDNSVersion:edns_version queries-only; dataset do_bit dns All:null D0:do_bit queries-only; dataset rd_bit dns All:null RD:rd_bit queries-only; dataset idn_vs_tld dns All:null TLD:tld queries-only,idn-only; dataset transport_vs_qtype dns Transport:transport Qtype:qtype queries-only; dataset direction_vs_ipproto ip Direction:ip_direction IPProto:ip_proto any; But, I can't see the graphics for the following datasets: dataset client_subnet dns All:null ClientSubnet:client_subnet queries-only max-cells=200; dataset certain_qnames_vs_qtype dns CertainQnames:certain_qnames Qtype:qtype queries-only; dataset client_subnet2 dns Class:query_classification ClientSubnet:client_subnet queries-only max-cells=200; dataset client_addr_vs_rcode dns Rcode:rcode ClientAddr:client replies-only max-cells=50; dataset chaos_types_and_names dns Qtype:qtype Qname:qname chaos-class,queries-only; dataset ipv6_rsn_abusers dns All:null ClientAddr:client queries-only,aaaa-or-a6-only,root-servers-n et-only max-cells=50; I don't see nothing in the apache error log. In the browser only appear "No Data To Display At This Time" for these graphics. These files have data, they're not empty: ~/20091103/client_subnet2_accum.dat 200.31.106.0 ok 517 200.107.160.0 ok 1363 8.0.4.0 ok 27 . ~/20091103/client_subnet2_trace.dat 1257224340 ok 9128 non-auth-tld 81 1257224400 ok 9686 non-auth-tld 28 1257224460 ok 9236 non-auth-tld 27 . ~/20091103/client_subnet2_count.dat 1257224340 ok 3834 non-auth-tld 14 1257224400 ok 3979 non-auth-tld 6 1257224460 ok 3828 non-auth-tld 6 . ~/20091103/client_subnet_accum.dat 109.106.147.0 12 109.106.148.0 9 109.165.133.0 9 . ~/20091103/second_ld_vs_rcode_accum.dat in-addr.arpa 5 340 resufm.pe 3 2 directvla.pe 0 7 . ~/20091103/third_ld_vs_rcode_accum.dat global_force.com.pe 3 4 g.gob.pe 3 4 connect.facebook.com 5 2 . ~/20091103/client_addr_vs_rcode_accum.dat 87.249.7.252 0 8 203.113.131.6 3 3 85.31.212.254 0 66 . and so on ... I trried enable debug in the dsc-grapher.cfg file with debug_level 0 and debug_file /tmp/debug, but it don't work. The only one error in the apache error log is when I try see the "IP Version" - "Query Types" plots: [Tue Nov 03 11:39:06 2009] [error] [client 161.132.1.3] Use of uninitialized value in addition (+) at /usr/local/lib/perl5/site_perl/5.8.9/DSC/grapher.pm line 1145., referer: http://atlas.rcp.net.pe/cgi-bin/dsc-grapher.pl?binsize=60 &window=86400&node=ichu&plot=client_addr_vs_rcode_accum&server=SDT [Tue Nov 03 11:39:06 2009] [error] [client 161.132.1.3] Argument "IPv4" isn't numeric in sort at /usr/local/lib/perl5/site_perl/5.8.9/DSC/ploticus.pm line 76., referer: http://atlas.rcp.net.pe/cgi-bin/dsc-grapher.pl?binsize=60 &window=86400&node=ichu&plot=client_addr_vs_rcode_accum&server=SDT I have also tried to see the these graphics by URL: http://~/cgi-bin/dsc-grapher.pl?binsize=60 &window=86400&content=image&node=ichu&plot=client_subnet2_accum&server=SDT And the browser shows: "Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request." In the apache error log I see: "[Wed Nov 04 11:17:15 2009] [error] [client 161.132.1.3] Premature end of script headers: dsc-grapher.pl" Someone has any idea about this problem? Thanks in advance. Regards Jose Romero From wessels at measurement-factory.com Fri Nov 6 13:04:01 2009 From: wessels at measurement-factory.com (Duane Wessels) Date: Fri, 6 Nov 2009 06:04:01 -0700 (MST) Subject: [dsc] Problems in DSC latest version (dsc-200910121649.tar.gz) In-Reply-To: <003f01ca5d6b$203b4700$60b1d500$@pe> References: <003f01ca5d6b$203b4700$60b1d500$@pe> Message-ID: On Wed, 4 Nov 2009, Jose Romero wrote: > But, I can't see the graphics for the following datasets: > > dataset client_subnet dns All:null ClientSubnet:client_subnet queries-only > max-cells=200; > > dataset certain_qnames_vs_qtype dns CertainQnames:certain_qnames Qtype:qtype > queries-only; > > dataset client_subnet2 dns Class:query_classification > ClientSubnet:client_subnet queries-only max-cells=200; > > dataset client_addr_vs_rcode dns Rcode:rcode ClientAddr:client replies-only > max-cells=50; > > dataset chaos_types_and_names dns Qtype:qtype Qname:qname > chaos-class,queries-only; > > dataset ipv6_rsn_abusers dns All:null ClientAddr:client > queries-only,aaaa-or-a6-only,root-servers-n et-only max-cells=50; Jose, the attached patch should solve most of these. Some of the datasets might be empty because you don't see any traffic that matches the right conditions. Duane W. -------------- next part -------------- Index: presenter/perllib/DSC/grapher.pm =================================================================== --- presenter/perllib/DSC/grapher.pm (revision 12031) +++ presenter/perllib/DSC/grapher.pm (revision 12039) @@ -25,7 +25,7 @@ END { } # CONSTANTS -my $dbg_lvl = 0; +my $dbg_lvl = 0; # also set debug_file in dsc-grapher.cfg my $DATAROOT = '/usr/local/dsc/data'; my $DEFAULTCONFIG = '/usr/local/dsc/etc/dsc-grapher.cfg'; my $CacheImageTTL = 60; # 1 min Index: presenter/perllib/DSC/grapher/plot.pm =================================================================== --- presenter/perllib/DSC/grapher/plot.pm (revision 12031) +++ presenter/perllib/DSC/grapher/plot.pm (revision 12039) @@ -241,7 +241,6 @@ }, client_subnet_accum => { - dataset => 'client_subnet', plot_type => 'accum1d', keys => [ qw(?? IA LA AP RI AR) ], names => [ qw(Unknown IANA LACNIC APNIC RIPE ARIN) ], @@ -279,7 +278,6 @@ # }, client_subnet2_accum => { - dataset => 'client_subnet2', plot_type => 'accum2d', yaxes => $std_accum_yaxes, keys => $client_subnet2_keys, @@ -293,7 +291,6 @@ }, client_subnet2_trace => { - dataset => 'client_subnet2', plot_type => 'trace', yaxes => $std_trace_yaxes, keys => $client_subnet2_keys, @@ -306,7 +303,6 @@ }, client_subnet2_count => { - dataset => 'client_subnet2', plot_type => 'trace', keys => $client_subnet2_keys, names => $client_subnet2_names, @@ -727,7 +723,6 @@ }, client_addr_vs_rcode_accum => { - dataset => 'client_addr_vs_rcode', plot_type => 'accum2d', yaxes => $std_accum_yaxes, keys => [ qw(0 1 2 3 4 5) ], @@ -741,7 +736,6 @@ }, ipv6_rsn_abusers_accum => { - dataset => 'ipv6_rsn_abusers', plot_type => 'accum1d', keys => [ qw(Other BIND8 BIND9 W2000 WINNT W2003 DJBDNS NoAns) ], names => [ qw(Other BIND8 BIND9 W2000 WINNT W2003 DJBDNS NoAns) ], From brian at pongonova.net Sat Nov 14 04:09:55 2009 From: brian at pongonova.net (Brian Koontz) Date: Fri, 13 Nov 2009 22:09:55 -0600 Subject: [dsc] Running collector and presenter on same machine Message-ID: <20091114040955.GC5839@pongonova.net> Just installed the latest version of dsc. I'm somewhat confused as to how to set up both collector and presenter on same machine. I have set up run/ns7/upload/local, and see .xml files being generated and placed in ns7. I also see date-formatted dirs being created in the local/ dir. So at this point, I assume the collector side of things is working as it should be. However, I'm not sure what my destination dir is supposed to be. I have data/ns7/local set up, but running the following doesn't seem to do anything: /usr/local/dsc/libexec/upload-rsync.sh ns7 local /usr/local/dsc/data/ns7 I've read through the PDF multiple times, and simply can't figure out where the files need to go in order to be accessed via the CGI script. --Brian From wessels at measurement-factory.com Sat Nov 14 06:08:40 2009 From: wessels at measurement-factory.com (Duane Wessels) Date: Fri, 13 Nov 2009 23:08:40 -0700 (MST) Subject: [dsc] Running collector and presenter on same machine In-Reply-To: <20091114040955.GC5839@pongonova.net> References: <20091114040955.GC5839@pongonova.net> Message-ID: On Fri, 13 Nov 2009, Brian Koontz wrote: > Just installed the latest version of dsc. I'm somewhat confused as to > how to set up both collector and presenter on same machine. I have > set up run/ns7/upload/local, and see .xml files being generated and > placed in ns7. I also see date-formatted dirs being created in the > local/ dir. So at this point, I assume the collector side of things > is working as it should be. > > However, I'm not sure what my destination dir is supposed to be. I > have data/ns7/local set up, but running the following doesn't seem to > do anything: > > /usr/local/dsc/libexec/upload-rsync.sh ns7 local /usr/local/dsc/data/ns7 > > I've read through the PDF multiple times, and simply can't figure out > where the files need to go in order to be accessed via the CGI script. Hi Brian, Yes, this is not a well-documented arrangement. What I typically do is make the "upload" directory a symbolic link into the "data" directory. For example: $ cd /usr/local/dsc $ ls -l run/SERVER/upload/local lrwxr-xr-x 1 dsc dsc 35 Oct 22 2008 run/SERVER/upload/local -> /usr/local/dsc/data/SERVER/NODE/incoming Where SERVER and NODE are set to your local server and node names. Then when the collector "upload-prep.pl" script runs, it will put the .xml files directly into the presenter data area. Duane W. From brian at pongonova.net Sat Nov 14 20:38:37 2009 From: brian at pongonova.net (Brian Koontz) Date: Sat, 14 Nov 2009 14:38:37 -0600 Subject: [dsc] Running collector and presenter on same machine In-Reply-To: References: <20091114040955.GC5839@pongonova.net> Message-ID: <20091114203837.GA10251@pongonova.net> On Fri, Nov 13, 2009 at 11:08:40PM -0700, Duane Wessels wrote: > Yes, this is not a well-documented arrangement. What I typically do is > make the "upload" directory a symbolic link into the "data" directory. OK, that makes sense. But I think I'm still missing something, as DSC::grapher is still complaining about no data being found. Here's my setup: [root at xenotime upload]# cd /usr/local/dsc [root at xenotime dsc]# ls -l run/ns7/upload/local lrwxrwxrwx 1 root root 30 Nov 14 21:30 run/ns7/upload/local -> /usr/local/dsc/data/ns7/local/ I see, under local/, a directory named 2009-11-14, with a couple of xml files in it (*.dscdata.xml). Accessing the cgi-bin script results in the menu being displayed, with a prompt to "Please select a server". Clicking ns7 brings up local, and clicking that results in "No data to display at this time." Here's my crontab: * * * * * /usr/bin/nice -10 /usr/local/dsc/libexec/refile-and-grok.sh 22 4 * * * find /usr/local/dsc/data/ | /usr/local/dsc/libexec/remove-xmls.pl 7 * * * * * /usr/local/dsc/libexec/upload-prep.sh #* * * * * /usr/local/dsc/libexec/upload-rsync.sh ns7 local /usr/local/dsc/data/ns7 Appreciate the help. --Brian From brian at pongonova.net Sat Nov 14 23:08:32 2009 From: brian at pongonova.net (Brian Koontz) Date: Sat, 14 Nov 2009 17:08:32 -0600 Subject: [dsc] Running collector and presenter on same machine In-Reply-To: <20091114203837.GA10251@pongonova.net> References: <20091114040955.GC5839@pongonova.net> <20091114203837.GA10251@pongonova.net> Message-ID: <20091114230832.GB10251@pongonova.net> Duane-- OK, I think I see where the problem is...your example mentioned an "incoming" directory. The dsc-extractor expects this, refile-and-grok mentions it, but there's no mention of this in the docs. So what I ended up doing is creating a symlink: [root at xenotime local]# ls -l incoming lrwxrwxrwx 1 root apache 2 Nov 14 23:52 incoming -> ./ refile-and-grok does what it's supposed to now (dumps a bunch of .dat files in yyyymmdd). But still, no graphs ("No data to display at this time"). --Brian From brian at pongonova.net Sat Nov 14 23:29:22 2009 From: brian at pongonova.net (Brian Koontz) Date: Sat, 14 Nov 2009 17:29:22 -0600 Subject: [dsc] Running collector and presenter on same machine In-Reply-To: <20091114230832.GB10251@pongonova.net> References: <20091114040955.GC5839@pongonova.net> <20091114203837.GA10251@pongonova.net> <20091114230832.GB10251@pongonova.net> Message-ID: <20091114232922.GD10251@pongonova.net> On Sat, Nov 14, 2009 at 05:08:32PM -0600, Brian Koontz wrote: > refile-and-grok does what it's supposed to now (dumps a bunch of .dat > files in yyyymmdd). But still, no graphs ("No data to display at this > time"). This one was my stupidity at work...forgot to make the cache directory writable by the apache group... Graphs are now working, everything looks great! --Brian From brian at pongonova.net Sat Nov 14 23:41:50 2009 From: brian at pongonova.net (Brian Koontz) Date: Sat, 14 Nov 2009 17:41:50 -0600 Subject: [dsc] Legends in graphs? Message-ID: <20091114234150.GF10251@pongonova.net> How do I get the legends in the the graphs to display labels? All I'm getting are very small, skinny legend symbols, but no labels. (They are, however, properly linked...) --Brian From brian at pongonova.net Sun Nov 15 02:53:27 2009 From: brian at pongonova.net (Brian Koontz) Date: Sat, 14 Nov 2009 20:53:27 -0600 Subject: [dsc] Legends in graphs? In-Reply-To: <20091114234150.GF10251@pongonova.net> References: <20091114234150.GF10251@pongonova.net> Message-ID: <20091115025327.GA11492@pongonova.net> On Sat, Nov 14, 2009 at 05:41:50PM -0600, Brian Koontz wrote: > How do I get the legends in the the graphs to display labels? All I'm > getting are very small, skinny legend symbols, but no labels. (They > are, however, properly linked...) OK, the problem is with Ploticus v 2.41. Either (1) downgrade to 2.40[1], or (2) apply the 12May2009 fixes[2] to your 2.41 source code tree and recompile. Still getting long skinny legend boxes (a pixel or so high) as opposed to squares... --Brian [1] http://www.measurement-factory.com/pipermail/dsc/2009-April/000149.html [2] http://ploticus.sourceforge.net/doc/news.html