Read datapoint from graphite programmatically

Asked by Earthlink

Hi,

I am trying (also, similarly to older questions) to compute and graph network interface rates from raw counter data (cumulative bytes/pkts) I am getting to a perl script from sflow probes.

Please don't respond with the derivative workaround or URL/webapp functions.

The solution looks like this:

Network Switch (sflow) -> sflow2graphite.pl (TCP socket) -> carbon-cache (port 2003)

Essentially I am using the plaintext protocol to write the counter to graphite via the socket (carbon listening on 2003).

If I could read the last <datapoint,timepoint> (cumulative bytes, timestamp) fed to graphite in this script, I could easily calculate bitrate (bytes seen-last bytes seen/dt) and store a rate metric to graphite.

Is there a way one can use the socket interface to read the last <datapoint,timestamp> from graphite for a selected metric?

Question information

Language:
English Edit question
Status:
Answered
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
rowan (rowanu) said :
#1

Not using the socket interface, no.

In some set ups there are many listeners/relays, all writing metrics to disk - querying them (ie. the right one) for a specific metric's last metric would be non-trivial, and not what Graphite is trying to do. Querying metrics from Graphite is done through the webapp interface.

Revision history for this message
Earthlink (kostas-katrinis) said :
#2

Thanks for the response and the hint.

Sharing here the outcome of my experimentation in the hope of saving "rate" fans from going through the same pain (or maybe improving my development).

So I created the following snippet (my frontend script for collecting sFLOW samples is in Perl):

#!/usr/bin/perl -w
use strict;
use WWW::Mechanize;
use JSON -support_by_pp;

my $GRAPHITE_SERVER="localhost";
my $metric=<your metric>;
my $time_window="2min";

my @arr = fetch_json_page("http://$GRAPHITE_SERVER/render?target=$metric&from=-$time_window&format=json");
print "\nTime=$arr[0], Sample=$arr[1]\n";

sub fetch_json_page
{
  my @last_dp=('-1','-1');
  my ($json_url) = @_;
  my $browser = WWW::Mechanize->new();
  eval{
    $browser->get( $json_url );
    my $content = $browser->content();
    my $json = new JSON;

    my $json_text = $json->allow_nonref->utf8->relaxed->escape_slash->loose->allow_singlequote->allow_barekey->decode($content);
    my $rootarrayref= $json_text;
    my $objhashref = $ { $json_text }[0];
    my $datapoint_arrayref = $objhashref->{datapoints};
    my @datapoints = reverse(@{$datapoint_arrayref});
    foreach (@datapoints){
      my @datapoint = @{$_};
      if (defined($datapoint[0])){
        @last_dp=($datapoint[1],$datapoint[0]);
        last;
      }
    }
  };
  if($@){
    print "[[JSON ERROR]] JSON parser crashed! $@\n";
  }
  return @last_dp;
}

What the above essentially does is go to the graphite store (whisper?) and retrieve the latest valid (non-empty) sample seen in the last 2 minutes (or [-1,-1] if no valid sample exists in this time window).

I went on and plugged this into my frontend script (sflow2graphite). Although it worked all the way, the raete results are inaccurate/fluctuating (compared the result to the rate of a synthetically generated network flow of steady rate). I suspect that this is due to the latest sample being aggregated (although I don't use any aggregation) or otherwise being stored in a delayed/gradual fashion to the graphite store.

Bottomline: for a quick solution: it makes more sense to work on your frontend script and just store the last seen <value,timestamp> of the desired metric(s) in an appropriate datastructure (e.g. hashmap) and use it to compute current rates.

Can you help with this problem?

Provide an answer of your own, or ask Earthlink for more information if necessary.

To post a message you must log in.