Graphite

Ability to extend Graphite with alerts/hooks system

Asked by Yury Smolsky

Hello!

I would like to know about the possibility to extend Graphite with a hook/trigger mechanism to notify other clients. I have examined the sources of Graphite and have not found appropriate place for this, nor a way to extend Graphite without modifying existing sources (and losing ability to upgrade Graphite in future without custom patches).

Existing external solutions like Tattle do not meet my requirements. I need to be able to setup hook/alerts for hundred of thousands of parameters and ideally I would like to check triggers inside of Graphite without making calls to /render? method, and possibly with some optimization b/c pulls will be made directly from whisper DB missing some of infrastructure (apache/etc).

For example, one case of alerts I would like to implement is to notify other software via HTTP requests, if the number of datapoints per minute per parameter would rich some threshold. And it would be tracked for 50k-80k of parameters roughly.
Another approach I have in my mind is to implement all this without altering Graphite. But then again I would need to setup duplicate listener for datapoints (not sure which is the best way to implement it) and make calls to "/render" method each time I need to check the trigger...

The big question is follows. How do you feel about extending Graphite with such functionality? Do you think it's appropriate to solve the task by introducing hook/alerts mechanism into software? If you know about other means of solving the problem of this scale, please let me know.

Thank you.

Question information

Language:
English Edit question
Status:
Answered
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Brian Lalor (blalor) said : 		#1

On Oct 16, 2013, at 7:01 AM, Yury Smolsky <email address hidden> wrote:

> I need to be able to setup hook/alerts for hundred of thousands of parameters and ideally I would like to check triggers inside of Graphite without making calls to /render? method, and possibly with some optimization b/c pulls will be made directly from whisper DB missing some of infrastructure (apache/etc).

One of Graphite's strengths comes from its ability to apply functions to multiple metrics. That's only exposed via the render api. Graphite is a higher level of abstraction on top of the raw whisper data files.

If you're happy looking at simple thresholds for a single metric, you can use the solution my team came up with: we wrote a proxy for carbon-cache that generates alerts based on the value of a metric and sends them to Sensu. It's dead simple, but effective for alerting in real time when the load average for a system has gone too high, and doesn't require Graphite to query the data.

Revision history for this message
Yury Smolsky (yury-v) said : 		#2

Thanks for the response, Brian!

Can you please provide more details how you have implemented the proxy for carbon-cache? Any chances to see sources snippets for that? :)

Revision history for this message
Brian Lalor (blalor) said : 		#3

On Oct 16, 2013, at 7:26 AM, Yury Smolsky <email address hidden> wrote:

> Can you please provide more details how you have implemented the proxy
> for carbon-cache? Any chances to see sources snippets for that? :)

Rather than having clients sending directly to carbon-cache on port 2003, I moved carbon-cache to 2002, started my proxy on 2003, and have the proxy send data to carbon-cache on 2002.

I wrote this module for Node.js: https://npmjs.org/package/carbon-streams

The sample code looks something like this:

var carbon = require("carbon");

var carbonListener = new carbon.TCP(2003);
var carbonWriter = new carbon.Writer("127.0.0.1", 2002);

// pass the data straight through
carbonListener.pipe(carbonWriter);

// capture metrics so they can be sent elsewhere
carbonListener.on("data", function(metricFrame) {
    var metric = metricFrame[0];
    var value = metricFrame[1];
    var timestamp = metricFrame[2];

    // do something with the metric value
});

Can you help with this problem?

Provide an answer of your own, or ask Yury Smolsky for more information if necessary.

To post a message you must log in.