Graphite

aggregationMethod for lower precision retentions - store all (average, sum, min, max, last) instead of just one?

Asked by Wilfried Schobeiri

I've been slightly flustered by the fact that an incremented count value will average over a time as i zoom out in the time series, instead of continuing to give me accurate data. I noticed that this (http://readthedocs.org/docs/graphite/en/latest/config-carbon.html#storage-aggregation-conf) is an option that can be configured per bucket, but I was wondering if there was a way to store all of those, instead of just having to pick one at a time.

It seems to me that there are definitely cases where having all five values at your disposal would be incredibly useful.

Question information

Language:
English Edit question
Status:
Answered
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
chrismd (chrismd) said : 		#1

I sympathize with your frustration, it is a common gotcha that we're working on fixing. The problem is Graphite can not "accurately" aggregate your datapoints unless you tell it what aggregation method to use for which metrics, and historically there has not been a good way to do that. There is a very active branch under development (https://code.launchpad.net/~chrismd/+junk/graphite-megacarbon) that addresses this issue among several others by refactoring the carbon configuration model. In the new model we generalize storage-schemas.conf to consist of rules that can apply arbitrary metadata to your metrics, so in addition to specifying the retention configuration you will also be able to specify the desired aggregation method using regexes, etc to specify which metrics use which method. This branch is still in flux but it is going to be in the next release (in fact the next release will primarily consist of the work already done on this branch). There is also a new API provided by the webapp in 0.9.9 that lets you get/set metadata for metrics programmatically but that is more intended for one-off changes rather than sweeping "handle all these metrics this way" type of changes that the storage rules will provide.

Revision history for this message
Wilfried Schobeiri (nphase) said : 		#2

So to clarify - this change still won't support multiple aggregation methods per metric - correct?

Revision history for this message
Nicholas Leskiw (nleskiw) said : 		#3

Graphite does not currently store more than one value per point, nor do I think it will.

Chris, if you're planning it, please re-open this and comment.

Revision history for this message
chrismd (chrismd) said : 		#4

Whisper can't support multiple aggregation methods but Ceres could, and the new storage-rules.conf would make configuring it possible. We'd have to extend Ceres' rollup aggregation plugin to support that.

Revision history for this message
chrismd (chrismd) said : 		#5

Regarding the carbon aggregation, you can configure as many aggregation rules as you'd like, so you could compute multiple aggregates from the same set of input metrics. Only sum and avg are implemented currently but adding more functions would be pretty trivial.

Revision history for this message
Brian Pitts (bpitts) said : 		#6

Chris, are you saying with Ceres it will be possible to do something like the below to keep multiple separate values for each metric during aggregation?

[collectd-min]
pattern = ^collectd\.
aggregationMethod = min

[collectd-average]
pattern = ^collectd\.
aggregationMethod = average

[collectd-max]
pattern = ^collectd\.
aggregationMethod = max

Can you help with this problem?

Provide an answer of your own, or ask Wilfried Schobeiri for more information if necessary.

To post a message you must log in.