Graphite

Consolidation Functions for data rollup?

Asked by Steve Keller

Hi,

I see in the whisper code that there is only the "average" consolidation function that is employed when rolling up data from one more granular section to another less granular one. In the function propagate():

...
if knownPercent >= xff: #we have enough data to propagate a value!
    aggregateValue = float(sum(knownValues)) / float(len(knownValues)) #TODO another CF besides average?
...

This implies that you have at least considered other approaches, is this on the roadmap? I ask because for some data (bandwidth utilization springs to mind) it might be more useful to keep, say, the maximum value over a time period rather than the average.

The ideal situation would be to allow us to define multiple rollup functions per database, if desired - so we could keep min, max and average for those data that it seems useful for.

Thanks,
Steve Keller

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
chrismd
Solved:
Last query:
Last reply:
Revision history for this message
Best chrismd (chrismd) said : 		#1

Hi Steve, so this will probably be addressed (somewhat) in the next month or so. Due to some projects at work I've had to implement a new database backend to replace whisper. Overall its pretty similar but has some important differences that enable some pretty cool functionality. There was a little discussion about it on the mailing list a few weeks ago. I've got it working right now but the goal is to make it work transparently along side existing whisper data from previous versions before making a release. This new backend will not automatically do roll-up of data at insertion time in the library code itself (like whisper does), but rather will be handled through a separate maintenance process that will provide a more general mechanism for transforming your stored data as you see fit.

Timeline-wise, early July looks like when the next release will be out.

Revision history for this message
Steve Keller (skeller-ea) said : 		#2

This answered my question. Thanks Chris!

Looking forward to the new storage backend.

-Steve Keller

Revision history for this message
James R Birchall (james-birchall) said : 		#3

Any update on this? I haven't been able to find the configuration to set up using something that isn't whisper and I have the same requirement.

Revision history for this message
chrismd (chrismd) said : 		#4

Yes, check out the new 1.1 branch. The new database library, Ceres, handles rollup through a separate maintenance process that uses plugins. The rollup plugin lets you determine the rollup behavior exactly by letting you define a custom function for computing the aggregate value. It isn't documented yet unfortunately but that is on my todo list.