Need a way to specify which retention to use

Asked by Esteban Dimitroff on 2013-11-19

Hello,

I have a retentions line like this:

10s:180d;15m:2y

So I'm storing 10 seconds data for six months, and 15 minutes data for 2 years.

The idea behind that amount of data points in the highest resolution retention is that we need to keep historical data with maximum precision available to be able of fine-grain comparing and because we are storing data that may not be looked at in a long time.

We are using the render API with raw data retrieval and some client side graphing tools: A customized version of timeserieswidget (https://github.com/vimeo/timeserieswidget) with Flot charts that allows us to zoom in and out.

So, when showing short time ranges we are fine with highest resolution retention, whether we look last hour or a single hour 3 months ago we want the highest resolution available.

But when comes to look at long range data (ie. a couple of months) we are having some performance issues. To avoid transferring high amount of data, I'm using summarize() function. But as far as I know it aggregates the information from the highest resolution retention, even if it is summarizing in buckets larger than the next retention ones.

As a result, for some data intensive graphs (using around 10 variables and performing some calculations) , and long time ranges, the server is reading really big chunks of data and aggregating it, even when it is already aggregated in whisper database (in next retention), having some high response times in the browser.

So, in summary, is there a way to force whisper to use a specific retention when using the render API per request? Another solution would be to improve the retention selection algorithm to use the 'interval' parameter in summarize() function to choose the appropriate retention.

Thanks in advance, and sorry for the long description, I wanted to be as clear as possible.

Regards,

Esteban

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Esteban Dimitroff
Solved:
Last query:
Last reply:
Revision history for this message
Launchpad Janitor (janitor) said :
#1

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Revision history for this message
Esteban Dimitroff (estebandh) said :
#2

  Status: Open => Expired

:(

Revision history for this message
Dave Rawks (drawks) said :
#3

If you want someone to look at this feature request I suggest you open a feature request issue on github, against graphite-web and/or carbon. If I understand your question it seems like something that would require changes to both.

Revision history for this message
Esteban Dimitroff (estebandh) said :
#4

So it seems that such a feature does not exist?

Thanks for the answer.

Revision history for this message
Sumeet Grover (sgrover) said :
#5

There -is- a work around to fetch data from a specific retention archive in a Whisper database. It's in Perl. The stand-alone CPAN Whisper.pm module can be used. Within this module, comment out Line 161, and you'll notice that fetching data using the wsp_fetch method will now fetch data from the longest-retention archive.

I shall leave more experimenting to everyone.

S.