Graphite

Retrieve data with a different step

Asked by Dani

Hello,

Is there any way to retrieve data with a different step than the one configured at storage-schemas. conf?

For example, I have a few metrics with a retention period of one month with a 60 seconds step. I'm not using graphite to render my graphs, just to get the rawData. If i call graphite and retrieve a whole day of data, i'll get 1440 datapoints.

I'll be easier if I can change the step and tell it to aggregate the data and maybe change the step to 15 minutes or something like that.

But I do not want to change the wps resolution, I'd like to choose wich step to use.

If I call /render/?target=my.metric&from=00%3A00_20110803&until=23%3A59_20110803&rawData=true

I get
my.metric,1312322460,1312408800,60|lots of datapoints

If i only had an option like &step=300 or a function to do that.

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Nicholas Leskiw
Solved:
Last query:
Last reply:
Revision history for this message
Nicholas Leskiw (nleskiw) said : 		#1

You can use the summarize() function. The description for this function isn't on readthedocs.org, I'll try and add it today. Basically, &target=summarize(my.metric.name,"15min") will give you 15 minute totals (sum, not average). Read the URL API on http://graphite.readthedocs.org for how to specify times.

Dani <email address hidden> wrote:

>New question #166935 on Graphite:
>https://answers.launchpad.net/graphite/+question/166935
>
>Hello,
>
>Is there any way to retrieve data with a different step than the one configured at storage-schemas. conf?
>
>For example, I have a few metrics with a retention period of one month with a 60 seconds step. I'm not using graphite to render my graphs, just to get the rawData. If i call graphite and retrieve a whole day of data, i'll get 1440 datapoints.
>
>I'll be easier if I can change the step and tell it to aggregate the data and maybe change the step to 15 minutes or something like that.
>
>But I do not want to change the wps resolution, I'd like to choose wich step to use.
>
>If I call /render/?target=my.metric&from=00%3A00_20110803&until=23%3A59_20110803&rawData=true
>
>I get
>my.metric,1312322460,1312408800,60|lots of datapoints
>
>If i only had an option like &step=300 or a function to do that.
>
>
>--
>You received this question notification because you are a member of
>graphite-dev, which is an answer contact for Graphite.
>
>_______________________________________________
>Mailing list: https://launchpad.net/~graphite-dev
>Post to : <email address hidden>
>Unsubscribe : https://launchpad.net/~graphite-dev
>More help : https://help.launchpad.net/ListHelp

Revision history for this message
Dani (dsolsona) said : 		#2

Hello,

Thanks for the answer, but I expressed myself incorrectly.

When I said aggregate, I really meant consolidate.

summarize() gives me the sum of those 15 minuts datapoints, but what I really need is those 15 datapoints consolidated in just 1 datapoint.

Any idea?

Revision history for this message
Nicholas Leskiw (nleskiw) said : 		#3

I think the summarize function does exactly what you want.

Example:

Request:
http://myserver/render/?width=803&height=308&_salt=1312494243.924&from=-30minutes&target=summarize(Server.apache1.BusyWorkers,
"5min")&target=Server.apache1.BusyWorkers&rawData=true

Output:
summarize(Server.apache1.BusyWorkers,
"5min"),1312492500,1312494600,300|255.0,224.0,289.0,249.0,252.0,247.0,None
Server.apache1.BusyWorkers,1312492500,1312494300,60|57.0,39.0,53.0,53.0,53.0,42.0,34.0,50.0,52.0,46.0,46.0,58.0,63.0,66.0,56.0,56.0,45.0,41.0,55.0,52.0,48.0,49.0,54.0,48.0,53.0,52.0,49.0,53.0,51.0,42.0

Notice only 6 datapoints are returned in the first line for that 30 minute
period.

-Nick

On Thu, Aug 4, 2011 at 10:31 AM, Dani
<email address hidden>wrote:

> Question #166935 on Graphite changed:
> https://answers.launchpad.net/graphite/+question/166935
>
> Status: Answered => Open
>
> Dani is still having a problem:
> Hello,
>
> Thanks for the answer, but I expressed myself incorrectly.
>
> When I said aggregate, I really meant consolidate.
>
> summarize() gives me the sum of those 15 minuts datapoints, but what I
> really need is those 15 datapoints consolidated in just 1 datapoint.
>
> Any idea?
>
> --
> You received this question notification because you are a member of
> graphite-dev, which is an answer contact for Graphite.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~graphite-dev
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~graphite-dev
> More help : https://help.launchpad.net/ListHelp
>

Revision history for this message
Dani (dsolsona) said : 		#4

Hi Nicholas,

The problem with summarize is that in the end, it's a sum of data. I don't really want a sum.

What I really need is the same data with less precission. I think it's just the same as when whisper consolidates data.

I can have a wsp with a retention of 1 day with datapoints every 60 seconds, and another archive inside the same wsp with a retention of 1 week with datapoints every 900 seconds.

I've I retrieve data from the first day whisper will give me n datapoints (one for each minut) but I want to retrieve data from last day, whisper will give n datapoints, but one for each 15 minutes. Is that right? But whisper does not sum the data, as far as I know, I'm don't now if whispers saves an average of 15 '1minut_datapoint' to make 1 '15minutes_datapoint'.

Do you understand what I mean?

Thanks in advance.

Revision history for this message
Best Nicholas Leskiw (nleskiw) said : 		#5

You have 3 options:

1. Use scale() in conjunction with summarize.
Example:
&target=scale(summarize(Server.apache1.BusyWorkers, "5min"),0.2)
You'll have to modify the scale factor to match the inverse of the number of
datapoints consolidated, (5 datapoints = 1/5 = 0.2. 15 datapoints = 1/15 =
0.06666...)

2. Use movingAverage()
It will not change the precision, but any datapoint you request will be the
average of the last N datapoints.

3. Create a new function called consolidate() in
webapp/graphite/render/functions.py that does what you want. I'd copy the
summarize() function from the same file and use it as a framework - all you
should need to do is add a division operation to the summation. If you add a
diff / patch to a bug report, I'll review it and try to get it added to
trunk.

-Nick

On Fri, Aug 5, 2011 at 4:16 AM, Dani
<email address hidden>wrote:

> Question #166935 on Graphite changed:
> https://answers.launchpad.net/graphite/+question/166935
>
> Status: Answered => Open
>
> Dani is still having a problem:
> Hi Nicholas,
>
> The problem with summarize is that in the end, it's a sum of data. I
> don't really want a sum.
>
> What I really need is the same data with less precission. I think it's
> just the same as when whisper consolidates data.
>
> I can have a wsp with a retention of 1 day with datapoints every 60
> seconds, and another archive inside the same wsp with a retention of 1
> week with datapoints every 900 seconds.
>
> I've I retrieve data from the first day whisper will give me n
> datapoints (one for each minut) but I want to retrieve data from last
> day, whisper will give n datapoints, but one for each 15 minutes. Is
> that right? But whisper does not sum the data, as far as I know, I'm
> don't now if whispers saves an average of 15 '1minut_datapoint' to make
> 1 '15minutes_datapoint'.
>
> Do you understand what I mean?
>
> Thanks in advance.
>
> --
> You received this question notification because you are a member of
> graphite-dev, which is an answer contact for Graphite.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~graphite-dev
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~graphite-dev
> More help : https://help.launchpad.net/ListHelp
>

Revision history for this message
Dani (dsolsona) said : 		#6

Hello,

Scale & summarize does the job but I've also created Bug #822560 with a patch to add consolidate function.

Thanks for all the help.

Revision history for this message
Dani (dsolsona) said : 		#7

Thanks Nicholas Leskiw, that solved my question.