# Is Graphite auto-scaling my graph?

Asked by Asa Ayers on 2011-06-10

I'm collecting load averages from one of my machines and sending it to graphite by way of StatsD (http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/). I can see that the numbers i'm sending are greater than 1 (example: 1.12841796875) but when graphite draws this it seems to be scaling them down. My graph never reaches the 1.00 line and most of the time is under 0.50 even though I'm watching the numbers going in and they're mostly over 1.0.

## Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Michael Janiszewski
Solved:
2011-06-14
Last query:
2011-06-14
2011-06-11
 chrismd (chrismd) said on 2011-06-10: #1

No what is probably happening is that you're looking at a graph with more datapoints than pixels, which forces Graphite to aggregate the datapoints. The default aggregation method is averaging, but you can change it to summing by applying the cumulative() function to your metrics.

If you're unsure, please post a graph and the raw data values you're expecting to see.

 Nicholas Leskiw (nleskiw) said on 2011-06-10: #2

Are there a lot of less-than-one or zero values in your graph?

If that is true, and if there are more datapoints than there are horizontal
pixels in your graph, graphite will average the values of all the datapoints
at each pixel.

If you want change this behavior, you can use the cumulative() function,
called "Aggregate By Sum" in the UI, to add all the values at each pixel.

You could also increase the width of your graph so there is at least 1 pixel
per datapoint (i.e., a 24 hour graph with 1440 datapoints should have a
width of about 1460-1480 pixels to account for the y-axis labels and
borders.)

If there are no zero or less than one values, please post a screenshot and
the raw data with the &rawData=true parameter.

-Nick

On Fri, Jun 10, 2011 at 5:25 PM, Asa Ayers <

> New question #161034 on Graphite:
>
> I'm collecting load averages from one of my machines and sending it to
> graphite by way of StatsD (
> http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/).
> I can see that the numbers i'm sending are greater than 1 (example:
> 1.12841796875) but when graphite draws this it seems to be scaling them
> down. My graph never reaches the 1.00 line and most of the time is under
> 0.50 even though I'm watching the numbers going in and they're mostly over
> 1.0.
>
> --
> You received this question notification because you are a member of
> graphite-dev, which is an answer contact for Graphite.
>
> _______________________________________________
> Post to : <email address hidden>
>

 Asa Ayers (asa-ayers) said on 2011-06-10: #3

I collect the load average every 10 seconds. Here are the last 41 data points at the moment I generated my graph (http://dl.dropbox.com/u/178903/graphite.png)

 Asa Ayers (asa-ayers) said on 2011-06-10: #4

In testing earlier I reduced the precision of what i'm sending since I don't need 10+ digit load averages. That didn't seem to make any difference.

cumulative() doesn't make any difference for me. I send one number from my server to StatsD every 10 seconds, and every 10 seconds StatsD sends that one number over to graphite.

 chrismd (chrismd) said on 2011-06-10: #5

I can't match up the raw values with the graph exactly because there aren't any timestamps listed. But I do see values over 1 in the raw data and the largest value on the graph is < 0.4. I can also see that rendering aggregation is not occurring because the datapoints are visibly 10-seconds apart in the graph.

One thing that does stand out is that the metric name in the graph, stats.server.loadAvg.1min.ps23, and the metric name you posted datapoints for do not match up (ie. the leading "stats."). Is this getting prepended by statsd? Perhaps statsd is doing some aggregation?

Barring all of that, here is what I would suggest (which I should've stated earlier, sorry):

Generate a graph, just like you already did. Then add &rawData=true to the query-string and capture that. Post both of those things here and we'll be able to match up the graphed datapoints with what is stored on disk precisely.

 Michael Janiszewski (janiszewski-michael) said on 2011-06-10: #6

StatsD averages over its reporting interval (10 seconds by default). If you're sending exactly one datapoint per metric every 10 seconds, you'll need to pass in a sampling rate to StatsD to get it to behave correctly. For the metric above, that would look something like

Alternately, look at the stuff stored by default under stats_counts instead of stats. That's a straight sum per metric rather than the average, which should work in your particular case because you're only sending one value in each interval.

 Asa Ayers (asa-ayers) said on 2011-06-11: #7

Sorry, I misunderstood your raw data request.

Yes, StatsD prepends 'stats.' to everything.

http://dl.dropbox.com/u/178903/graphite2.png

@Michael Janiszewski

I don't see a stats_counts anywhere. I have a statsd.numStats, but it's just constantly 8.0.

I don't see why I need a sampling rate, I'm collecting 100% of the data I want. If I tell it i'm sampling 10% it just multiplies the number by 10 to find out 100% would be like.

 Michael Janiszewski (janiszewski-michael) said on 2011-06-11: #8

Odd that you don't see a stats_counts. The version found at https://github.com/etsy/statsd definitely creates a stats_counts bucket for non-averaged values.

As far as the sampling rate goes, the reason for it is that StatsD was designed (presumably - I'm not one of the devs) to be a real-time aggregation tool. It expects to get multiple values for a given metric over its reporting interval, which it will then average and push up to Carbon. I agree that the sample rate thing is non-intuitive, but there's no other way around the aggregation.

 chrismd (chrismd) said on 2011-06-11: #9

Thanks for explaining the StatsD behavior Michael, that does sound like a very likely cause.

Asa, depending on the type of aggregation you need done the new carbon-aggregator.py daemon may fit the bill. Take a peak at /opt/graphite/conf/aggregation-rules.conf.example and the [aggregator] section of carbon.conf to configure it.

 Asa Ayers (asa-ayers) said on 2011-06-14: #10

It turns out I was running an outdated version of StatsD that didn't have stats_counts. I've corrected that can can keep up with upstream now.

The line of code that was the problem was "var value = counters[key] / (flushInterval / 1000);" Because it only got one data point over the 10 seconds, it averaged the total over 10 seconds. I mistakenly thought it was averaging the total over the number of points received. My options at this point are to simply use stats_counts now that I upgraded, or send the load average every second. I think stats_counts is the better solution.

 Asa Ayers (asa-ayers) said on 2011-06-14: #11

Thanks Michael Janiszewski, that solved my question.