Graphite

Logs not matching graphs

Asked by Bryan Peterson

We have a metric that we increment every time a user performs a certain action on our website, but the graphs don't seem to be accurate.

So going off this hunch, we invested the updates.log of carbon and discovered that the action had happened over 4 thousand times today(using grep and wc), but according the Integral result of the graph it returned only 220ish.

What would be the cause of this?

We are using:

graphite 0.9.6 with statsD (etsy)

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Bryan Peterson
Solved:
Last query:
Last reply:
Revision history for this message
cbrinley (cbrinley) said : 		#1

couple things to consider:
do you have aggregation rules setup? From what I am seeing the last value received for a given metric ID will take precedence if no rule is defined (such as average or sum).
so for example if your interval for some metric is 1 minute and you receive 10 values for that metric in a minute only the 10th value is applied barring any aggregation rules like "sum".

So over time you may have received 40K+ metric values but only some small fraction of them actually persist on disk because there is no rule to do anything other than replace the last value. if you are doing event driven graphs where timing between metric values may vary this is worth investigating.

secondly
in your url that displays the graph you may want to add another url parameter by hand rawData=true
example: graphite.com?target=foo.bar&rawData=true (data may need to be lower case i don't remember off hand)
this should dump the raw data to screen so that things like graph size are not causing the data points to be averaged together. Based on the size of the difference i don't think this is the case however but may give you some more insight.

Revision history for this message
Bryan Peterson (lazyshot) said : 		#2

We don't have any aggregation rules set up, but we figured out our issue, which was our retention settings.