flushinterval

Asked by Mohan on 2012-05-28

My statsd flushinterval is 1000.

Retentions are

retentions = 10:2160,60:10080,600:262974

I have turned on statsd logging and I see that in a second several values are being pushed to graphite as intended. I am using stats_count. The graph though seems to lose the precision and it looks like the byte count is being averaged.
Do I reduce the flush interval to increase the precision ? Should it be less than 1000ms specificed above.

Apology if it wasn't clear. I have edited the question.

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Mohan
Solved:
2012-06-06
Last query:
2012-06-06
Last reply:
2012-06-04
Michael Leinartas (mleinartas) said : #1

If your first retention is 10 (1 point every 10 seconds) your flushInterval in statsd should match it as 10000 (10,000 milliseconds). With a flushInterval of 1000 you'll lose data as statsd will keep overwriting the previous value 9 seconds out of the 10.

Mohan (radhakrishnan-mohan) said : #2

I have retentions = 1:2160,10:2160,60:10080,600:262974 now.

to match the flushinterval of 1000ms. The graph though is not sufficiently granular. These values are distinctly seen when I fire

http://localhost/render?from=-2minutes&bgcolor=FFFFFF&width=550&height=310&fgcolor=darkblue&target=alias%28stats_counts.nio.auth.client.bytes,%22Byte%20Count%22%29&title=Bytes%20vs%20Time&vtitle=Bytes

29 May 18:04:19 - nio.auth.client.bytes:366|c
29 May 18:04:19 - nio.auth.client.bytes:486|c
29 May 18:04:19 - nio.auth.client.bytes:4|c
29 May 18:04:19 - nio.auth.client.bytes:164|c
29 May 18:04:19 - nio.auth.client.bytes:584|c
29 May 18:04:19 - nio.auth.client.bytes:418|c
29 May 18:04:19 - nio.auth.client.bytes:268|c

Thanks.

Mohan (radhakrishnan-mohan) said : #3

Apology. These values are *NOT* distinctly seen when I fire the url.

Michael Leinartas (mleinartas) said : #4

You listed different retentions in the original comment from your last - did you recently change your retentions in your storage-schemas? If so you'll need to migrate the existing files to the new schema using whisper-resize.py - storage-schemas.conf only influences the creation of new metrics. If this is the case, it would explain why you're not seeing the correct granularity on your graphs

Mohan (radhakrishnan-mohan) said : #5

I am missing something obvious here. I have resized the metric databases and restarted carbon-cache. But the graph is not reflecting the values. I was expecting individual values to show distinctly.

Mohan (radhakrishnan-mohan) said : #6

Some more data after debugging. The actual data that is returned as a csv is shown below. This is different from the individual byte counts that I was looking for.
Now the graph format has changed and it shows only dots and not a continuous line. Don't know the reason.

maxRetention: 1080
xFilesFactor: 0.5
aggregationMethod: average
fileSize: 12988

Archive 0
retention: 1080
secondsPerPoint: 1
points: 1080
size: 12960
offset: 28

stats_counts.nio.auth.client.bytes 5/30/2012 7:44 24008
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44 38000
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44
stats_counts.nio.auth.client.bytes 5/30/2012 7:44 38000

Nicholas Leskiw (nleskiw) said : #7

Looks like you're only getting data every 10 seconds. Can you check your datasource to verify it's sending data every second?

Michael Leinartas (mleinartas) said : #8

It looks like now you may have the opposite problem I guessed at above - you're only getting one data point every 10 seconds stored. It looks like your flushInterval may actually be at 10000 instead of 1000. If you're sure you restarted after changing you might want to try setting debug: true in your rdioConfig.js to see the log of updates from statsd. That will also confirm that statsd has actually picked up the correct config file.

Mohan (radhakrishnan-mohan) said : #9

I had turned on dumpMessages in statsd config and I still see multiple values being sent in a few seconds. flushinterval in statsd config is only '1000'.

{
  graphitePort: 2003
, graphiteHost: "localhost"
, port: 8125
, dumpMessages: true
, flustInterval: 1000
}

Thanks.

Mohan (radhakrishnan-mohan) said : #10

Ok. I turned on debug and it looks like an aggregated amount is sent by statsd.

4 Jun 17:05:14 - Counters:
{ 'nio.auth.client.bytes': 36000 }

Assuming the config above was copy-and-pasted, the problem looks to be a typo in the flushInterval setting - it's pasted above as 'flustInterval' but should be 'flushInterval'

I hope that fixes it for you

Mohan (radhakrishnan-mohan) said : #12

Thanks. Apology for the spelling mistak. It works now. The interval at which flushers is 1000ms.

It looks like I need to reduce the interval to less than a second. Because multiple values are received by statsd in a second. So it has to flush it sooner. 1000ms is still not giving me these individual values.

29 May 18:04:19 - nio.auth.client.bytes:366|c
29 May 18:04:19 - nio.auth.client.bytes:486|c
29 May 18:04:19 - nio.auth.client.bytes:4|c
29 May 18:04:19 - nio.auth.client.bytes:164|c
29 May 18:04:19 - nio.auth.client.bytes:584|c
29 May 18:04:19 - nio.auth.client.bytes:418|c
29 May 18:04:19 - nio.auth.client.bytes:268|c

Mohan (radhakrishnan-mohan) said : #13

Would like to close and open a new one. Thanks.