Graphite

collectd data - whisper only keeping last datapoint

Asked by Charlie Schluting ☃

Hi!

I recently started using collectd to shove metrics in graphite. It worked well on small scale for a few days, but when I rolled it out everywhere, I began noticing that data were getting "lost".

All my other data/whisper files are fine. But when I look at collectd data, I only see the latest datapoint.
I'm using http://github.com/indygreg/collectd-carbon.git to get data to graphite.. and at one point, I had changed the collectd interval from 10s to 60s -- I thought "hmm, perhaps like RRD, I'll have to start over" ... I deleted the whisper files for collectd, and sure enough I started seeing data. But the next day, this is my data:

drawAsInfinite(stats.prod.ops.collectd.apiservices-a012.memory.buffered),1338139200,1340731200,600|None,None,None,None,None,.... followed by ~1000 "None" items.

The mtime on the files are all recent - around 1 minute. Therefore collectd is sending, and I don't have any relay-rules or other craziness causing this - data is getting there... but only one data point is kept :(

Anyone experienced this? Or have ideas for debugging steps?

Thanks :)

 -Charlie

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Nicholas Leskiw
Solved:
Last query:
Last reply:
Revision history for this message
Nicholas Leskiw (nleskiw) said : 		#1

What does your $graphite_root/conf/storage-schemas.conf file look like?
Also, what does whisper-info.py $graphite_root/storage/whisper/stats/prod/ops/collectd/apiservices-a012/memory/buffered.wsp return?

Revision history for this message
Charlie Schluting ☃ (cschluti) said : 		#2

Hi, thanks for the reply. Answers below:

> Nicholas Leskiw requested more information:
> What does your $graphite_root/conf/storage-schemas.conf file look like?

[stats]
priority = 110
pattern = ^stats\..*
retentions = 10:2160,60:10080,600:262974

(and then a similar stats_counts section, which doesn't apply here (and
we blacklist those anyway))

> Also, what does whisper-info.py $graphite_root/storage/whisper/stats/prod/ops/collectd/apiservices-a012/memory/buffered.wsp return?

Ooh, I didn't know about whisper-info. Looks sane to me, but I'm not
super familiar with whisper yet.

Output:

maxRetention: 157784400
xFilesFactor: 0.5
aggregationMethod: average
fileSize: 3302620

Archive 0
retention: 21600
secondsPerPoint: 10
points: 2160
size: 25920
offset: 52

Archive 1
retention: 604800
secondsPerPoint: 60
points: 10080
size: 120960
offset: 25972

Archive 2
retention: 157784400
secondsPerPoint: 600
points: 262974
size: 3155688
offset: 146932

Revision history for this message
Best Nicholas Leskiw (nleskiw) said : 		#3

Your xFilesFactor is set to 0.5, try changing it to 0.

whisper-resize.py --xFilesFactor 0 MIN_LOAD_AVG.wsp 10:21600 60:10080 600:262974

You can set that value in storage-schemas.conf for all new metrics as well.

[stats]
priority = 110
pattern = ^stats\..*
retentions = 10:2160,60:10080,600:262974
xFilesFactor = 0

Keep in mind priority is ignored now, the patterns are tested in the order that they're listed in the file and the first match is used.

Revision history for this message
Charlie Schluting ☃ (cschluti) said : 		#4

Thanks Nicholas Leskiw, that solved my question.