All of a sudden metrics getting lost
I'm using graphite to monitor our systems, especially aerospike nodes. Till last week, I was able to see the historical data (3months, 6 months) of the metrics. Two days back when I checked, all the old metrics are gone and only a day of metrics were present.
Today when I checked, only yesterday's data is present.
I read through forums and understood that metrics are retained as per definitions in /etc/carbon/
It has the default entries.
[default_
pattern = .*
retentions = 60s:1d
I checked the creates.log and see the below entries.
15/10/2018 05:04:18 :: new metric instances.
15/10/2018 05:04:18 :: new metric instances.
15/10/2018 05:04:18 :: creating database file /var/lib/
So as per my understanding, the metrics are matching the [default_
My question is how all of a sudden the metrics are matching the [default_
I confirm that no changes are made in the configurations or the metrics exporters.
Also can you explain the term "The retentions line is saying that each datapoint represents 10 seconds, and we want to keep enough datapoints so that they add up to 14 days of data." referring to 'retentions = 10s:14d'?
Does it mean that all data will be retained for 14days? What will happen if i change the 10s to 1s?
Question information
- Language:
- English Edit question
- Status:
- Solved
- For:
- Graphite Edit question
- Assignee:
- No assignee Edit question
- Solved by:
- ak
- Solved:
- 2018-10-26
- Last query:
- 2018-10-26
- Last reply:
- 2018-10-26
Denis Zhdanov (deniszhdanov) said : | #1 |
Whisper (Graphite data storage format) was designed to have predictable disk usage.
So, retentions = 60s:1d means that Graphite will store datapoint for every 60 seconds and it will keep it for 1 day. It will not drop anything after 1 day, it will just update old points with new data, so, file size will be the same, but if you request more then 1 day of data you will get nothing.
So, if you was able to see the historical data before means that files before was created using different retention, not 60s:1d. I do not see any other explanation, so, probably retention was changed recently.
If you using retention 10s:14d, then it will store 1 datapoint every 10 seconds and it will keep data for 14 days, indeed. If you change 10s to 1s then file will be 10 times bigger.
Please check Whisper documentation for details - https:/
You can use multiple archives if you want to balance file size and data retention, e.g.
retention = 10s:1d,60s:7d,5m:1y
ak (ffsak) said : | #2 |
Thanks for the clarification.
I changed the retention to 60s:180d, i.e store data points every 1m and keep it for 180d.
I read that whisper resize command has to be executed to make the changes effective.
Could you guide how to execute that? I tried the options mentioned in the manual , but getting error.
root@ip:
[ERROR] File 'default_
Usage: whisper-resize path timePerPoint:
timePerPoint and timeToStore specify lengths of time, for example:
60:1440 60 seconds per datapoint, 1440 datapoints = 1 day of retention
15m:8 15 minutes per datapoint, 8 datapoints = 2 hours of retention
1h:7d 1 hour per datapoint, 7 days of retention
12h:2y 12 hours per datapoint, 2 years of retention
root@ip:
Usage: whisper-resize path timePerPoint:
timePerPoint and timeToStore specify lengths of time, for example:
60:1440 60 seconds per datapoint, 1440 datapoints = 1 day of retention
15m:8 15 minutes per datapoint, 8 datapoints = 2 hours of retention
1h:7d 1 hour per datapoint, 7 days of retention
12h:2y 12 hours per datapoint, 2 years of retention
root@ip-
Traceback (most recent call last):
File "/usr/bin/
info = whisper.info(path)
File "/usr/lib/
info = __readHeader(fh)
File "/usr/lib/
raise CorruptWhisperF
whisper.
ak (ffsak) said : | #3 |
I figured out. Thanks.