Graphite

Strange line progression

Asked by Joeman1

I had this working for several weeks without issue using data writer from Icinga2, but when I went to change the data retention values to allow for more data to be retained, I got the following issue as shown from here:

https://www.joeman1.com/work/icinga_graphite_issue.png

That is drilled down to one hour.

Again, I did have this working fine... I changed the retention values, deleted the .wsp files, and started again, fresh and then this happened. NOTE: This only has ran for about an hour, but I have let it run for a week and same result...

Here are my config files:
storage-schemas.conf

[everything_1min_12months]
priority = 100
pattern = .*
retentions = 1m:365d

I did have:
[icinga]
pattern = ^icinga\.
retentions = 10s:7d,1m:30d,10m:1y

But this didnt work either so I tried to go basic :).

Im using Ubuntu 14.04 and the following versions:
ii graphite-carbon 0.9.12-3 all backend data caching and persistence daemon for Graphite
ii graphite-web 0.9.12+debian-3 all Enterprise Scalable Realtime Graphing
ii libgraphite2-3:amd64 1.2.4-1ubuntu1 amd64 Font rendering engine for Complex Scripts -- library

Here is an example of a wsp file:
whisper-info rta.wsp
maxRetention: 31536000
xFilesFactor: 0.5
aggregationMethod: average
fileSize: 6307228

Archive 0
retention: 31536000
secondsPerPoint: 60
points: 525600
size: 6307200
offset: 28

Let me know if you need any more information.

Thanks!
Joe

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Joeman1
Solved:
Last query:
Last reply:
Revision history for this message
Joeman1 (jgiles) said : 		#1

I forgot the icinga2 versions:

ii icinga2 2.3.5-1~ppa1~trusty1 amd64 host and network monitoring system
ii icinga2-bin 2.3.5-1~ppa1~trusty1 amd64 host and network monitoring system - daemon
ii icinga2-common 2.3.5-1~ppa1~trusty1 all host and network monitoring system - common files
ii icinga2-doc 2.3.5-1~ppa1~trusty1 all host and network monitoring system - documentation
ii icinga2-ido-mysql 2.3.5-1~ppa1~trusty1 amd64 host and network monitoring system - MySQL support

Thanks!
Joe

Revision history for this message
Joeman1 (jgiles) said : 		#2

More information:

So, if I do a whisper-dump on a good wsp file, I see all the data points filled in:

Archive 0 data:
0: 1436208360, 15335424
1: 1436208420, 15511552
2: 1436208480, 15548416
3: 1436208540, 15568896
4: 1436208600, 15601664
5: 1436208660, 15601664
6: 1436208720, 15601664
7: 1436208780, 15601664
8: 1436208840, 15601664
9: 1436208900, 15601664
10: 1436208960, 15601664
11: 1436209020, 15601664
12: 1436209080, 15601664
13: 1436209140, 15605760
14: 1436209200, 15605760
15: 1436209260, 15605760
16: 1436209320, 15605760
17: 1436209380, 15605760
18: 1436209440, 15630336
19: 1436209500, 15740928
20: 1436209560, 15740928
21: 1436209620, 15745024
22: 1436209680, 15749120
23: 1436209740, 15765504
24: 1436209800, 15765504
25: 1436209860, 15765504
26: 1436209920, 15765504
27: 1436209980, 15765504
28: 1436210040, 15765504
29: 1436210100, 15765504
30: 1436210160, 15765504
31: 1436210220, 15765504
32: 1436210280, 15765504
33: 1436210340, 15765504
.....

However, on an icinga wsp file I see lots of holes (Translating into dashes instead of lines:

Archive 0 data:
0: 1436286960, 0.03109700000000000311151104881446372
1: 0, 0
2: 0, 0
3: 0, 0
4: 0, 0
5: 1436287260, 0.030326000000000002260192033531893685
6: 0, 0
7: 0, 0
8: 0, 0
9: 0, 0
10: 1436287560, 0.030975000000000002531308496145356912
11: 0, 0
12: 0, 0
13: 0, 0
14: 0, 0
15: 1436287860, 0.031101998999999998229970543661693227
16: 0, 0
17: 0, 0
18: 0, 0
19: 0, 0
20: 1436288160, 0.031146999000000001600607646423668484
21: 0, 0
22: 0, 0
23: 0, 0
24: 0, 0
25: 0, 0
26: 0, 0
27: 0, 0
28: 0, 0
29: 0, 0
.....

Any ideas?

Thanks!

Revision history for this message
Dave Rawks (drawks.) said : 		#3

whisper files are meant to hold consistent periodic updates. your whisperfile with the gaps looks like you just aren't sending updates at the same periodicity as your schema defines.

Revision history for this message
Joeman1 (jgiles) said : 		#4

Thanks Dave for the reply!

I have the Icinga2 devs looking at this too and have relayed this info to them.

Do you have any experiance with Icinga2 datawriter sending data to carbon?

Thanks again!
Joe

Revision history for this message
Joeman1 (jgiles) said : 		#5

For allot more detailed information:

http://www.monitoring-portal.org/wbb/index.php?page=Thread&threadID=33771

Thanks!
Joe

Revision history for this message
Joeman1 (jgiles) said : 		#6

Ok, issue was resolved by changing the check_interval in Icinga2 back to 1 min. This was a user error :).

Thanks for the reply's!
Joe