Can't record some data to graphite

Asked by Jaemok Jeong on 2016-06-27

Graphite doesn't record some data.

This fails

echo "lnc.simple 12000 1466499600" | nc -c <server> 2003

Unix epoch time is as follow:

GMT: Tue, 21 Jun 2016 09:00:00 GMT
Your time zone: 2016. 6. 21. pm 6:00:00 GMT+9:00

This succeed

echo "lnc.simple 12000 1466499300" | nc -c <server> 2003

I can't figure out how it can be solved. I am using graphite 0.9.15.

Who can verify it?

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Denis Zhdanov
Solved:
2016-06-28
Last query:
2016-06-28
Last reply:
2016-06-28
Denis Zhdanov (deniszhdanov) said : #1

Graphite time is always UTC. You should send Unix timestamp in UTC timezone.

Jaemok Jeong (jmjeong) said : #2

Please see my script. I sent Unix timestamp in UTC time zone.

echo "lnc.simple 12000 1466499600" | nc -c <server> 2003

Denis Zhdanov (deniszhdanov) said : #3

Then it should work. :)
How do you know that its "failed"? Normally Graphite just accepts data w/o giving you answer back.

Jaemok Jeong (jmjeong) said : #4

In graphite, the data is not recorded. I tested it in two instances. But the result is the same.

Denis Zhdanov (deniszhdanov) said : #5

Then you're trying to push "future" data points, and Graphite ignores future data points.

Jaemok Jeong (jmjeong) said : #6

Thank you for your answer, but...

Please read the message I wrote before you answer.

I sent the the following data.

echo "lnc.simple 12000 1466499600" | nc -c <server> 2003

Unix Epoch time(146649600) is 6/21/2016. Today is 6/28.

Jaemok Jeong (jmjeong) said : #7

http://cl.ly/0Z1A1Z3X0x2j/screen.jpg

Some data is missing even though I sent the data to graphite after 6/21/2016 17:00

Denis Zhdanov (deniszhdanov) said : #8

Sorry, no more ideas left. Something is wrong with your setup, but I'm puzzled what exactly.

Denis Zhdanov (deniszhdanov) said : #9

What's your storage and aggregation scheme? (storage-schemas.conf and storage-aggregation.conf)

Jaemok Jeong (jmjeong) said : #10

After many try to insert data, some data are inserted. The recent inserted data is not processed correctly.

* system-aggregation.conf

[min]
pattern = \.lower$
xFilesFactor = 0.1
aggregationMethod = min

[max]
pattern = \.upper(_\d+)?$
xFilesFactor = 0.1
aggregationMethod = max

[sum]
pattern = \.sum$
xFilesFactor = 0
aggregationMethod = sum

[count]
pattern = \.count$
xFilesFactor = 0
aggregationMethod = sum

[count_legacy]
pattern = ^stats_counts.*
xFilesFactor = 0
aggregationMethod = sum

[default_average]
pattern = .*
xFilesFactor = 0.3
aggregationMethod = average

* storage-schemas.conf

[carbon]
pattern = ^carbon\.
retentions = 10s:6h,1min:90d

[default_1min_for_1day]
pattern = .*
retentions = 10s:6h,1min:6d,10min:1800d

Jaemok Jeong (jmjeong) said : #11

Total number of data = 365 * 24 * 4 (4 data are generated per hour, and are kept in 1 year)

Jaemok Jeong (jmjeong) said : #12

I investigated wsp data file using whisper-dump(http://bazaar.launchpad.net/~amos-shapira/graphite/whisper-dump/revision/733)

JeongJaemokui-Mac-Pro:TmpDisk jmjeong$ python amos/graphite/whisper-dump/whisper/bin/whisper-dump.py denyCount.wsp | head -40
Meta data:
  aggregation method: average
  max retention: 155520000
  xFilesFactor: 0.3

Archive 0 info:
  offset: 52
  seconds per point: 10
  points: 2160
  retention: 21600
  size: 25920

Archive 1 info:
  offset: 25972
  seconds per point: 60
  points: 8640
  retention: 518400
  size: 103680

Archive 2 info:
  offset: 129652
  seconds per point: 600
  points: 259200
  retention: 155520000
  size: 3110400

Denis Zhdanov (deniszhdanov) said : #13

If I understand correctly, you are inserting four data points per hour, and you need to keep it for a year. Then you need to have:

* storage-schemas.conf

[carbon]
pattern = ^carbon\.
retentions = 10s:6h,1min:90d

[default_15min_for_1year]
pattern = .*
retentions = 15m:1y

* system-aggregation.conf

[default]
pattern = .*
xFilesFactor = 0
aggregationMethod = average

Best Denis Zhdanov (deniszhdanov) said : #14

PS: And you need to remove old wsp files after changing schemas.
Or you can stay on old storage schema, and not delete files - but it will be a waste of storage.
Setting xFilesFactor = 0 for your aggregation is mandatory, otherwise Graphite will delete your old data if it got less than 30% of it - http://graphite.readthedocs.io/en/latest/config-carbon.html#storage-aggregation-conf

Jaemok Jeong (jmjeong) said : #15

Thank you. I will try it.