graphite-api not returning data with multiple retentions

Asked by Glen

Hello,

I'm really scratching my head here. We've been running a grafana/graphite-api/carbon/whisper stack for a while now and it's working generally ok. However, I've noticed that if I drill into data in grafana, once I get to a certain level of detail, the chart is blank.

Here is some config. Our storage schema looks like this, store on a 10 sec interval for 7 days, then 1 minute for 2 years.

[Web_Prod]
priority = 90
pattern = ^Production.*.web.*.WebServer.*
retentions = 10s:7d,1m:2y

I can verify this in the whisper files themselves, like this: -

/usr/local/src/whisper/bin/whisper-dump.py /opt/graphite/storage/whisper/Production/Live/web/web2-vm/WebServer/Customer/HPS.wsp | less

Meta data:RETURN)
  aggregation method: average
  max retention: 63072000
  xFilesFactor: 0

Archive 0 info:
  offset: 40
  seconds per point: 10
  points: 60480
  retention: 604800
  size: 725760

Archive 1 info:
  offset: 725800
  seconds per point: 60
  points: 1051200
  retention: 63072000
  size: 12614400

I've noticed the problem only happens, when querying data older than 7 days i..e after it's been averaged to a 60 second interval. If I pick a time period older than 7 days, across a three minute interval, and look directly inside the whisper file, it all looks good: -

/usr/local/src/whisper/bin/whisper-fetch.py --from 1454230700 --until 1454230880 /opt/graphite/storage/whisper/Production/Live/web/web2-vm/WebServer/Customer/HPS.wsp

1454230740 8.000000
1454230800 8.700000
1454230860 8.233333

However, if I query through graphite-api, it returns a 10 second interval (the wrong retention period, because I'm querying older than 7 days), and all items (even the ones that match the timestamps above) are null.

http://www.dashboard.com/render?target=Production.Live.web.web2-vm.WebServer.Customer.HPS&from=1454230700&until=1454230880&format=json&maxDataPoints=1000

[{"target": "Production.Live.web.571854-web2-vm.WebServer.Customer.HPS", "datapoints": [[null, 1454230710], [null, 1454230720], [null, 1454230730], [null, 1454230740], [null, 1454230750], [null, 1454230760], [null, 1454230770], [null, 1454230780], [null, 1454230790], [null, 1454230800], [null, 1454230810], [null, 1454230820], [null, 1454230830], [null, 1454230840], [null, 1454230850], [null, 1454230860], [null, 1454230870], [null, 1454230880]]}]

If I go for a wider time span, I start to get data back, but some are null and some are populated. What am I doing wrong?!

Thanks,
Glen.

Question information

Language:
English Edit question
Status:
Expired
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Launchpad Janitor (janitor) said :
#1

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Revision history for this message
Fergus (fmacgregor) said :
#2

Is it possible to reopen this question? It is an ongoing unresolved issue.

Revision history for this message
Denis Zhdanov (deniszhdanov) said :
#3

Of course, you can reopen it. Looks like not much graphite-api users here - it's not part of Graphite project btw.