servers show up in tree, but no graph data

Asked by Brett

I have had graphite working previously, but with my current configuration I see no graph data.

I'm using puppet to do the install and config, and have a vagrant box to demonstrate:

https://github.com/brettswift/puppet-sensu_suite/tree/ea0ee0bcfd2235b13512c8ae32974e6f71fde73c

using that commit (or currently what's on the feature/separate_graphite_grafana branch.

a Vagrant up of the 'sensuserver' box first, followed by the 'graphite' box will give you the result I am seeing.

Where can I look to find out what might cause this?

Question information

Language:
English Edit question
Status:
Expired
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
dggreenbaum (dggreenbaum) said :
#1

Have you confirmed that carbon is actually getting data? There are a couple of ways to do this:
* Look for .wsp files corresponding to your metrics. They should be in a directory structure similar to the metric name beneath /var/lib/carbon/whisper/ or /opt/graphite/storage/whisper.
* ls those files. When were they last updated? If it's not recently make sure carbon can write to them. (don't forget SELinux)
* Try tailing /var/log/carbon/*. See what pops up. Pay special attention to /var/log/carbon/listener.log as this should have an entry when carbon receives data.

If the above doesn't get you anywhere dump some of the data in those files using /usr/bin/whisper-fetch.py. Look for null values. This may indicate a problem with your aggregation rules. This may be helpful: https://theholyjava.wordpress.com/2014/05/05/graphite-shows-metrics-but-no-data-troubleshooting/

Revision history for this message
Brett (brettswift) said :
#2

Using the puppet module all the *.wsp files live here: /opt/graphite/storage/whisper/graphite/, and the files are being updated.

I do see this in the listener log: (using a tip from here: http://stackoverflow.com/questions/26149849/show-graphite-invalid-lines)

07/04/2015 08:59:11 :: [listener] invalid line received from client 172.17.2.10:33699, ignoring
07/04/2015 08:59:11 :: [listener] invalid line - [ ]

I only see that one node .. other nodes are showing up in the tree (172.17.2.11, and *.12 as well), but I don't see them in the listener log. the *.10 server is not the same server as the graphite server.

Revision history for this message
Brett (brettswift) said :
#3

This may be a problem with my sensu / relay configuration and not graphite.

I see this in my sensu-server.log
{"timestamp":"2015-04-06T12:28:46.331789-0600","level":"info","message":"handler extension output","extension":{"type":"extension","name":"relay","mutator":"metrics"},"output":"","status":0}

specifically: output:""

These checks and relays were working before, I know the checks work as when I run them manually they output the right values.

If the whisper files are there, it's probably not safe to assume those files contain data points - maybe it's all null data?

Any ideas on where I could look next?

Revision history for this message
dggreenbaum (dggreenbaum) said :
#4

You can view the contents of the *.wsp files by running:

/usr/bin/whisper-fetch.py [FILENAME]

This way you can confirm data is being written.

Revision history for this message
Launchpad Janitor (janitor) said :
#5

This question was expired because it remained in the 'Open' state without activity for the last 15 days.