Graphite

Graphite Displays No Metrics

Asked by dggreenbaum

Hello,

I have a graphite installation that displays no metrics. I have been able to confirm that carbon is receiving data by watching the .wsp files in /var/lib/carbon/whisper/ and by dumping their contents with whisper-fetch.py. There are no gaps in the data suggesting a problem with the aggregation policy. Also, every metric I expect to be present is represented by a populated .wsp file under /var/lib/carbon/whisper/.

I am running CentOS release 6.6 (Final) with the following versions:
carbon (0.9.12)
Django (1.4.18)
graphite-web (0.9.12)
whisper (0.9.13)

The only other hint I have regarding the trouble is as follows. This occurs when I view the graphite dashboard:

tail -f /opt/graphite/storage/log/*/*

==> /opt/graphite/storage/log/webapp/exception.log <==

==> /opt/graphite/storage/log/webapp/info.log <==
Mon Apr 06 16:34:04 2015 :: [IndexSearcher] performing initial index load
Mon Apr 06 16:34:04 2015 :: [IndexSearcher] reading index data from /opt/graphite/storage/index
Mon Apr 06 16:34:04 2015 :: [IndexSearcher] index reload took 0.000032 seconds (0 entries)
Mon Apr 06 16:34:04 2015 :: [IndexSearcher] performing initial index load
Mon Apr 06 16:34:04 2015 :: [IndexSearcher] reading index data from /opt/graphite/storage/index
Mon Apr 06 16:34:04 2015 :: [IndexSearcher] index reload took 0.000030 seconds (0 entries)
Mon Apr 06 16:34:05 2015 :: graphite.wsgi - pid 31111 - reloading search index
Mon Apr 06 16:34:05 2015 :: [IndexSearcher] performing initial index load
Mon Apr 06 16:34:05 2015 :: [IndexSearcher] reading index data from /opt/graphite/storage/index
Mon Apr 06 16:34:05 2015 :: [IndexSearcher] index reload took 0.000031 seconds (0 entries)
Mon Apr 06 16:46:19 2015 :: find_view query=* local_only=0 matches=0

Where should I look next?

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
dggreenbaum
Solved:
Last query:
Last reply:
Revision history for this message
dggreenbaum (dggreenbaum) said : 		#1

After rereading my message i should clarify a little:

When I say that there are no metrics I am referring to the contents of the Metric sub menu of the web interface. I should be seeing a tree structure of items to select but I am instead greeted with nothing.

EG: http://i.imgur.com/ZBBNc7B.png
There's nothing in the red box.

Revision history for this message
Brett (brettswift) said : 		#2

I have a similar issue, and just posted this: https://answers.launchpad.net/graphite/+question/264691

I think we may have slightly different problems as I see nodes in my tree, whereas you don't.

This is what I see:
https://cloud.githubusercontent.com/assets/723989/7024256/aa246736-dcf8-11e4-8912-12b829327911.png

I don't know if any posts there will help your issue out.

Revision history for this message
dggreenbaum (dggreenbaum) said : 		#3

@brettswift thank you for the response. I don't think our problems are related but I left you what advice I could.

I finally managed to fix this right before I went to bed. The problem stemmed from a disagreement between carbon and graphite about where the .wsp files should be located. Graphite was looking for them at /opt/graphite/storage/whisper whereas carbon was putting them in /var/lib/carbon/whisper. Symlinking /var/lib/carbon/whisper to /opt/graphite/storage/whisper solved the problem immediately.

The reason carbon was writing to the wrong place became obvious after running ps aux | grep carbon. The carbon-cache python process had been provided with the non existent /etc/carbon/carbon.conf as configuration. This is the result of migrating this machine from an unmanaged graphite/whisper/carbon setup installed via yum to a puppet managed setup installed via pip. When I uninstalled the old version of carbon I never stopped the old process. Thus the new process could not start. After killing the carbon process and restarting the carbon service .wsp files were now being written to the correct place and I could delete the symlink.