Graphite

Data not being rendered on the front end.Facing issues

Asked by Madhurranjan

Hi,
I have recently begun to use Graphite . I have used the default example_client.py and that data is generated and I can view it. However, we have quite a bit of data that was collected since Jan and we are trying to retrofit the data. We have defined the following kinds of values in storage-schemas.conf

[carbon]
pattern = ^carbon\.
retentions = 60s:120d

[other_data]
pattern = ^Test\.
priority = 100
retentions = 60:43200,1800:17520

[default_1min_for_1day]
pattern = .*
retentions = 60s:1d,1800s:90d

I wrote up a script to transfer data to this . Initially the listener.log showed me errors :19/03/2012 14:27:49 :: invalid line received from client 10.0.1.2:45212, ignoring . Fixed that bit . Then started the carbon-cache via the debug mode and I can see stuff being written to the disk in the front end :

19/03/2012 16:06:22 :: [console] Sorted 1 cache queues in 0.000018 seconds
19/03/2012 16:06:22 :: [updates] wrote 47 datapoints for Test.Data in 0.00154 seconds
19/03/2012 16:06:22 :: [listener] MetricLineReceiver connection with 10.0.1.2:33402 established
19/03/2012 16:06:22 :: [listener] MetricLineReceiver connection with 10.0.1.2:33402 closed cleanly

When I choose the range in the front end , I can only see the timeline being shown but no data is displayed. The web log at times shows me things like:
storage/log/webapp/error.log <==
 storage/log/webapp/error.log <==
[Mon Mar 19 05:27:33 2012] [error] No handlers could be found for logger "cache"
[Mon Mar 19 05:27:34 2012] [error] No handlers could be found for logger "cache"
[Mon Mar 19 05:27:34 2012] [error] ding
[Mon Mar 19 05:27:34 2012] [error] No handlers could be found for logger "cache"
[Mon Mar 19 05:27:35 2012] [error] No handlers could be found for logger "cache"
[Mon Mar 19 05:27:35 2012] [error] ding
[Mon Mar 19 05:29:33 2012] [error] Exception KeyError: KeyError(139962788792288,) in <module 'threading' from '/usr/lib64/python2.6/threading.pyc'> ignored
[Mon Mar 19 05:29:34 2012] [error] Exception KeyError: KeyError(139962788792288,) in <module 'threading' from '/usr/lib64/python2.6/threading.pyc'> ignored
[Mon Mar 19 05:29:34 2012] [error] Exception KeyError: KeyError(139962788792288,) in <module 'threading' from '/usr/lib64/python2.6/threading.pyc'> ignored
[Mon Mar 19 05:29:35 2012] [error] Exception KeyError: KeyError(139962788792288,) in <module 'threading' from '/usr/lib64/python2.6/threading.pyc'> ignored

And
Mon Mar 19 01:35:04 2012 :: Exception Caught
Traceback (most recent call last):
  File "/opt/graphite/webapp/graphite/render/datalib.py", line 226, in fetchData
    cachedResults = CarbonLink.query(dbFile.real_metric)
  File "/opt/graphite/webapp/graphite/render/datalib.py", line 135, in query
    results = self.send_request(request)
  File "/opt/graphite/webapp/graphite/render/datalib.py", line 161, in send_request
    result = self.recv_response(conn)
  File "/opt/graphite/webapp/graphite/render/datalib.py", line 173, in recv_response
    len_prefix = recv_exactly(conn, 4)
  File "/opt/graphite/webapp/graphite/render/datalib.py", line 188, in recv_exactly
    raise Exception("Connection lost")
Exception: Connection lost

Can you point me as to where I am going wrong ? The storage folder is owned by the apache user. We are running Graphite on Centos and followed the instructions on the Graphite website.

Thanks
Ranjan

Question information

Language:
English Edit question
Status:
Answered
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Michael Leinartas (mleinartas) said : 		#1

It looks like at least some of the data is being written if you're able to browse for your metrics from the front-end. You can verify the data on the commandline by finding the .wsp file for one of your metrics (in /opt/graphite/storage) and using the whisper-fetch.py utility which will display all of the points for the past 24 hours. You can also verify the values by copying the graph url and adding &format=raw or &format=csv to the end.

Revision history for this message
Ade Rixon (ade-rixon) said : 		#2

I'm seeing somewhat similar behaviour with a subset of hosts logging via collectd and the collectd-carbon plugin. Checking the whisper database files with whisper-fetch.py shows that values are being entered, but the graphs are coming up blank. If I try to load the URL directly, I get an empty GIF. If I append "format=raw" to the URL, I still get an empty GIF.

The only common factor I can see between all the non-working hosts is that they all have a name that, when translated from the FQDN by collectd-carbon, contains three underscores, in the form:

shortname_xx_ad_local

Those hosts with working graphs have names containing either fewer underscores or none. Is there some odd matching or parsing happening with the metric names here?

I've tried deleting some of the whisper files and letting Graphite re-create them, but the result is the same.

Can you help with this problem?

Provide an answer of your own, or ask Madhurranjan for more information if necessary.

To post a message you must log in.