graphite is intermittently failing

Asked by mike

I configured graphite/opentsd integration. I've build the trees on opentsdb and had graphite use opentsdb data.

It is intermittely working, but sometimes is it failing. When it fails, these are the errors from apache errors:

[Tue Nov 04 12:08:27 2014] [error] Internal Server Error: /metrics/find/
[Tue Nov 04 12:08:27 2014] [error] Traceback (most recent call last):
[Tue Nov 04 12:08:27 2014] [error] File "/usr/local/lib64/python2.6/site-packages/django/core/handlers/base.py", line 112, in get_response
[Tue Nov 04 12:08:27 2014] [error] response = wrapped_callback(request, *callback_args, **callback_kwargs)
[Tue Nov 04 12:08:27 2014] [error] File "/opt/graphite/webapp/graphite/metrics/views.py", line 120, in find_view
[Tue Nov 04 12:08:27 2014] [error] matches = list( STORE.find(query, fromTime, untilTime, local=local_only, drilldown=drilldown) )
[Tue Nov 04 12:08:27 2014] [error] File "/opt/graphite/webapp/graphite/storage.py", line 34, in find
[Tue Nov 04 12:08:27 2014] [error] for node in finder.find_nodes(query):
[Tue Nov 04 12:08:27 2014] [error] File "/opt/graphite/webapp/graphite/finders.py", line 56, in find_nodes
[Tue Nov 04 12:08:27 2014] [error] for node in self.fetch_branch(query):
[Tue Nov 04 12:08:27 2014] [error] File "/opt/graphite/webapp/graphite/finders.py", line 265, in fetch_branch
[Tue Nov 04 12:08:27 2014] [error] self.conn.request("GET", "/api/tree/branch?branch=" + query.pattern)
[Tue Nov 04 12:08:27 2014] [error] File "/usr/lib64/python2.6/httplib.py", line 920, in request
[Tue Nov 04 12:08:27 2014] [error] self._send_request(method, url, body, headers)
[Tue Nov 04 12:08:27 2014] [error] File "/usr/lib64/python2.6/httplib.py", line 937, in _send_request
[Tue Nov 04 12:08:27 2014] [error] self.putrequest(method, url, **skips)
[Tue Nov 04 12:08:27 2014] [error] File "/usr/lib64/python2.6/httplib.py", line 824, in putrequest
[Tue Nov 04 12:08:27 2014] [error] raise CannotSendRequest()
[Tue Nov 04 12:08:27 2014] [error] CannotSendRequest

This is the error on the ui when I try to duplicate the same url call:

CannotSendRequest at /metrics/find/
No exception message supplied
Request Method: GET
Request URL: http://10.5.151.236:8090/metrics/find/?_dc=1415131696288&query=*&format=treejson&path=&node=GraphiteTree
Django Version: 1.6.8
Exception Type: CannotSendRequest
Exception Location: /usr/lib64/python2.6/httplib.py in putrequest, line 824
Python Executable: /usr/bin/python
Python Version: 2.6.9

Any ideas what might be happening here?

Question information

Language:
English Edit question
Status:
Answered
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Jason Dixon (jason-dixongroup) said :
#1

Sorry, I'm not sure which Graphite + OpenTSDB integration you're referring to. Do you have a link?

Revision history for this message
mike (gsaray101) said :
#2

Hi Jason:

I install this graphite-web version: https://github.com/manolama/graphite-web/tree/opentsdb_scratch

my opentsdb version is 2.1.0

Do you think those errors are due to network error?

Revision history for this message
mike (gsaray101) said :
#3

 I also see these errors:

CannotSendRequest at /render
No exception message supplied
Request Method: POST
Request URL: http://10.5.151.236:8090/render
Django Version: 1.6.8
Exception Type: CannotSendRequest
Exception Location: /usr/lib64/python2.6/httplib.py in putrequest, line 824
Python Executable: /usr/bin/python
Python Version: 2.6.9

Revision history for this message
Jason Dixon (jason-dixongroup) said :
#4

I wish I could help you out, but that's a fork of graphite-web that's neither maintained nor supported by the Graphite project. The most recent commit on there is well over a year old. Your best bet is to open an issue with that project and hope the author can assist you. Sorry!

Jason

Can you help with this problem?

Provide an answer of your own, or ask mike for more information if necessary.

To post a message you must log in.