Graphite

timeout: timed out

Asked by Stephen

Hello,

Sending data is not seeing in Graphite. I can see the following logs in exception.log:

/var/log/graphite/exception.log
  File "/usr/lib/python2.7/httplib.py", line 541, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/httplib.py", line 574, in _read_chunked
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
timeout: timed out

I have tried CARBONLINK_TIMEOUT from 1.0 to 5.0
# grep CARBONLINK_TIMEOUT local_settings.py
CARBONLINK_TIMEOUT = 5.0

Also tried restarting carbon-cache , carbon-relay and graphite-web, but still no luck.

Ans where in the dmesg of the server showing ::
[ 2120.903735] init: carbon-cache main process ended, respawning
[ 2120.907936] init: carbon-cache main process ended, respawning
[ 2120.912134] init: carbon-cache main process ended, respawning
[ 2120.915969] init: carbon-cache respawning too fast, stopped

Question information

Language:
English Edit question
Status:
Expired
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Denis Zhdanov (deniszhdanov) said : 		#1

it seems that your carbon-cache daemon is dead, because of broken config, probably.
Try to run it with --nodaemon flag to find why, something like

/usr/bin/python /opt/graphite/bin/carbon-cache.py --nodaemon start

(looks like you are using only one instance of carbon)

Revision history for this message
Stephen (ajeeshbkannan) said : 		#4

/usr/bin/python /opt/graphite/bin/carbon-cache.py --nodaemon start this is getting the following error message::

leveldb.LevelDBError: IO error: lock /mnt/graphite/leveldb/tsd-map.db/LOCK: Resource temporarily unavailable

 I am running carbon in three server cluster setup.

Revision history for this message
Denis Zhdanov (deniszhdanov) said : 		#5

leveldb.LevelDBError ? Which version of Graphite are you using?
I know only one LevelDB based Graphite backend - https://github.com/InMobi/level-tsd and I heard nothing about using it in PROD...

Revision history for this message
Denis Zhdanov (deniszhdanov) said : 		#6

Yep, it's looks like lelvel-tsd, wow!
Unfortunately have no idea about it. Just check is path /mnt/graphite/leveldb/tsd-map.db really exist and writable for carbon daemon.
Also check if another carbon already running and hold file /mnt/graphite/leveldb/tsd-map.db/LOCK open.

Revision history for this message
Launchpad Janitor (janitor) said : 		#7

This question was expired because it remained in the 'Needs information' state without activity for the last 15 days.