Disable requests to carbon-cache on front

Hi,

A few words more. I'm fighting with recently emerged delays/hangings in Graphite dashboard that result in either non-rendered graphs or empty list of dashboards. /var/log/gunicorn/graphite.log has lots of errors like '[CRITICAL] WORKER TIMEOUT (pid:7626)' and I am trying to figure out what is the root of that. Each back-end has 50-70Mbit of incoming bandwidth (metrics) but there's no evidence that networks issues exist.

Regards,
Alexander

You need to set CARBONLINK_HOSTS = [] explicitly in local_setings.py. Commenting it out causes it to use the default settinng of CARBONLINK_HOSTS = ["127.0.0.1:7002"] in settings.py.

Jason

Hi Jason,

I've tried it before posting the question. It start logs an exception:

Thu Sep 10 15:49:42 2015 :: Exception Caught
Traceback (most recent call last):
  File "/opt/graphite/webapp/graphite/render/datalib.py", line 231, in fetchData
    cachedResults = CarbonLink.query(dbFile.real_metric)
  File "/opt/graphite/webapp/graphite/render/datalib.py", line 140, in query
    results = self.send_request(request)
  File "/opt/graphite/webapp/graphite/render/datalib.py", line 162, in send_request
    host = self.select_host(metric)
  File "/opt/graphite/webapp/graphite/render/datalib.py", line 114, in select_host
    return self.hash_ring.get_node(metric)
  File "/opt/graphite/webapp/graphite/render/hashing.py", line 83, in get_node
    index %= len(self.ring)
ZeroDivisionError: integer division or modulo by zero

Regards,
Alexander

It looks like we need 'if' block to test if len(self.ring) is zero.

Regards,
Alexander

No, it should correctly set hosts to [] on https://github.com/graphite-project/graphite-web/blob/0.9.x/webapp/graphite/render/datalib.py#L106 based on the logic in https://github.com/graphite-project/graphite-web/blob/0.9.x/webapp/graphite/render/datalib.py#L239-L249. This would mean that hash_ring.get_node in https://github.com/graphite-project/graphite-web/blob/0.9.x/webapp/graphite/render/datalib.py#L113-L114 never actually gets called since hosts is empty.

Are you sure you restarted the webapp after setting CARBONLINK_HOSTS = []?

Jason

Hi Jason,

Absolutely.

Regards,
Alexander

Not sure, then. I'll try to recreate tonight.

Sorry for the delay. I'm not able to reproduce with HEAD of branch 0.9.x. What version are you using?

I am still getting this exact error. I added the emtpy host list becuase I was getting the previous socket error. Now I am getting the ZeroDivisionError.

OS: Ubuntu 14.04.4 LTS
graphite-web: 0.9.12+debian-3
python: 2.7.6

Error:
Wed Nov 16 20:37:34 2016 :: Exception Caught
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/graphite/render/datalib.py", line 231, in fetchData
    cachedResults = CarbonLink.query(dbFile.real_metric)
  File "/usr/lib/python2.7/dist-packages/graphite/render/datalib.py", line 140, in query
    results = self.send_request(request)
  File "/usr/lib/python2.7/dist-packages/graphite/render/datalib.py", line 162, in send_request
    host = self.select_host(metric)
  File "/usr/lib/python2.7/dist-packages/graphite/render/datalib.py", line 114, in select_host
    return self.hash_ring.get_node(metric)
  File "/usr/lib/python2.7/dist-packages/graphite/render/hashing.py", line 83, in get_node
    index %= len(self.ring)
ZeroDivisionError: integer division or modulo by zero

local_settings.py:
CARBONLINK_HOSTS = []
CLUSTER_SERVERS = ["IP1:8080", "IP2:8080"]
LOG_RENDERING_PERFORMANCE = True
LOG_CACHE_PERFORMANCE = True
LOG_METRIC_ACCESS = True
GRAPHITE_ROOT = '/usr/share/graphite-web'
CONF_DIR = '/etc/graphite'
STORAGE_DIR = '/var/lib/graphite/whisper'
CONTENT_DIR = '/usr/share/graphite-web/static'
WHISPER_DIR = '/var/lib/graphite/whisper'
LOG_DIR = '/var/log/graphite'
INDEX_FILE = '/var/lib/graphite/search_index'
DATABASES = {
    'default': {
        'NAME': '/var/lib/graphite/graphite.db',
        'ENGINE': 'django.db.backends.sqlite3',
        'USER': '',
        'PASSWORD': '',
        'HOST': '',
        'PORT': ''
    }
}
TIME_ZONE = 'UTC'