Graphite

500 Internal Server Error

Asked by Yousef F on 2018-04-12

I have one panel on Grafana that has 7k+ metrics,, it's a big one but there is a reason behind it.

The panel is working fine on our 0.9.13 although it taking around 17 seconds one one carbon instance [a].

updated recently to 1.1.2 on another box to use more carbon instances[a-j], using 10 now, when I am trying to have the same exact panel it's timing out with the below error message

I am using Django 1.8

When I build the query and adding each query separately i can see the data but once i reach ~3000 it starts timing out. So it's only failing when more data is added to that panel.


<center>
<h2>
<p>Graphite encountered an unexpected error while handling your request.</p>
<p>Please contact your site administrator if the problem persists.</p>
</h2>
<br>
<div>

</div>

<div>
<pre>Traceback (most recent call last):
 File "/usr/lib64/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
 response = wrapped_callback(request, *callback_args, **callback_kwargs)
 File "/graphite/webapp/graphite/render/views.py", line 117, in renderView
 data.extend(evaluateTarget(requestContext, targets))
 File "/graphite/webapp/graphite/render/evaluator.py", line 15, in evaluateTarget
 prefetchData(requestContext, pathExpressions)
 File "/graphite/webapp/graphite/render/datalib.py", line 257, in prefetchData
 for result in STORE.fetch(pathExpressions, startTime, endTime, now, requestContext):
 File "/graphite/webapp/graphite/storage.py", line 179, in fetch
 'fetch for %s' % str(patterns))
 File "/graphite/webapp/graphite/storage.py", line 121, in wait_jobs
 raise Exception(message)
Exception: Timed out after 6.001573s for fetch for [u'xxxx.xxx.*.*.*.*.*.*xxx']

</pre>
</div>

</center>
