Graphite

MemoryError when running an intensive maximumAbove

Asked by Gerhard Lazu

When running a MaximumAbove over ~3k whisper files on a dedicated 16GB of DDR3 + Intel320 SSD, I'm getting this from the graphite-web:

Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "./graphite/render/views.py", line 105, in renderView
    seriesList = evaluateTarget(requestContext, target)
  File "./graphite/render/evaluator.py", line 10, in evaluateTarget
    result = evaluateTokens(requestContext, tokens)
  File "./graphite/render/evaluator.py", line 21, in evaluateTokens
    return evaluateTokens(requestContext, tokens.expression)
  File "./graphite/render/evaluator.py", line 28, in evaluateTokens
    args = [evaluateTokens(requestContext, arg) for arg in tokens.call.args]
  File "./graphite/render/evaluator.py", line 21, in evaluateTokens
    return evaluateTokens(requestContext, tokens.expression)
  File "./graphite/render/evaluator.py", line 24, in evaluateTokens
    return fetchData(requestContext, tokens.pathExpression)
  File "./graphite/render/datalib.py", line 227, in fetchData
    dbResults = dbFile.fetch( timestamp(startTime), timestamp(endTime) )
  File "./graphite/storage.py", line 295, in fetch
    (timeInfo,values) = whisper.fetch(self.fs_path, startTime, endTime)
  File "/usr/local/lib/python2.6/dist-packages/whisper.py", line 643, in fetch
    return file_fetch(fh, fromTime, untilTime)
  File "/usr/local/lib/python2.6/dist-packages/whisper.py", line 709, in file_fetch
    unpackedSeries = struct.unpack(seriesFormat, seriesString)
MemoryError

Memcached has 4G allocated, currently using 205MB. The uwsgi handling the request does not crash or harakiri itself, it just stops working after about 10'.

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Gerhard Lazu
Solved:
Last query:
Last reply:
Revision history for this message
Michael Leinartas (mleinartas) said : 		#1

Whisper loads the entire requested series in memory twice (raw binary + unpacked) so it's not too surprising that loading 3000 simultaneously would exhaust available memory.

Note that memcached isn't involved at this point - this is memory being used within the Python interpreter

I dont have any solution to this at this point. Feel free to file a bug describing the problem

Revision history for this message
Gerhard Lazu (gerhard-p) said : 		#2

I have increased the memory for the uwsgi workers from 512MB to 1024MB, it seems to have fixed the issue. If it comes back, I'll re-open the thread. Thanks!