Unable to find problem wsp file which is filling my disk with log messges

Asked by Dewey Sasser

My console.log file is filing up with a great deal of stack traces like the one below. I'd guess from reading Bug #758261 that one of my .wsp files is corrupt. I have no zero length wsp files, and (brute force alert) running whisper-info or whisper-fetch on all of my wsp files resulted in no abnormal output.

I've tried using strace to figure out what file is being access by this message but it's not useful. When my log filesystem fills up graphite stops answering the sockets and recording data.

How do I track down what is causing me this problem?

Thanks

01/12/2011 21:24:33 :: Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/twisted/python/threadpool.py", line 210, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/twisted/python/context.py", line 59, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.6/dist-packages/twisted/python/context.py", line 37, in callWithContext
    return func(*args,**kw)
  File "/opt/graphite/lib/carbon/writer.py", line 157, in writeForever
    writeCachedDataPoints()
--- <exception caught here> ---
  File "/opt/graphite/lib/carbon/writer.py", line 124, in writeCachedDataPoints
    whisper.update_many(dbFilePath, datapoints)
  File "/usr/local/lib/python2.6/dist-packages/whisper.py", line 515, in update_many
    return file_update_many(fh, points)
  File "/usr/local/lib/python2.6/dist-packages/whisper.py", line 549, in file_update_many
    __archive_update_many(fh,header,currentArchive,currentPoints)
  File "/usr/local/lib/python2.6/dist-packages/whisper.py", line 616, in __archive_update_many
    if __propagate(fh, header, interval, higher, lower):
  File "/usr/local/lib/python2.6/dist-packages/whisper.py", line 421, in __propagate
    (lowerBaseInterval,lowerBaseValue) = struct.unpack(pointFormat,packedPoint)
struct.error: unpack requires a string argument of length 12

Question information

Language:
English Edit question
Status:
Answered
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Launchpad Janitor (janitor) said :
#1

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Revision history for this message
Brian Hatfield (bmhatfield) said :
#2

Apologies that this question expired. I've actually been hit by this problem as well. The trick is to add a log message that outputs the file path that was being written to when the exception was generated.

I've just added such messaging. See here: http://bazaar.launchpad.net/~graphite-dev/graphite/main/revision/630

Revision history for this message
chrismd (chrismd) said :
#3

Thanks guys, I haven't seen this error before but it makes sense in a situation where the wsp file is only partially truncated. I just committed Brian's extra logging to trunk.

Can you help with this problem?

Provide an answer of your own, or ask Dewey Sasser for more information if necessary.

To post a message you must log in.