Graphite

Carbon not able to write to whisper file

Asked by Priyanka

I have a very simple Graphite setup and trying it out for the first time. Just 1-node, carbon-cache running. When I send data points (simple-text protocol being used). Whisper files are getting created however no data points get written due to the following errors seen in carbon logs. I have tried deleting whisper files but cannot seem to get around this issue. Any help is greatly appreciated.

More details on my setup..
I used pip to install whisper, carbon and graphite-web on CentOS 7. I have not modified any default configurations in carbon.conf.

python-whisper-0.9.12-4.el7.noarch
graphite2-1.2.2-5.el7.x86_64
python-carbon-0.9.12-7.el7.noarch

03/02/2015 09:57:34 :: Error writing to /opt/graphite/storage/whisper/edp/test/testing/priyanka-dev/TotalDiscoveredItems.wsp
03/02/2015 09:57:34 :: Unhandled Error
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 196, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/opt/graphite/lib/carbon/writer.py", line 158, in writeForever
    writeCachedDataPoints()
--- <exception caught here> ---
  File "/opt/graphite/lib/carbon/writer.py", line 137, in writeCachedDataPoints
    whisper.update_many(dbFilePath, datapoints)
  File "/usr/lib/python2.7/site-packages/whisper.py", line 577, in update_many
    return file_update_many(fh, points)
  File "/usr/lib/python2.7/site-packages/whisper.py", line 611, in file_update_many
    __archive_update_many(fh,header,currentArchive,currentPoints)
  File "/usr/lib/python2.7/site-packages/whisper.py", line 635, in __archive_update_many
    currentString += struct.pack(pointFormat,interval,value)
struct.error: 'L' format requires 0 <= number <= 4294967295

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Dave Rawks
Solved:
Last query:
Last reply:
Revision history for this message
Dave Rawks (drawks) said : 		#1

Please post your storage schema and also maybe a sample of the data you are sending. The error indicates that you are overflowing the type of one of the fields in an internal struct, the likely cause would be a really wierd schema. I.E. one that has a rentention period of a gajillion years or something.

Revision history for this message
Priyanka (priyanka-mitra) said : 		#2

Thanks Dave for your response. This is my sample metric data point which is being sent to carbon. Let me know if you need to look at any other configuration. I just have 1 schema for now. The schema file is getting created correctly.

Sample metric datapoint (uploads every 1 sec)
<timestamp> <metric_name> <metric_value>
1422992255552 edp.test.services.priyanka-dev.TotalDiscoveredItems 10
1422992260572 edp.test.services.priyanka-dev.TotalDiscoveredItems 20
1422992255573 edp.test.services.priyanka-dev.TotalDiscoveredItems 30
1422992255575 edp.test.services.priyanka-dev.TotalDiscoveredItems 40
1422992255577 edp.test.services.priyanka-dev.TotalDiscoveredItems 50
1422992255579 edp.test.services.priyanka-dev.TotalDiscoveredItems 60
1422992255580 edp.test.services.priyanka-dev.TotalDiscoveredItems 70

storage-schemas.conf (I intend to use the last [default] for all the schema)

[carbon]
pattern = ^carbon\.
retentions = 60:90d

[default_1min_for_1day]
pattern = .*
retentions = 60s:1d

[default]
pattern = .*
retentions = 1s:8h, 1m:3d, 5m:8d, 15m:32d, 1h:1y

====================================
Schema dump

[root@graphitetest priyanka-dev]# whisper-dump.py TotalDiscoveredItems.wsp | more
Meta data:
  aggregation method: average
  max retention: 86400
  xFilesFactor: 0.5

Archive 0 info:
  offset: 28
  seconds per point: 60
  points: 1440
  retention: 86400
  size: 17280

Archive 0 data:
0: 0, 0
1: 0, 0
2: 0, 0

===================================
creates.log

03/02/2015 09:55:34 :: new metric edp.test.testing.priyanka-dev.TotalDiscoveredItems matched schema default_1min_for_1day
03/02/2015 09:55:34 :: new metric edp.test.testing.priyanka-dev.TotalDiscoveredItems matched aggregation schema default_average
03/02/2015 09:55:34 :: creating database file /opt/graphite/storage/whisper/edp/test/testing/priyanka-dev/TotalDiscoveredItems.wsp (archive=[(60, 1440)] xff=0.5 agg=average)
03/02/2015 11:36:51 :: new metric edp.test.testing.priyanka-dev.TotalDiscoveredItems matched schema default_1min_for_1day
03/02/2015 11:36:51 :: new metric edp.test.testing.priyanka-dev.TotalDiscoveredItems matched aggregation schema default_average
03/02/2015 11:36:51 :: creating database file /opt/graphite/storage/whisper/edp/test/testing/priyanka-dev/TotalDiscoveredItems.wsp (archive=[(60, 1440)] xff=0.5 agg=average)

Revision history for this message
Priyanka (priyanka-mitra) said : 		#3

Small correction, the metric name is edp.test.testing.priyanka-dev.TotalDiscoveredItems and not edp.test.services.priyanka-dev.TotalDiscoveredItems as stated above.

Revision history for this message
Best Dave Rawks (drawks) said : 		#4

That timestamp is either way in the future or is in milliseconds. Either way it is bigger than a long integer and cannot be used. I suspect that whatever you're using to generate/send your metrics is using millisecond epoch and not the expected and documented standard second epoch.

Revision history for this message
Priyanka (priyanka-mitra) said : 		#5

Yes, that was it. I was sending timestamp in millis. I can see the data points getting written into the whisper file after converting the timestamp in seconds.
One more quick question..is there any limitations for metric name for instance?
Many thanks for your help!

Revision history for this message
Priyanka (priyanka-mitra) said : 		#6

I mean length of metric name...

Revision history for this message
Denis Zhdanov (deniszhdanov) said : 		#7

AFAIR there's limit for total path and .wsp file name in linux (usually 255 chars for filename and 4096 chars for all path)

Revision history for this message
Priyanka (priyanka-mitra) said : 		#9

Thanks Denis!

Revision history for this message
Priyanka (priyanka-mitra) said : 		#10

Thanks Dave Rawks, that solved my question.