Invalid retention format?

Asked by Seth Fitzsimmons

Hi.

(Note: this is with graphite/carbon/whisper trunk.)

I was trying to add a new section to storage-schemas.conf with a retention setting of "10s:6h,1m:1w,10m:5y". It initially chokes on the "m", but if I comment out all but the first component, it complains about the "6h":

Traceback (most recent call last):
  File "./bin/carbon-cache.py", line 30, in <module>
    run_twistd_plugin(__file__)
  File "/opt/graphite/lib/carbon/util.py", line 84, in run_twistd_plugin
    runApp(config)
  File "/usr/lib/python2.7/dist-packages/twisted/scripts/twistd.py", line 23, in runApp
    _SomeApplicationRunner(config).run()
  File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 374, in run
    self.application = self.createOrGetApplication()
  File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 434, in createOrGetApplication
    ser = plg.makeService(self.config.subOptions)
  File "/opt/graphite/lib/twisted/plugins/carbon_cache_plugin.py", line 21, in makeService
    return service.createCacheService(options)
  File "/opt/graphite/lib/carbon/service.py", line 111, in createCacheService
    from carbon.writer import WriterService
  File "/opt/graphite/lib/carbon/writer.py", line 40, in <module>
    schemas = loadStorageSchemas()
  File "/opt/graphite/lib/carbon/storage.py", line 126, in loadStorageSchemas
    archives = [ Archive.fromString(s) for s in retentions ]
  File "/opt/graphite/lib/carbon/storage.py", line 110, in fromString
    (secondsPerPoint, points) = whisper.parseRetentionDef(retentionDef)
  File "/usr/local/lib/python2.7/dist-packages/whisper.py", line 96, in parseRetentionDef
    raise ValueError("Invalid unit: '%s'" % pointsUnit)
ValueError: Invalid unit: 'w'

Traceback (most recent call last):
  File "./bin/carbon-cache.py", line 30, in <module>
    run_twistd_plugin(__file__)
  File "/opt/graphite/lib/carbon/util.py", line 84, in run_twistd_plugin
    runApp(config)

  File "/usr/lib/python2.7/dist-packages/twisted/scripts/twistd.py", line 23, in runApp
    _SomeApplicationRunner(config).run()
  File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 374, in run
    self.application = self.createOrGetApplication()
  File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 434, in createOrGetApplication
    ser = plg.makeService(self.config.subOptions)
  File "/opt/graphite/lib/twisted/plugins/carbon_cache_plugin.py", line 21, in makeService
    return service.createCacheService(options)
  File "/opt/graphite/lib/carbon/service.py", line 111, in createCacheService
    from carbon.writer import WriterService
  File "/opt/graphite/lib/carbon/writer.py", line 40, in <module>
    schemas = loadStorageSchemas()
  File "/opt/graphite/lib/carbon/storage.py", line 126, in loadStorageSchemas
    archives = [ Archive.fromString(s) for s in retentions ]
  File "/opt/graphite/lib/carbon/storage.py", line 110, in fromString
    (secondsPerPoint, points) = whisper.parseRetentionDef(retentionDef)
  File "/usr/local/lib/python2.7/dist-packages/whisper.py", line 90, in parseRetentionDef
    points = int( points[:-1] )
ValueError: invalid literal for int() with base 10: '6h'

I managed to work around it by using "10:2160,60:10080,600:262974" instead, but I'd prefer to use the human-readable form.

thanks.
seth

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Nicholas Leskiw
Solved:
Last query:
Last reply:
Revision history for this message
Scott Smith (ohlol) said :
#1

In whisper.py the units available appear to be:

UnitMultipliers = {
  's' : 1,
  'm' : 60,
  'h' : 60 * 60,
  'd' : 60 * 60 * 24,
  'y' : 60 * 60 * 24 * 365,
}

So if you want to specify a week, use 7d.

Revision history for this message
Best Nicholas Leskiw (nleskiw) said :
#2

Try this and let us know if it works.

retentions = 10s:6h,1m:7d,10m:5y

A new testing script was created to validate these:
bin/validate-storage-schemas.py

I may change the detection from an index to a regex match ([0-9])([A-Za-z])
and add sec, min, day, w=7d and mon, (=30 days - I know it's a hack but
people are just expecting a "month" option) so fewer people get nasty
surprises when changing this file.

On Thu, Sep 15, 2011 at 10:30 AM, Scott Smith <
<email address hidden>> wrote:

> Question #171240 on Graphite changed:
> https://answers.launchpad.net/graphite/+question/171240
>
> Scott Smith posted a new comment:
> In whisper.py the units available appear to be:
>
> UnitMultipliers = {
> 's' : 1,
> 'm' : 60,
> 'h' : 60 * 60,
> 'd' : 60 * 60 * 24,
> 'y' : 60 * 60 * 24 * 365,
> }
>
> So if you want to specify a week, use 7d.
>
> --
> You received this question notification because you are a member of
> graphite-dev, which is an answer contact for Graphite.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~graphite-dev
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~graphite-dev
> More help : https://help.launchpad.net/ListHelp
>

Revision history for this message
Seth Fitzsimmons (mojodna) said :
#3

Excellent, that works perfectly.

thanks!

Revision history for this message
Seth Fitzsimmons (mojodna) said :
#4

Thanks Nicholas Leskiw, that solved my question.

Revision history for this message
Scott Smith (ohlol) said :
#5

It seems like the storage schema units in whisper should match those of graphite. Is that possible?

Revision history for this message
Nicholas Leskiw (nleskiw) said :
#6

They should, and I'll work on it.

On Thu, Sep 15, 2011 at 12:10 PM, Scott Smith <
<email address hidden>> wrote:

> Question #171240 on Graphite changed:
> https://answers.launchpad.net/graphite/+question/171240
>
> Scott Smith posted a new comment:
> It seems like the storage schema units in whisper should match those of
> graphite. Is that possible?
>
> --
> You received this question notification because you are a member of
> graphite-dev, which is an answer contact for Graphite.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~graphite-dev
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~graphite-dev
> More help : https://help.launchpad.net/ListHelp
>

Revision history for this message
Nicholas Leskiw (nleskiw) said :
#7

Linked to bug #851428