Graphite

whisper-resize.py is losing data

Asked by Tyson Tate

Using whisper-resize.py on my wsp files is dropping a lot of information For example, here's my file before resizing:

https://gist.github.com/tysontate/5500037 (whisper-fetch.py output)

http://i.imgur.com/69ImgKe.png (screenshot)

    $ whisper-info.py used_memory.wsp
    maxRetention: 315360000
    xFilesFactor: 0.5
    aggregationMethod: average
    fileSize: 115372

    Archive 0
    retention: 21600
    secondsPerPoint: 10
    points: 2160
    size: 25920
    offset: 76

    Archive 1
    retention: 86400
    secondsPerPoint: 60
    points: 1440
    size: 17280
    offset: 25996

    Archive 2
    retention: 604800
    secondsPerPoint: 3600
    points: 168
    size: 2016
    offset: 43276

    Archive 3
    retention: 31536000
    secondsPerPoint: 14400
    points: 2190
    size: 26280
    offset: 45292

    Archive 4
    retention: 315360000
    secondsPerPoint: 86400
    points: 3650
    size: 43800
    offset: 71572

After resizing the file with: "whisper-resize.py --nobackup used_memory.wsp 1m:2d 15m:2w 1h:8w 3h:12w 12h:2y", my file now looks like this:

https://gist.github.com/tysontate/5500053 (whisper-fetch.py output)

http://i.imgur.com/SZcETV5.png (screenshot)

    $ whisper-info.py used_memory.wsp
    maxRetention: 63072000
    xFilesFactor: 0.5
    aggregationMethod: average
    fileSize: 92476

    Archive 0
    retention: 172800
    secondsPerPoint: 60
    points: 2880
    size: 34560
    offset: 76

    Archive 1
    retention: 1209600
    secondsPerPoint: 900
    points: 1344
    size: 16128
    offset: 34636

    Archive 2
    retention: 4838400
    secondsPerPoint: 3600
    points: 1344
    size: 16128
    offset: 50764

    Archive 3
    retention: 7257600
    secondsPerPoint: 10800
    points: 672
    size: 8064
    offset: 66892

    Archive 4
    retention: 63072000
    secondsPerPoint: 43200
    points: 1460
    size: 17520
    offset: 74956

Is there something I'm missing? I'd like to, if at all possible, resize the database without losing information. I saw that the latest HEAD version of whisper-resize.py includes a "--aggregate" option, but that option doesn't help (in fact, it makes the data loss worse).

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Tyson Tate
Solved:
Last query:
Last reply:
Revision history for this message
Tyson Tate (tyson-f) said : 		#1

To answer my own question:

What's happening here is that I'm moving from a less-accurate density of points to a more-accurate density of points. Whisper does not interpolate points in between the existing ones, so in the more-accurate database, the new points are null.

The solution is to either re-work whisper-resize.py to add in the missing points or simply use the keepLastValue() graphite filter to fill in the missing points with the last point's value.