Importing historical summed data gives weird aggregated result

So you're finding out that the particulars of how aggregation works in the whisper database are a bit wonky..

I'm looking at the first example primarily right now. To start, you are going about inspecting the retentions in the correct way, but are a little bit off in the time. Whisper will return data from the highest precision archive (retention definition) that will satisfy the entire period specified. Requesting 3 seconds of data to verify the 1s:3s archive is correct, however Whisper does everything relative to the current time so you instead want to use $(($(date +%s) - 3)) to get the first archive - you should get 3 points in that case returned, all with a value of 1. You can also update several points at once with whisper-update.py so that it happens quicker (before the current second rolls over). Finally, I've noticed that aggregation behaves unexpectedly when there aren't enough points in the first archive to satisfy the 2nd archive (you found a weird edge case). The minimum retention you should use in this case is 1s:5s. Here's a slightly modified script:

Script 1 modified
=======================================
#!/bin/bash

rm -f test.wsp
whisper-create.py --xFilesFactor=0 --aggregationMethod=sum test.wsp 1s:5s 5s:20s
CREATED=$(date +%s)
echo "Created: $CREATED"
whisper-update.py test.wsp $(($(date +%s))):1 $(($(date +%s)-1)):1 $(($(date +%s)-2)):1 $(($(date +%s)-3)):1 $(($(date +%s)-4)):1
echo
echo Using 1s resolution:
whisper-fetch.py --from=$(($(date +%s)-5)) test.wsp
echo

echo Using 5s resolution:
whisper-fetch.py --from=$(($(date +%s)-30)) test.wsp

Output from modified script 1:
=======================================
Created: test.wsp (148 bytes)
Created: 1334621059
[('1334621059', '1'), ('1334621058', '1'), ('1334621057', '1'), ('1334621056', '1'), ('1334621055', '1')]

Using 1s resolution:
1334621055 1.000000
1334621056 1.000000
1334621057 1.000000
1334621058 1.000000
1334621059 1.000000

Using 5s resolution:
1334621040 None
1334621045 None
1334621050 None
1334621055 5.000000

This should look like you expect. The 5 points in the first archive are aggregated into the 1334621055 bucket as a sum. Running it multiple times will show that sometimes those 5 points will end up in a single bucket and sometimes they'll be split between two (depending on what second it's run on).

The 2nd script isn't doing what you expect because whisper-resize.py is 'dumb.' It iterates through the archives in reverse order (lowest resolution and longest retention to highest resolution and shortest retention), pulls the data out of each, and writes it to a new archive. It's best suited for simple resizes - extending a whisper file to cover a longer period at the lowest resolution for example.

Aggregation happens at storage time. Once a point is stored in an archive (starting with the highest resolution archive), each lower archive will read all of the points from the higher archive, aggregate them, and store them. When you store points beyond the first archive in age (through a resize or explicit storage) this propagation doesn't happen. Instead, it's writing into the same bucket several times and overwriting the last one each time.

What you'll need to do is to pre-aggregate your historical data for back-loading. Generally you'll work on getting the data sent to carbon and worry about back-loading later. That way you can also only worry about aggregating for your lowest precision archive (the 1d:730) if you wait a week for live data to load up.

Hope this helps

Hi Michael,

Thank you SO much for this answer. I have two follow-up questions:

1) Would you like me to file a bug about the corner case? I am not entirely sure what to write, but could put something together and refer to this question. Are there are test cases for whisper that I could update accordingly. I can't find any.

2) I'm seeing a problem with whisper if two events happen to happen at the exact same second, or far back in time (second level of resolution). Something that would be great as an addition to carbon would be to allow "my.data +1 1334621057" and possibly

    $ whisper-update.py test.wsp 1334621057:+1

. This would solve both of these issues and also make it way easier to import old data to whisper/graphite (without writing custom aggreagation script yourself). I am well aware of that this would require an additional read for every write, but that would really only be a penalty on that kind of incremental write (which would be optional). Just curious, has this been discussed?

/Jens

I added an additional validation that should catch this case in r740 of trunk just now, it will prevent this case in the future.

As far as the change to whisper-update.py - yeah, it may make sense to add that additional functionality to the script to make this sort of thing easier. Can you file a bug with a description of that enhancement and any other useful additions you may think of?

Great! Thanks for fixing that corner case.

I have filed a bug here: https://bugs.launchpad.net/graphite/+bug/987176