Allow whisper to store data in a whisper file relative to the timestamp of the latest datapoint instead of current date.

Asked by Styx

Hello,

From my observation and a quick look at the code, it looks like whisper is storing data in the files relative to the current time.

I.E.: I create a new whisper file (empty) with the following retention : 1s:5m 10s:1d 1m:30d 5m:90d 15m:1y
If the current date is January 1st 2016 and for whatever reason I want to store a datapoint from august 15h 2015, this datapoint (and its timestamp) will be stored directly in the last archive (15m:1y)

I am wondering if it would be easy to modify whisper's behavior to the following:
for a given whisper file, as a new datapoint is added to the file:
 => check its timestamp
 ==> if it's timestamp is newer than any timestamp previously recorded, make this new timestamp the relative "starting date" of the file. (if this is the first datapoint added to the file, then make its timestamp be the "starting date")
And then use this "starting date" instead of the current time.

This way, if I take my previous example again, this time, when I add my datapoint from august 15th 2015, it will be stored in all archives, and august 15th 2015 will be considered as the "starting date" of this whisper file.
Then, if I add a new datapoint to that whisper file long after that date, let say on December 25th 2015, then December 25th 2015 becomes the new "starting date" and this new datapoint get stored in all archives, while my previous datapoint gets removed form all archives but the last.

If this makes sense, the idea behind that is to allow to store in whisper some old data but still at a high precision as long as there has been no new datapoint added.

Let's say I have stored somewhere (a cvs file for example) some datapoint from last year at a precision of 10 seconds and just for a duration of 4 hours.
If I wanted to import them to the whisper database, with the current behavior of whisper and with the retention setup indicated above, it would get stored with a 15 minutes precision (so I would lose my 10s precision).
With the modified behavior, it would actually store it with a 10s precision.

Anybody has an idea if this could be done easily ?

If I am not clear in my explanation, let me know and I will try to explain better.

Question information

Language:
English Edit question
Status:
Expired
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Launchpad Janitor (janitor) said :
#1

This question was expired because it remained in the 'Open' state without activity for the last 15 days.