1d retentions visualisation or storage problem.

Is the timezone set in your Graphite configuration file? (local-settings.py I think, maybe carbon.conf.)

Dmitry Molchanov <email address hidden> wrote:

>New question #193066 on Graphite:
>https://answers.launchpad.net/graphite/+question/193066
>
>Hello!
>We have a couple of metrics that store daily calculated values.
>At first look the problem looks like an problem of visualisation as we can see here (http://molchanov.pp.ru/savt.png).
>But further investigation shown that data sent to carbon-cache with correct local time (e.g. 12:00PM converted to unix timestamp).
>Timestamp is aligned to beginning of time interval of retention (00:00 GMT) when it is stored to database
>(whisper.py: myInterval = timestamp - (timestamp % archive['secondsPerPoint']) )
>Offset for our TZ (Europe/Moscow) is added when it is displayed, so data point is displayed at 04:00 instead of 12:00PM or 12:00AM at least.
>This makes a couple of problems with reading graphs and data manipulation. How should we handle it?
>
>
>--
>You received this question notification because you are a member of
>graphite-dev, which is an answer contact for Graphite.
>
>_______________________________________________
>Mailing list: https://launchpad.net/~graphite-dev
>Post to : <email address hidden>
>Unsubscribe : https://launchpad.net/~graphite-dev
>More help : https://help.launchpad.net/ListHelp

Timezone is set in my local-settings.py, but not set in carbon.conf. I've found nothing about timezone in carbon.conf or carbon.conf.example.

Timezone set in local-settings is correct:
$ grep TIME_ZONE /opt/graphite/webapp/graphite/local_settings.py
TIME_ZONE = 'Europe/Moscow'

Also I checked every *settings.py for timezone settings - it's correct everywhere it mets.

Sorry, i still need help.

What version of Graphite are you running? There have been a few fixes for timezone handling recently. I cant be sure if they'll fix your problem, but you may want to take a look.

http://bazaar.launchpad.net/~graphite-dev/graphite/main/revision/739
http://bazaar.launchpad.net/~graphite-dev/graphite/main/revision/753

I think this diffs will apply cleanly to 0.9.9 if you're on that. If you're on a recent checkout of trunk then I'd suggest updating

Unfortunately those patches didn't help.
I'm running graphite 0.9.9.

What information can i provide for you to help me fix the problem ?

2 other things I'm wondering:
1. Are you only seeing this behavior for your daily summary data or for all data?
2. Is your system's timezone Europe/Moscow or UTC (or something else)?

If it's only for this data, you can mitigate the problem somewhat by using the timeShift function to shift the data backward in time (apply this if you need to go forward: http://bazaar.launchpad.net/~graphite-dev/graphite/main/revision/759).

1. I see this only for daily summary data because it's aligned to midnight (Start of period) UTC. but it shows in my timezone (+4).
2. As far as I understand it's set to Europe/Moscow:
# md5sum /usr/share/zoneinfo/Europe/Moscow /etc/localtime
200ae8a915612559d032d7787dc9b4f4 /usr/share/zoneinfo/Europe/Moscow
200ae8a915612559d032d7787dc9b4f4 /etc/localtime
# date
Wed Apr 25 11:14:54 MSK 2012

I'll think about timeShift, but anyway it is strange behavio for graphite+carbon to align time of datapoint to the beginning of time period in UTC and then add time offset to it.

We've got a lot of graphs based on daily dummary data and some of them are compicated enough. So we (me and my colleagues) would like to use timeShift only if we have no other choices.

Sorry - I only *just* now understood what was going on. Re-reading the description it sounds like you had a good idea of what was happening already

I'll summarize:

Your daily summary graphs are generated from a process that fires metrics off once per day. Because of the way Whisper rounds times into buckets (as quoted in the description), the timestamp stored is at 00:00 UTC. The expected behavior is that the metric will be stored at the beginning of the day - this is true if you're in UTC, but when the timezone adjustment goes into effect, it's displaying at 4am which is *not* the beginning of your day.

This rounding happens for any retention, but once the retention value goes above 1 hour the timezone conversion starts to shift the time buckets away from the correct local time.

Unfortunately, I don't see a clear route to fix this behavior in the webapp at the moment. There are a few ways you can mitigate the effect though:

1. Use timeShift as above. This is certainly a pain as it involves modifying each target individually
2. Set the timezone to UTC for the affected graphs - this can be done by appending &tz=UTC to the graph URL.
3. Use a retention of 1hour instead of 1day for these graphs and use the 'connected' line mode (&lineMode=connected)

None of these are great, but I think 2 is most reasonable as it's merely appending to the existing URLs and doesn't involve any data migration.

I'll file a bug to track this as well