stats data started to be much behind after updating to 0.9.9

Did you update both carbon and the webapp? The symptoms you are seeing indicate that cache queries are failing. The protocol for cache queries changed in 0.9.9 so both have to be upgraded.

Yes. The whole list of what were upgraded is the following:
Python-2.7.2
Twisted-11.0.0
carbon-0.9.9
graphite-web-0.9.9
whisper-0.9.9

Up. Sorry, I'm still waiting for the reply.

Look for exceptions in /opt/graphite/storage/log/webapp/*.log

The vast majority of exceptions looks like this:

Sun Nov 06 21:18:08 2011 :: Exception Caught
Traceback (most recent call last):
  File "/opt/graphite/webapp/graphite/render/datalib.py", line 229, in fetchData
    cachedResults = CarbonLink.query(dbFile.real_metric)
  File "/opt/graphite/webapp/graphite/render/datalib.py", line 135, in query
    results = self.send_request(request)
  File "/opt/graphite/webapp/graphite/render/datalib.py", line 161, in send_requ
est
    result = self.recv_response(conn)
  File "/opt/graphite/webapp/graphite/render/datalib.py", line 173, in recv_resp
onse
    len_prefix = recv_exactly(conn, 4)
  File "/opt/graphite/webapp/graphite/render/datalib.py", line 186, in recv_exac
tly
    data = conn.recv( num_bytes - len(buf) )
timeout: timed out

And the record from the log/carbon-cache-a/query.log when requesting a graph:

06/11/2011 23:11:59 :: [127.0.0.1:42751] cache query for "ounters.e2_api.search.r.all" returned 0 values

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Actually I still need a solution.

Sorry I let your question expire, I've been on hiatus for a couple weeks but I'm back now. This is the second time I've heard of the second issue you mention, (cache query for "ounters...") where the first character of the metric name is chopped off. I've never been able to reproduce it myself but one other user has reported it, long before 0.9.9 though. I suspect this is due to mixed versions of the code, can you reinstall and use 'setup.py install --force'? If you don't specify --force and it is an upgrade then often times some files don't get overwritten. I have no idea why distutils sometimes does not update files without --force but I've experienced this with other python apps as well.

The exception in datalib.py is a cache query timing out. What does your carbon-cache daemon's cpu usage look like? If it's too high this can happen, also you can increase the cache query timeout in local_settings.py by setting CARBONLINK_TIMEOUT to something more than 1.0 (the default), the unit is seconds.

Yuri, did you ever find a resolution to this problem? I just came across the same issue after updating my retention rates and clearing old data and can't figure out how to fix it.

Any help would be much appreciated!

I am seeing the same thing in my fresh installation. The query.log file shows short metric names. Even under the built-in 'carbon' metrics, the first few characters of the metric name are missing: "carbon.agents.ec" is missing in the example below. I'm masking the example to hide my hostname, but you get the idea:

29/05/2012 14:03:29 :: [127.0.0.1:38171] cache query for "x-xx-xxx-xxx-xx_compute-1_amazonaws_com-a.cache.size" returned 0 values

Should be: "carbon.agents.ecx-xx-xxx-xxx-xx_compute-1_amazonaws_com-a.cache.size"

The charts lag behind for 20 minutes and then they catch up and render all the missing points all at once.

Any ideas what's going on here? I'm guessing it must finally get the query right without truncating the metric name and that's when it catches up.

For what it's worth, if you compare the httpd GET requests with query.log , it looks like the first 16 characters of the metric names are being removed:

$ tail -f /opt/graphite/storage/log/webapp/access.log /opt/graphite/storage/log/carbon-cache-a/query.log

Jesse, Yuri - are your metric names missing the first 16 characters in query.log too?

Is it the webapp that's truncating the metric names when it queries the cache server?

just to add that i too see the issue of lagging data. i'm on 0.9.10. however, i do not appear to have a query.log (only console.log) - how do i enable it to check if i too am missing the first few characters from the cache query?

i tried extending CARBONLINK_TIMEOUT to 5s, but i still get the same result.

I have tracked this problem down in my installation to use of symbolic links in my metric storage.

METRIC NAME: carbon.agents.ny2-laa-005_dev_tradingscreen_com-a.committedPoints
TRUNCATED: s.ny2-laa-005_dev_tradingscreen_com-a.committedPoints

The code that is truncating the name is in storage.py (in __init__)

  if real_fs_path != self.fs_path:
    relative_fs_path = self.metric_path.replace('.', '/') + self.extension
    base_fs_path = realpath(self.fs_path[ :-len(relative_fs_path) ])
    relative_real_fs_path = real_fs_path[ len(base_fs_path)+1: ]
    self.real_metric = relative_real_fs_path[ :-len(self.extension) ].replace('/', '.')

Here are my install details:

FILENAME: /data/psgapps/graphite/storage/whisper/carbon/agents/ny2-laa-005_dev_tradingscreen_com-a/committedPoints.wsp

SYMLINK:
/data/psgapps/graphite/storage/whisper/carbon -> /data/versions/graphitedata00/whisper/carbon

REALFILE:
/data/versions/graphitedata00/whisper/carbon/agents/ny2-laa-005_dev_tradingscreen_com-a/committedPoints.wsp

Here is my local patch to fix the truncated query problem when using symlinks in whisper storage:

# diff render/datalib.py.20130325 render/datalib.py
226c226,228
< cachedResults = CarbonLink.query(dbFile.real_metric)
---
> # BUG: real_metric (from storage.py) breaks in at least some cases when symlinks are used.
> #cachedResults = CarbonLink.query(dbFile.real_metric)
> cachedResults = CarbonLink.query(dbFile.metric_path)

I am not sure if what else this might break. There was probably a reason for the code that attempts to resolve symlinks and then use that result in the cache query. Caveat Emptor.