Graphite data

The data returned by the rawData=true query is exactly what is stored in the underlying whisper database, which is a direct record of the data points sent to carbon. If the graph is looking bad intermittently then it is probably a rendering issue. Do the graphs of this data set look bad sporadically or do they always look bad whenever they have one "scaled down" data point in them?

If you think that your client application should not be sending scaled down data points I would recommend logging the data points it sends along with the timestamps of when it sends them, so when you next find a scaled down data point in your graph we can compare the log to see if whisper is storing an incorrect value somehow (shouldn't be possible but you never know).

We've seen both cases where Graphite's displaying the graph incorrectly intermittently (more common) and during a specific time period no matter how many times I refresh (less common).

I began logging the data points that we are sending to carbon and it looks like we're sending the correct values (e.g. I don't see the scaled down numbers in the log).

-----Original Message-----
From: <email address hidden> on behalf of chrismd
Sent: Tue 10/7/2008 8:24 AM
To: Eric Yeh
Subject: Re: [Question #47363]: Graphite data

Your question #47363 on Graphite changed:
https://answers.launchpad.net/graphite/+question/47363

    Status: Open => Needs information

chrismd requested for more information:
The data returned by the rawData=true query is exactly what is stored in
the underlying whisper database, which is a direct record of the data
points sent to carbon. If the graph is looking bad intermittently then
it is probably a rendering issue. Do the graphs of this data set look
bad sporadically or do they always look bad whenever they have one
"scaled down" data point in them?

If you think that your client application should not be sending scaled
down data points I would recommend logging the data points it sends
along with the timestamps of when it sends them, so when you next find a
scaled down data point in your graph we can compare the log to see if
whisper is storing an incorrect value somehow (shouldn't be possible but
you never know).

--
To answer this request for more information, you can either reply to
this email or enter your reply at the following page:
https://answers.launchpad.net/graphite/+question/47363

You received this question notification because you are a direct
subscriber of the question.

So the intermittent bad graph issue is definitely different than the scaled down data point issue. I would like to help you figure that out as well, perhaps as a separate question? Possibly related to your previous question #41517?

As for the data points getting scaled down I really have no idea but I want to help you figure it out. When you send a data point to carbon, all it does is write it to disk, it does not modify the value in any way. So I am very confused as to how you could send one value and carbon stores another. What we might have to do is put some print statements in carbon-persister.py to see what the actual values its writing to disk are. When I run into strange problems like this my approach is typically to look at the data at each point in the process to see where it is different than what I expect. Since you log the data points you send, the next place to look would be what carbon writes to disk, it should be exactly the same as what you sent. After that the next place to look would be in the rendering code, to see what value it is trying to draw.

I wish I could help in a more direct way but I have never run across this issue before and I have no idea how to re-create it on my end.

I've confirmed that it must have something to do with the way carbon is persisting the data.
I have a minute's worth of data (60 data points) and I reconciled what we sent to carbon vs the data points that were persisted and two of them are way off.

One happens to be off by exactly a factor of 100. The other is off by a factor of 100 + 24.

What we sent:
charts.ES.ESZ8 100488 1223560151
charts.ES.ESZ8 100450 1223560152

What Carbon persisted:
1005.12
1004.5

Eric

-----Original Message-----
From: <email address hidden> on behalf of chrismd
Sent: Thu 10/9/2008 9:49 AM
To: Eric Yeh
Subject: RE: [Question #47363]: Graphite data

Question #47363 on Graphite changed:
https://answers.launchpad.net/graphite/+question/47363

    Status: Open => Answered

chrismd proposed the following answer:
So the intermittent bad graph issue is definitely different than the
scaled down data point issue. I would like to help you figure that out
as well, perhaps as a separate question? Possibly related to your
previous question #41517?

As for the data points getting scaled down I really have no idea but I
want to help you figure it out. When you send a data point to carbon,
all it does is write it to disk, it does not modify the value in any
way. So I am very confused as to how you could send one value and carbon
stores another. What we might have to do is put some print statements in
carbon-persister.py to see what the actual values its writing to disk
are. When I run into strange problems like this my approach is typically
to look at the data at each point in the process to see where it is
different than what I expect. Since you log the data points you send,
the next place to look would be what carbon writes to disk, it should be
exactly the same as what you sent. After that the next place to look
would be in the rendering code, to see what value it is trying to draw.

I wish I could help in a more direct way but I have never run across
this issue before and I have no idea how to re-create it on my end.

--
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.

Did you confirm this by adding print statements to carbon-persister to see the values before the whisper.update_many call? Or did you just look at the output of a graph request with rawData=true? The latter leaves more possibilities open for potential places the values are getting messed up. We need to narrow it down to find where in the code they are getting changed because it is definitely not an explicit or intentional change.

Here's what we can do... I just added a bunch of debug print statements to the carbon scripts in trunk, please pull those changes and use those modified carbon-*.py scripts. When you find a data point in your log that does not match up, grep out any log entries (from $GRAPHITE/carbon/log/*.log) that contain the word DEBUG and post them here. That should give me enough information to see your data at each point in the pipeline so I can figure out where it is getting messed up. Thanks.

I was using the rawData=true.

I've pulled the latest code with the debug statements and have it running now.

Also, I had to change models.py and change the max_length args to maxlength.

I will let you know when I see a mismatch in the data.

-----Original Message-----
From: <email address hidden> on behalf of chrismd
Sent: Thu 10/9/2008 12:40 PM
To: Eric Yeh
Subject: RE: [Question #47363]: Graphite data

Question #47363 on Graphite changed:
https://answers.launchpad.net/graphite/+question/47363

    Status: Open => Needs information

chrismd requested for more information:
Did you confirm this by adding print statements to carbon-persister to
see the values before the whisper.update_many call? Or did you just look
at the output of a graph request with rawData=true? The latter leaves
more possibilities open for potential places the values are getting
messed up. We need to narrow it down to find where in the code they are
getting changed because it is definitely not an explicit or intentional
change.

Here's what we can do... I just added a bunch of debug print statements
to the carbon scripts in trunk, please pull those changes and use those
modified carbon-*.py scripts. When you find a data point in your log
that does not match up, grep out any log entries (from
$GRAPHITE/carbon/log/*.log) that contain the word DEBUG and post them
here. That should give me enough information to see your data at each
point in the pipeline so I can figure out where it is getting messed up.
Thanks.

--
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.

Chris,

It ended up being another program that was sending data (with the same graph name) to graphite but its data is scaled down!

Thanks for your help.

Eric

-----Original Message-----
From: <email address hidden> on behalf of chrismd
Sent: Thu 10/9/2008 12:40 PM
To: Eric Yeh
Subject: RE: [Question #47363]: Graphite data

Question #47363 on Graphite changed:
https://answers.launchpad.net/graphite/+question/47363

    Status: Open => Needs information

chrismd requested for more information:
Did you confirm this by adding print statements to carbon-persister to
see the values before the whisper.update_many call? Or did you just look
at the output of a graph request with rawData=true? The latter leaves
more possibilities open for potential places the values are getting
messed up. We need to narrow it down to find where in the code they are
getting changed because it is definitely not an explicit or intentional
change.

Here's what we can do... I just added a bunch of debug print statements
to the carbon scripts in trunk, please pull those changes and use those
modified carbon-*.py scripts. When you find a data point in your log
that does not match up, grep out any log entries (from
$GRAPHITE/carbon/log/*.log) that contain the word DEBUG and post them
here. That should give me enough information to see your data at each
point in the pipeline so I can figure out where it is getting messed up.
Thanks.

--
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.

Turned out not to be a graphite issue.