Graphite

Dataset filled with nulls?

Asked by Lisa

I'm trying to figure out what data I'm actually sending to Graphite. when I query the raw data, it appears to be filled with null vales for every minute.

E.g. I ask for "http://graphite.example.com/render?target=server.events.hourly_created&format=json"

I get:
[{"target": "server.events.hourly_created", "datapoints": [[null, 1395619260], [null, 1395619320], [null, 1395619380], [null, 1395619440], [null, 1395619500], [null, 1395619560], [null, 1395619620], [null, 1395619680], [null, 1395619740], [null, 1395619800], [null, 1395619860], [null, 1395619920], [null, 1395619980], [null, 1395620040], [null, 1395620100], [null, 1395620160], [null, 1395620220], [null, 1395620280], [null, 1395620340], [null, 1395620400], [null, 1395620460], [null, 1395620520], [null, 1395620580], [null, 1395620640], [null, 1395620700],...

Is the data before 1395619260 (2014-03-23 17:01:00 -0700, 24 hours ago) deleted? It's not showing up in a graph either.

Is the data set supposed to be filled in with nulls for every 60-seconds elapsed? AFAICT, I'm only sending every hourly value and only if it's not zero.

What I'm trying to achieve is figure out if I'm using Graphite in a useful way, and I'm hampered by not understanding where all these null values are coming from. I've tried looking for information but haven't found much; in part because googling on "graphite" and "null values" and "data" isn't really helping.

Thanks
Lisa

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Lisa
Solved:
Last query:
Last reply:
Revision history for this message
Lisa (lisa-9) said : 		#1

I think I'm slowly figuring this out, I see others have had the same confusion or the same desire to see what data I sent without seeing all the nulls.
(e.g. https://bugs.launchpad.net/graphite/+bug/961374 -- I second the request for a function that filters out null values)

It would be helpful to explain this up front; I couldn't tell whether there was an error in the way my code sent data to graphite (there was) because of this.

Revision history for this message
Lisa (lisa-9) said : 		#2

The more I figure this out on my own, the more I conclude that the documentation could have been a huge help here.

From reading the graphite documentation pretty carefully but not necessarily every page of it, I was left with the impression that something was wrong with my site or its data. I read https://graphite.readthedocs.org/en/1.0/url-api.html carefully, and looked at the data I'd sent to Graphite, and it didn't look at all similar -- I was seeing nulls every 60 seconds, with the occasional non-null values (e.g. every hour). Plus this made it hard to figure out how to graph the data in a way that looked decent. With the graphing not looking very sensible, and the data looking worse, I ended up spending a lot of time looking for problems earlier up the process, when in fact the major thing I needed to change was the mental model I'd been given through reading the introductory documentation.

Revision history for this message
Lisa (lisa-9) said : 		#3

Last comment , with a concrete suggestion: the URL-API docs (https://graphite.readthedocs.org/en/1.0/url-api.html) don't mention that the view defaults to some period around a day. Without knowing to expect that, it's hard for a beginner to know if they're looking at the right data, or not sending the right data in the first place.

Revision history for this message
Tim Zehta (timzehta) said : 		#4

Troubleshooting Graphite can be a bother. Determining where a problem lies requires verification of the data at all of the points of communication. The best tool I've found for this is ngrep:

sudo ngrep -lqd any ping tcp dst port 2003

The following one-line command finds ping metrics submitted over TCP to Graphite and highlights them using grep:

match='ping'; sudo ngrep -lqd any "${match}" tcp dst port 2003 | grep -F --color=tty "${match}"

Changing the protocol and port allows easy inspection of metrics sent to a statsd listener, from carbon-aggregator to carbon-cache, etc.

See the man page for more information: http://manpages.ubuntu.com/manpages/precise/en/man8/ngrep.8.html