Graphite

ignore (don't graph) series without current data

Asked by Ross M.

using and loving graphite for monitoring a set of systems in AWS. i have a lot of data going in to it and i'm really happy with everything with one exception.

at any given time i only have a handful of hosts, but they are periodically replaced as part of rolling upgrades. the issue i'm having when i graph host-specific data (cpu from diamond for instance) using wildcards (e.g. servers.web-*.cpu.total.idle) the legend lists every host we've ever had. they change frequently enough that i don't want to have to edit to graphs to add/remove the hosts each time. as is i can tell if there's a problem with a host(s,) but it's nearly impossible to tell which one.

is there a way to only graph/use datasets that have data during the current time interval?

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Michael Leinartas
Solved:
Last query:
Last reply:
Revision history for this message
Best Michael Leinartas (mleinartas) said : 		#1

No, unfortunately there's not though it would be possible to write a function to do this. It'd look somewhat like currentAbove() - take a wildcard list and filter out anything with no points.

There are a couple of things you might think about doing:
  Use carbon-aggregator to create aggregated metrics - These metrics would always reflect the running servers at any given time and would provide the 'clean' view.
  Combined with above, regular purging of old metrics - in this case you'd have detailed per-server metrics for your currently running cluster, and an aggregated historical view for viewing long-term trends.
  Instead of purging old files, move old files (keeping the subtree intact) to an 'archive' directory. You can even gzip them to save on space (the webapp can read gzip'd whisper files directly).

Revision history for this message
Ross M. (rwmcfa1) said : 		#2

On Sun, Jun 17, 2012 at 11:31 PM, Michael Leinartas
<email address hidden> wrote:

> Michael Leinartas proposed the following answer:
> No, unfortunately there's not though it would be possible to write a
> function to do this. It'd look somewhat like currentAbove() - take a
> wildcard list and filter out anything with no points.

thanks. using currentAbove as an example i created currentlyActive:
    ~rwmcfa1/graphite/currentlyActive
    http://bazaar.launchpad.net/~rwmcfa1/graphite/currentlyActive/revision/834

works well for me right now. i'd imagine over time we may end up with
enough series that performance will be a problem. if/when that happens
i'll look in to your other suggestions.

thanks,
--
-rm

Revision history for this message
Ross M. (rwmcfa1) said : 		#3

Thanks Michael Leinartas, that solved my question.