Grafana, Graphite and Netdata - Having constant gaps in my graphs

Asked by Dugi

I have setup resource monitoring for my servers using:

- Netdata - To fetch the servers' resource information and send it over to the main server where Graphite
- Graphite - Installed on a server that is there only to display all these collected metrics. So, collects metrics that come in from Netdata
- Grafana - Display metrics stored in Graphite in a pretty UI.

My graphs currently look like this:
https://i.stack.imgur.com/LInrO.png

My graphite retention policies look like this:

    [carbon]
    pattern = ^carbon\.
    retentions = 60:90d

    [netdata]
    pattern = ^netdata\.
    retentions = 60:182d

Netdata is supposed to send data to the Graphite server at least every 10 seconds. What is confusing me most is that when I tried this on my local virtual machines, I had it working perfectly and with no gaps. I did copy the same configurations from my local VMs, identical operating systems, identical services installed - but, different results.

The Graphite/Grafana server is on Ubuntu 14.04 and the server(s) with Netdata installed are on Ubuntu 16.04.

I have no clue how to proceed or where the problem lies. Any help would be highly appreciated.

Question information

Language:
English Edit question
Status:
Expired
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Denis Zhdanov (deniszhdanov) said :
#1

I would recommend to install some carbon monitoring dashboard (like https://grafana.com/dashboards/311 or https://grafana.com/dashboards/43) and compare graphs for laptop and VM. Maybe you're just overloading single carbon instance and you need to use relay and 4-6 carbon daemons in parallel.

Revision history for this message
Dugi (dugajean) said :
#2

@deniszhdanov: How would I do that? How can I relay 4-6 carbon daemons in parallel?

Revision history for this message
Denis Zhdanov (deniszhdanov) said :
#3

You need to add additonal sections for each carbon-cache in carbon.conf, setup relay there for load distribution and setup web part to properly ask all cache instances for data.
It's better to show that using example. Could you please post your carbon.conf and local_settings.py (without comments) here?
Or check e.g. "Multiple Carbon-Cache daemons" part in https://grey-boundary.io/the-architecture-of-clustering-graphite/

Revision history for this message
Launchpad Janitor (janitor) said :
#4

This question was expired because it remained in the 'Needs information' state without activity for the last 15 days.