Graphite

Collectd- Carbon-Relay issue

Asked by Johnu George

Hi All,

I am using collectd to collect system metrics which is sent to carbon cache. Now, I need to use skyline (https://github.com/etsy/skyline/wiki/Getting-Data-Into-Skyline) where I have to use carbon relay to forward pickles to skyline listener. However, It is not able to get data from carbon. I couldn't understand the issue. Collectd, carbon, graphite and skyline host is same. (172.29.86.27)

carbon.conf

[cache]

STORAGE_DIR = /var/lib/graphite/
CONF_DIR = /etc/carbon/
LOG_DIR = /var/log/carbon/
PID_DIR = /var/run/

LOCAL_DATA_DIR = /var/lib/graphite/whisper/
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2013

ENABLE_UDP_LISTENER = True
UDP_RECEIVER_INTERFACE = 0.0.0.0
UDP_RECEIVER_PORT = 2013

PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2014

[relay]
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2004
ENABLE_UDP_LISTENER = True
UDP_RECEIVER_INTERFACE = 0.0.0.0
UDP_RECEIVER_PORT = 2003

RELAY_METHOD = rules
USE_FLOW_CONTROL = False

DESTINATIONS = 172.29.86.27:2024, 127.0.0.1:2014

relay-rules.conf

[all]
pattern = .*
destinations = 172.29.86.27:2024, 127.0.0.1:2014

[default]
default = true
destinations = 172.29.86.27:2024, 127.0.0.1:2014

collectd.conf

<Plugin write_graphite>
        <Node "graphing">
                Host "172.29.86.27"
                Port "2003"
                Protocol "udp"
                LogSendErrors true
                Prefix "collectd"
                Postfix "collectd"
                StoreRates true
                AlwaysAppendDS false
                EscapeCharacter "_"
        </Node>
</Plugin>

In my default collectd.conf, I had graphite port configured as 2013 (carbon-cache udp port). Later, to make carbon-relay as primary listener for skyline, I changed port( in collectd.conf) to 2003. But, I still couldn't get anything forwarded to skyline. However in both cases, I can see that whisper files are updated in /var/lib/graphite. (only *.wsp files)

when I am issuing a tcpdump -i lo port 2003, i can see udp packets. so, collectd is working fine but, carbon relay has some problems.

Additional observations

I have seen new whisper files under carbon directory. This might help in debugging.

/var/lib/graphite/whisper/carbon/relays/bd-1-2-a/destinations/127_0_0_1\:2014\:None/
     attemptedRelays.wsp
    queuedUntilReady.wsp
     sent.wsp

/var/lib/graphite/whisper/carbon/relays/bd-1-2-a/destinations/172_29_86_27\:2024\:None/
    attemptedRelays.wsp
    fullQueueDrops.wsp

Am I having the right configuration?

Question information

Language:
English Edit question
Status:
Expired
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Launchpad Janitor (janitor) said : 		#1

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Revision history for this message
Denis Zhdanov (deniszhdanov) said : 		#2

Please try
RELAY_METHOD = consistent-hash
REPLICATION_FACTOR = 2

It will copy your metrics for two destinations properly.

Revision history for this message
Johnu George (johnugeo) said : 		#3

Thanks. I will try out. Can you tell me if this is the same case with carbon-aggregator? If I have multiple destinations in Carbon-Aggregator and if I need to copy all metrics to multiple destinations properly, what should I do?

Revision history for this message
Denis Zhdanov (deniszhdanov) said : 		#4

Yes, if I understand your question correctly... Main idea that rules.conf with multiply destinations with REPLICATION_FACTOR=1 is not working. Not sure if just REPLICATION_FACTOR=2 helps, w/o consistent hashing.

Revision history for this message
Johnu George (johnugeo) said : 		#5

Hi Denis,
                I am confused.
1.I read that consistent hashing is way to balance the metrics. Is it right?. I felt, relay is the right method if I need to copy all metrics to multiple servers. Still not clear wht relay-method= rules and REPLICATION_FACTOR=1 is not working. Do you suspect anything wrong in my configuration?.

2.Are you saying that relay-method= consistent-hashing and REPLICATION_FACTOR=2 will work?

3. I have a separate setup with carbon-aggregator as my primary listener. How should I configure such that all metrics are copied to multiple desinations like in previous case?. I could not see any "relay-method" setting in carbon-aggregator section. I could find only REPLICATION_FACTOR setting. I am seeing that all metrics are not getting to both destinations. Some metrics are sent to 1 machine while others are sent to the 2nd machine.

Revision history for this message
Denis Zhdanov (deniszhdanov) said : 		#6

Hi Johnu,

1. Yes, consistent hashing is way to balance the metrics. Is copying metrics to two destinations counting as balancing? I think so.
AFAIK relay method "rules" was designed to route different metrics to different destinations (using regex match by metric name) and with REPLICATION_FACTOR=1 it will put every metric to first destination and stops after that. If you need to copy metrics to two destinations you need to set up REPLICATION_FACTOR=2, maybe even relay_method="rules" will work in this case, but I'm not sure.

2. Yes. Maybe even relay_method="rules" will work with REPLICATION_FACTOR=2, please check.

3. Yes, that's whole point - no relay rules supported in aggregator. I think
REPLICATION_FACTOR=2
DESTINATIONS = 172.29.86.27:2024, 127.0.0.1:2014
in [aggregator] section will help.

Revision history for this message
Denis Zhdanov (deniszhdanov) said : 		#7

PS: You can also check https://github.com/grobian/carbon-c-relay - it's Graphite relay rewritten in C, it supports aggregations too.

Revision history for this message
Launchpad Janitor (janitor) said : 		#8

This question was expired because it remained in the 'Open' state without activity for the last 15 days.