Graphite

carbon-relay full HA setup

Asked by bhardy

I am hoping to setup full HA on two servers but am unsure what should be in the relay-rules.conf file on each server. The federated-storage works fine when I traverse the tree in graphite but the backend data is NOT being duplicated to each server.

Both are running an instance of carbon-cache and the webApp
Each has individual SAN storage
Each has the CLUSTER_SERVERS line in local_setting.py setup to point to itself and theotherguy
One mySql database is used for both servers
The default ports were changed in carbon.conf

We are hoping to have carbon-relay running on each server to send data to the other server but this is not working. What am I missing or what should be turned on/off?

relay-rules.conf
# You must have exactly one section with 'default = true'
[default]
default = true
servers = servername01.my.company, servername02.my.company

I have read the previously posted documentation but still cannot get the data duplicated.
https://answers.launchpad.net/graphite/+question/107945

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
bhardy
Solved:
Last query:
Last reply:
Revision history for this message
chrismd (chrismd) said : 		#1

Sure, if you want all data duplicated to both machines then your relay-rules.conf should look like this:

[default]
default = true
servers = graphite01, graphite02

But then you also need to edit the [relay] section at the bottom of carbon.conf, which should look something like this:

[relay]
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2013

PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2014

CACHE_SERVERS = graphite01, graphite02
MAX_QUEUE_SIZE = 10000

Notice that I changed CACHE_SERVERS and I changed the ports to 2013 and 2014 so as not to conflict with the cache listening on 2003/2004.

Finally you should just make sure that your VIP is setup to send connections to the relay ports, not the cache ports. You should be able to see in the relay's logs that it is receiving and relaying data.

Revision history for this message
bhardy (bhardy) said : 		#2

Your comments in harmony with the bug report given at the link below allowed me to resolve this issue. Thanks alot!

Can't relay to non-default ports on destination server
https://bugs.launchpad.net/graphite/+bug/649118