RSYNC whisper file to new Graphite cluster

Asked by victoria kozel on 2016-04-26

Hello,
what is the best way to transfer data (whisper files) to the new Graphite cluster? Will rsync work? Should original cluster be stopped while rsync is happening?
Any ideas are appreciated!!
Vicki

Question information

Language:
English Edit question
Status:
Answered
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
2016-04-26
Last reply:
2016-04-26
Brian Hatfield (bmhatfield) said : #1

Hi,

Try https://github.com/graphite-project/carbonate - a collection of tools for syncing metrics around Graphite clusters. You don't need to stop the existing cluster while syncing, though it may impact disk IO.

Brian

Jason Dixon (jason-dixongroup) said : #2

Carbonate is great, but it's primarily used to heal clusters or redistribute when scaling a cluster up (e.g. adding a node). If you're doing a simple 1:1 copy from an old server to a new server, you're fine to just rsync your files.

Jason

victoria kozel (vkozel) said : #3

Brian, Jason, thank you for your prompt replies!

Brian, I did tried carbon-sync, here's how:

1) I configured my /etc/carbonate.conf to:

[main]
DESTINATIONS = Graphite_node1_private_ip:2004:aggregator1, Graphite_node2_private_ip:2004:aggregator2
REPLICATION_FACTOR = 2

2) Ran command carbon-sync -s old_graphite

and this command just hung . Our whisper data are at the default locations on both old and the new cluster. I am not sure if carbonate needs to be installed on the old cluster as well. Bottom line, something is failing - I just can't figure out how to trouble shoot this.

Regarding RSYNC. It does transfer data from my old to my new graphite servers; My question though - the old server is constantly being updated, which means that RSYNC is never going to bring the source and the destination to the identical state. Not quite sure how to deal with this.

Jason Dixon (jason-dixongroup) said : #4

You could theoretically point the stream to a relay and try to have them buffer in memory "artificially", that is to say, block the relay's ability to flush to the cache(s) on the new server until they're fully synced. This is not something I've attempted myself, but is at least theoretically possible using a network filter.

Personally, I would just configure graphite-web to read from both servers until some point at where the data on the old server is no longer operationally relevant, and then decommission the old server.

TL;DR migrating from one Graphite server to another will typically involve some small amount of data gap.

Dieter P (dieter-plaetinck) said : #5

@Jason that's an interesting idea.
Using https://github.com/graphite-ng/carbon-relay-ng you could create a route with a destination tcp address that doesn't work, do your sync, and then update your route in the relay, it will connect and start sending all those points that were missed, so you'll have no gaps.

Can you help with this problem?

Provide an answer of your own, or ask victoria kozel for more information if necessary.

To post a message you must log in.