RSYNC whisper file to new Graphite cluster

Hi,

Try https://github.com/graphite-project/carbonate - a collection of tools for syncing metrics around Graphite clusters. You don't need to stop the existing cluster while syncing, though it may impact disk IO.

Brian

Carbonate is great, but it's primarily used to heal clusters or redistribute when scaling a cluster up (e.g. adding a node). If you're doing a simple 1:1 copy from an old server to a new server, you're fine to just rsync your files.

Jason

Brian, Jason, thank you for your prompt replies!

Brian, I did tried carbon-sync, here's how:

1) I configured my /etc/carbonate.conf to:

[main]
DESTINATIONS = Graphite_node1_private_ip:2004:aggregator1, Graphite_node2_private_ip:2004:aggregator2
REPLICATION_FACTOR = 2

2) Ran command carbon-sync -s old_graphite

and this command just hung . Our whisper data are at the default locations on both old and the new cluster. I am not sure if carbonate needs to be installed on the old cluster as well. Bottom line, something is failing - I just can't figure out how to trouble shoot this.

Regarding RSYNC. It does transfer data from my old to my new graphite servers; My question though - the old server is constantly being updated, which means that RSYNC is never going to bring the source and the destination to the identical state. Not quite sure how to deal with this.

You could theoretically point the stream to a relay and try to have them buffer in memory "artificially", that is to say, block the relay's ability to flush to the cache(s) on the new server until they're fully synced. This is not something I've attempted myself, but is at least theoretically possible using a network filter.

Personally, I would just configure graphite-web to read from both servers until some point at where the data on the old server is no longer operationally relevant, and then decommission the old server.

TL;DR migrating from one Graphite server to another will typically involve some small amount of data gap.

@Jason that's an interesting idea.
Using https://github.com/graphite-ng/carbon-relay-ng you could create a route with a destination tcp address that doesn't work, do your sync, and then update your route in the relay, it will connect and start sending all those points that were missed, so you'll have no gaps.