The need to rebalance data after adding cache nodes

Asked by Kevin on 2016-10-25

Hi
Is it necessary to re-balance metrics after adding a new cache node alongside already existing single cache node, when using 0.9.15 for all components?

We plan to migrate from a single carbon relay/cache instance to a 3 node cluster, that would include a single relay node, which, in turn, would distribute metrics based on consistent hashing among the 2 cache nodes. The central relay node would also act as the main graphite-web instance.

When testing this migration from a single-node setup to 3 node cluster, the central graphite-web instance shows uninterrupted graphs, as if data was fetched, added and displayed from all nodes, not just the first found, as initially expected. The whisper files in question do indeed stop updating on the old node, and start on the new node after adding the new node.

There is also an issue of carbonate and buckytools both not agreeing with carbon-relay on which metrics do and do not belong to each of the cache nodes, but maybe we do not need to bother with it, if it is not actually necessary..

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Denis Zhdanov
Solved:
2016-10-25
Last query:
2016-10-25
Last reply:
2016-10-25
Best Denis Zhdanov (deniszhdanov) said : #1

Yes, it works, because 0.9.15 has REMOTE_STORE_MERGE_RESULTS = True by default - https://github.com/graphite-project/graphite-web/blob/0.9.x/webapp/graphite/settings.py#L71
I didn't test that on really massive installations, or when you adding nodes multiple times - but theoretically, it should work even if data is split between nodes.

Kevin (kevkev) said : #2

Thanks Denis Zhdanov, that solved my question.