How to build a scalable graphite architecture

Hi,

i am currently building a new performance monitoring solution
for our VMs. At the moment we have about 2000 VMs, but this will be
ramped up soon up to 50000 VMs. The performance collection will be done
with collectd.

My plan is to start with (A):

- one carbon-relay which delegates the performance data to
- an arbitrary amount of carbon-cache instances
- which all write the data into the same nfs share

When more and more VMs are build up i plan to (B):

- add more carbon-cache instances
- which will write the data into different nfs shares

My question:

When starting with (A) the carbon-relay will distribute the
data between all carbon-cache instances (RELAY_METHOD = consistent-hashing).

Now i start adding new carbon-cache instances whith a different
nfs share (B).

How Iwill the data be distributed? Only the performance data that was not
already be distributed to one of the old carbon-cache instances will be distributed
to the new carbon-cache instances, is that right?

So, just adding more carbon-cache instances with additional nfs share will not
immediately improve the performace, right?

Is this possible at all?

Thanks and greetings,
-timo