DESTINATIONS vs CARBONLINK_HOSTS in a cluster

Yeah, I wander too.

When webapp receives a render metric , it first tries to determine whether the appropriate .wsp file(s) can be found locally ( under the DATA_DIRS path from local_settings.py )
If found - data are read from the file.
If .wsp is not found locally, then webapp sends a request (iterates) to all other webapp nodes defined in CLUSTER_SERVERS, which in turn, looks up for data file in their local DATA_DIRS storage. Eventually, one of the nodes has it.
Then,
 webapp needs to merge received 'cold' file data with the 'hot' carbon-cache data (that are carbon-cache'ed but were not written to disk yet). For this reason, webapp uses consistent-hashing to select a cache instance (host:port) to send query to.
Cold and hot data are merged, converted (csv/json/png/) , and send back in response to inital request.

Both carbon-relay and webapp uses the same consistent-hashing get_node(metric_key) function to select the node to work with. Buth the hash_ring is generated on different sets of arguments.

For carbon-relay : hash_ring = ConsistentHashRing(DESTINATIONS)
For webapp : hash_ring = ConsistentHashRing(CARBONLINK_HOSTS)

I just made a quick test to confirm that these two rings may return different hosts:

DESTINATIONS = ["127.0.0.1:2013:a","127.0.0.1:2023:b","127.0.0.2:2013:a","127.0.0.2:2023:b"]
CARBONLINK_HOSTS = ["127.0.0.1:2013:a","127.0.0.1:2023:b"]

>>> hash_ring = ConsistentHashRing(DESTINATIONS)
>>> hash_ring2 = ConsistentHashRing(CARBONLINK_HOSTS)

, they may return the same instance for one key:

>>> hash_ring.get_node('system')
'127.0.0.1:2013:a'

>>> hash_ring2.get_node('system')
'127.0.0.1:2013:a'

, and completely different instance for another key:

>>> hash_ring.get_node('system.load')
'127.0.0.2:2023:b'

>>> hash_ring2.get_node('system.load')
'127.0.0.1:2023:b'

My opinion is that CARBONLINK_HOSTS should always match the DESTINATIONS, otherwise hash_ring.get_node(key) simply return wrong result as above. In this example, carbon-relay will writes system.load metric to 127.0.0.2:2023:b, webapp is able to get 'cold' data but fails to merge it with 'hot' data, because it looks them up at wrong carbon-cache host. It is not that bad eventually, but we loose rel-time statistics.

carbon.conf documentation puts a lot of confusion here saying that only local carbon-cache instances should be listed in CARBONLINK_HOSTS.

It is possible that I'm just getting it wrong, of course, but I'd be very grateful and buy a pint of beer to someone who would point out my mistakes here :)
Chris, we need you !

With regards,
Anatoliy Dobrosynets

Some more test results with different carbon-cache instance lists and orders.

https://gist.github.com/anatolijd/5576536

Would it be possible that you could share you're "hash_ring_selector_test.py" script? I would also like to do some tests

Hi,
scroll the gist page to the bottom, please. It should be there.

Hello Brian and Anatoliy:

I'm no expert in Graphite, so take my explanations with a grain of salt. I can be wrong.

Graphite-web of course uses ConsistentHashing.

> b.) Id' suspect we want all the destinations in CARBONLINK_HOSTS so
> we don't get different consistent-hashing results vs DESTINATIONS
> c.) Is it smart enough to simply limit to local instances when
> deciding what to ask?

Graphite-web uses ConsistentHashing only for local carbon-cache instances cached-hot data (CARBONLINK_HOSTS). Remote instances are queried through their graphite-web instances (CLUSTER_SERVERS), which in fact internally use ConsistentHashing to query theirs local carbon-cache instances.

* DESTINATIONS should include all the carbon-cache instances in the cluster, including local and remote.
* CARBONLINK_HOSTS should include all the local carbon-cache instances.
* CLUSTER_SERVERS should include the other graphite-web instances, but not itself.

Graphite-web searches for metrics in the following order:

1) Search local DATA_DIRS.
2) If not found, search remote CLUSTER_SERVERS (through graphite-web) without using ConsistentHashing, simple loop.
3) In both cases, merge the results with CARBONLINK_HOSTS (carbon-cache cached-hot data) queried using ConsistentHashing.

Anatoliy: If you include all the instances in CARBONLINK_HOSTS, not only local, remote carbon-caches will be queried twice, once by the local graphite-web and one by the remote graphite-web. In your second example, graphite-web will not need to query CARBONLINK_HOSTS, because CLUSTER_SERVERS already returned the cached -hot data. So no matter if ConsistentHashing returns different results when the metric are in a remote instance, graphite-web already returned them. Am I wrong in this?

To clarify things:

DESTINATIONS (used by carbon-relay & carbon-aggregator):
* Lists all the carbon-cache instances in the cluster.
* Uses PICKLE_RECEIVER_PORT.
* Format: (IP|FQDN):PICKLE_RECEIVER_PORT:instance (for example "1.2.3.4:2004:a")

CLUSTER_SERVERS (used by graphite-web):
* Lists all the graphite-web instances except itself.
* Uses graphite-web HTTP listen port.
* Format: (IP|FQDN):listen_port (for example "1.2.3.4:80")

CARBONLINK_HOSTS (used by graphite-web):
* Lists all the local carbon-cache instances.
* Uses CACHE_QUERY_PORT.
* Format: (IP|FQDN):CACHE_QUERY_PORT:instance (for example "1.2.3.4:7002:a").

Note: The used hostnames/IPs should match in DESTINATIONS and CARBONLINK_HOSTS due to the ConsistentHashing (I mean avoid using localhost or private IPs for referring to local instances). But the contents of their arrays may differ. They will match when we have only local carbon-caches, nothing more, only one server.

Regards.

Yes, graphite-web fetchData algorythm you described is correct.

My only concern was ConsistentHashRing.
Is it really that smart to return the same value for a given key, despite the fact it's hash_ring has been generated on differnt base ?

I couldn't believe that if
   DESTINATIONS = ['10.4.0.1:a', '10.4.0.1:b', '10.4.0.2:c', '10.4.0.2:d']
   carbon-relay routes metric to any carbon-cache at host-1,
   and host-1 webapp has CARBONLINK_HOSTS = ['10.4.0.1:a', '10.4.0.1:b']
then this is always True for any metric:
   ConsistentHashRing(DESTINATIONS).get_host(metric) == ConsistentHashRing(CARBONLINK_HOSTS).get_host(metric)

Today I made more tests and got really impressed - https://gist.github.com/anatolijd/5611927 .

I don't understand this magic but I've got the evidence of the faith now :)

So if I'm interpreting the test results correctly, it's preferable to use only local cache instances in CARBONLINK_HOSTS and still expect carbonlink queries to be directed to the correct cache instance via consistent-hashing ring _even if_ it's based on a different list of cache destnations between the sender (carbon-relay containing all cluster cache instances) and consumer (graphite-web containing only local node cache instances)?

@anatolijd

That's just how the Consistent Hashing works. If you remove a node from the array, only the keys in this node will need to be remapped. The others stay in the same place.

In this case, with graphite, you remove "localhost" instances from the ring, which you've checked before (with CARBONLINK_HOSTS). So the results are the same.

http://en.wikipedia.org/wiki/Consistent_hashing
http://www.paperplanes.de/2011/12/9/the-magic-of-consistent-hashing.html

@bluke you are right. You need to put local instances in CARBONLINK_HOSTS.

carbon-relay & carbon-aggregator => DESTINATIONS
graphite-web => CARBONLINK_HOSTS (local instances) + CLUSTER_SERVERS (other graphite-webs)

Hashing results are the same.

Hello, Xavier

>3) In both cases, merge the results with CARBONLINK_HOSTS (carbon-cache cached-hot data) queried using ConsistentHashing.

Looks like merge only take place when data fetched locally from DATA_DIRS, not through remote webapp (at least this is what I can see for version 0.9.12) - mergeResults called only if dbFile.isLocal() :

# Data retrieval API
def fetchData(requestContext, pathExpr):
  seriesList = []
  startTime = requestContext['startTime']
  endTime = requestContext['endTime']

  if requestContext['localOnly']:
    store = LOCAL_STORE
  else:
    store = STORE

  for dbFile in store.find(pathExpr):
    log.metric_access(dbFile.metric_path)
    dbResults = dbFile.fetch( timestamp(startTime), timestamp(endTime) )
    results = dbResults

    if dbFile.isLocal():
      try:
        cachedResults = CarbonLink.query(dbFile.real_metric)
        results = mergeResults(dbResults, cachedResults)
      except:
        log.exception()

    if not results:
      continue

    (timeInfo,values) = results
    (start,end,step) = timeInfo
    series = TimeSeries(dbFile.metric_path, start, end, step, values)
    series.pathExpression = pathExpr #hack to pass expressions through to render functions
    seriesList.append(series)

  return seriesList