federated storage and complex wildcard targets = suboptimal
I recently sliced up my graphite installation into two systems. The federation-fu is working. Nice work! However, many of my common graphs are quite complex, using wildcards and sumSeries() for example. They are now very, very slow when the data resides on remote storage. Here's one plot target from a "suck-o-meter" (ratio of connections active vs those storing data):
movingAverage(
This one line expands to 270 wsp files. In total, the whole graph needs 1,540 files. Each one of these matches results in a single request for pickle data: "GET /render/
This seems to be expected behavior, probably because the django app will do a find for who has the needed .wsp without regard for the relay-rules.conf. It is possible someone, even myself, may start slicing things up in even more complicated ways, duplicating updates to multiple systems, etc., so I think I get why it's doing this.
So, my question is: Am I doing something wrong here? (other than not sending my requests to the host that has them local) If this is expected behavior, perhaps we can find some optimization? For example, keep the target= in the /render/ pickle request as a wildcard. It should return all the same results as the /metrics/find/ request did. Possible issues here when there is duplication of data?
[cue joke about one big pickle in the pickle jar]
Kevin Blackham
Mozy
Question information
- Language:
- English Edit question
- Status:
- Solved
- For:
- Graphite Edit question
- Assignee:
- No assignee Edit question
- Solved by:
- chrismd
- Solved:
- Last query:
- Last reply: