Graphite

Could someone clarify this federated storage question

Asked by densone

Say I write some graphite data using netcat:

echo "com.sean.test2 1400 $(date +%s)" | nc graphite1 2003

Then I write some data to another carbon cache:

echo "com.sean.test2 1450 $(date +%s)" | nc graphite2 2003

The federated storage backend does not unify this data. It keeps the graphs separate. Is this the expected result? When I read about federated storage, it sounded like the data would be merged together when rendered.

Thanks,

Sean

Question information

Language:
English Edit question
Status:
Answered
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
chrismd (chrismd) said : 		#1

Currently federated storage does not mix datapoints from two servers that have the same metric, though that will be coming soon. What it does right now is allow you to have metric 'foo' on one server and metric 'bar' on another and show you both of them whenever you browse the hierarchy on either server, giving the appearance that both files are on each machine.

Revision history for this message
Sean Byron (sean-seanbyron) said : 		#2

Chris,

Is this feature (mixing datapoints from two servers that have the same metric) available yet in 0.96 or Trunk? I think it would be very useful for anyone who isn't concerned with wasting disk space and would like to maintain a single default entry in relay-rules.conf. I ask because I'm planning on setting up a fully-HA multi-shard graphite instance, and don't want to have to configure multiple relay rules.

Thanks,
Sean

Revision history for this message
Sean Byron (sean-seanbyron) said : 		#3

Also, is there any documentation on relay-rules.conf? I'd like to experiment with segregating metrics based on name to different HA sets of carbon-cache backends, but I'm not sure what the format of relay-rules.conf should be.

Thanks (and sorry for the double-post),
Sean

Revision history for this message
chrismd (chrismd) said : 		#4

It is almost done, I was supposed to finish that project 2 months ago but it got postponed do to other priorities at work. I am just now getting back on the track to implement it so, soon. It will not make the next release (because I'm going to get that out before the new year) but I fully expect it to be in the release after that which I'm really going to try and make sooner than a whole year away :). Either way it will be in trunk in the next month or two.

As for the relay-rules, yes there isn't really any documentation but fortunately it is very simple. Here is an example file:

[app-metrics]
pattern = apps\..*
servers = server1, server2

[db-metrics]
pattern = db\..*
servers = server2

[everything-else]
default = true
servers = server1

With this configuration any apps.* metrics will be sent to carbon-cache on both server1 and server2, the db.* metrics will be sent only to server2, and all other metrics will be sent only to server1. You must have exactly one section with 'default = true' and any number of other sections. The non-default sections get evaluated in order and the first that matches a given metric is the one that applies to it.

Revision history for this message
chrismd (chrismd) said : 		#5

Just to clarifyt the 'pattern' values are regular expressions, hence the \..*

Revision history for this message
Sean Byron (sean-seanbyron) said : 		#6

Thanks very much Chris! Great product, I'm enjoying using it and hope to contribute something back to it some day.

Can you help with this problem?

Provide an answer of your own, or ask densone for more information if necessary.

To post a message you must log in.