Graphite

Using multiple Caron relay consumers on a single AMQP exchange

Asked by Oded

I'm trying to scale my carbon AMQP ingestion layer by having multiple carbon realys consume metrics from a single AMQP exchange, did anyone got such a configuration working?

Can this be done with "topic" type exchange? RabbitMQ docs suggests that a single message can be routed to multiple queue/consumers(?) which is not desirable in my config.
This is also what I see in the RabbitMQ GUI when sending a single metric.

I tried to "patch" Graphite(lib/carbon/amqp_listener.py ) to use "direct" type exchange but that didn't work at all - the message was not routed to any of the queues.

Question information

Language:
English Edit question
Status:
Expired
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Oded (oded-b) said : 		#1

So, further reading on AMQP shows that I can use a topic exchange, but need the Carbon Relays consume on a single queue,
seems that it could be possible be making small changes to amqp_listener.py, if only I could find txAMQP documentation :(

Revision history for this message
Oded (oded-b) said : 		#2

*seems* to work with this change:

[root@graphite005 graphite]# diff /opt/graphite/lib/carbon/amqp_listener.py /opt/graphite/lib/carbon/amqp_listener.py.orig
96c96
< wfd = waitForDeferred(chan.queue_declare(exclusive=False, queue="cr_queue"))
---
> wfd = waitForDeferred(chan.queue_declare(exclusive=True))
 [root@graphite005 graphite]#

Metrics are distributed between Carbon Relays in a round robin fashion.

Revision history for this message
Launchpad Janitor (janitor) said : 		#3

This question was expired because it remained in the 'Open' state without activity for the last 15 days.