Graphite

Graphite Backend connections and volume

Asked by bhardy

I have two clustered centos5.5 x86_64 machines (both 8cpu 2323mhz, 31gb mem, with SAN storage) and am using the most recent 0.9.6 release. My curiousity is if these servers would be able to handle multiple high volume connections (totaling possibly 150,000 metrics/minute)?? Or if a pypeD or message bus solution would better serve my needs.

I have read other entries about volume and one mentions that the backend was not designed for such large volume connections while the other states that the carbon-cache would grow in size to accomodate processing. Was the statement about the limitations of the backend and volume specific to eariler versions of graphite, pre 0.9.6?

References:
https://answers.launchpad.net/graphite/+question/37881
In the answer below you stated that the graphite carbon backend was not designed to do itself is multiplex a large number of high volume client connections.

https://answers.launchpad.net/graphite/+question/105233
I've never run out of memory because of this myself. I have some very long running carbon processes that are at about 1G rss, that is a lot of memory but they don't seem to ever need more than that

Thanks.

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
chrismd
Solved:
Last query:
Last reply:
Revision history for this message
Best chrismd (chrismd) said : 		#1

Hi Bryant, actually I'd recommend using a checkout of trunk instead of 0.9.6. I really need to cut a new release as there have been a lot of fixes since 0.9.6. As for PypeD, the only purpose that ever served was offloading the work of multiplexing lots of client connections (particularly from the ITA farms). It was better suited to that task than carbon because it was written with the Twisted framework, but a while ago carbon was rewritten to use Twisted as well so pyped really isn't necessary anymore.

Your machine specs sound really good, and using a SAN and Linux with separate mounts is excellent. Have you run into any particular performance bottlenecks in this new setup? I've got a 2 node graphite cluster where each machine has a local RAID array instead of a SAN mount and it handles ~250k datapoints per minute very well so you should be able to handle your load easily.

One thing thing that is important to know is that having lots of clients sending data directly to carbon using the plaintext ("metric-name value timestamp\n") protocol isn't particularly scalable. The most scalable approach is to have fewer clients send the datapoints in large batches via the pickle protocol (connect to port 2004 and send length-prefixed pickled lists of (metric, value, timestamp) tuples). If that doesn't make sense let me know and I can explain more thoroughly. That really only matters if carbon is using lots of CPU, as the pickle method can be much less CPU intensive if done right. Anyways, let me know what if any bottlenecks you've run into in load testing your system.

Revision history for this message
bhardy (bhardy) said : 		#2

So far there are no bottlenecks seen. I will try to download the lastest trunk. Currently everyone sends data to pypeD and pypeD forwards it to graphite. Some use the milton pylib, streambase uses a java class, and ITA uses typewriter monkey. If you could, I would like you to explain how to send data using the pickle port. It sounds as if plain text would be easier but we were concerned that direct connections using plain text would cause the system to run out of file descriptors.

And thanks very much. This is exactly the information I needed.

Revision history for this message
bhardy (bhardy) said : 		#3

Last question and I should be able to mark this as solved after your replies.

I am using the 0.9.6 release from the launchpad overview page. It looks to be from 2/26/2010 and is to be working fine. How would I download/checkout the latest trunk. I've search thru launchpad but cannot find this info. Thanks.

Revision history for this message
bhardy (bhardy) said : 		#4

Ignore last post. I did not have bzr installed. I was able to retrieve the latest branch.

So when you have an opportunity if you could speak to the two items below and then I can mark this as solved. Thanks.

1) how to send data using the pickle port
2) if direct connections would cause any file descriptor issues (specifically for ITA farms)

Revision history for this message
bhardy (bhardy) said : 		#5

Thanks chrismd, that solved my question.

Revision history for this message
Bill Wheatley (bwheatley) said : 		#6

Yes how do you send in data via the pickle format?

Revision history for this message
Kevin Blackham (thekev.) said : 		#7

Use python?

Revision history for this message
Bill Wheatley (bwheatley) said : 		#8

a tad too vague perhaps :P

Revision history for this message
Kevin Blackham (thekev.) said : 		#9

Yes. Vague. I don't know the answer, except that pickle is like perl storable for python. I don't know the data structure being serialized, but the source has it. :)

I'll go away now and let someone with clue answer.

Revision history for this message
chrismd (chrismd) said : 		#10

Sorry, I promise I will document this soon. Here's the 2-minute version:

import socket
import struct
import cPickle

sock = socket.socket()
sock.connect( ('graphite-server', 2004) ) # pickle protocol is done on port 2004

data = gather_me_some_data()
# where data = [ (metic, datapoints), ... ]
# and datapoints = [ (timestamp, value), ... ]

serialized_data = cPickle.dumps(data, protocol=-1)
length_prefix = struct.pack("!L", len(serialized_data))
message = length_prefix + serialized_data
sock.sendall(message)

A brief note on efficiency. In general the larger each message is (the more metrics/datapoints in each call to cPickle.dumps) the less deserialization overhead is incurred by carbon, but messages cannot exceed 1 meg (beyond that weird buffering issues hurt performance), so I usually send about 500 datapoints per message.