Graphite

delays in graphing

Asked by FRLinux

Hello,

We have a few servers connected to netapps via NFS for whisper storage, right now, usually when we reach a consistent 40.000+ metrics per cache, it fails to graph for a few hours, and then eventually flushes onto disk.

We are using 2 instances per server (24 threads CPUs with 48GB of RAM), with the following config:

Do you see any way to make this faster? The system seems fairly IDLE so not a RAM/CPU issue in my view.

[cache]
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = 3000
MAX_CREATES_PER_MINUTE = inf

LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003

PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2004

CACHE_QUERY_INTERFACE = 0.0.0.0
CACHE_QUERY_PORT = 7002

LOG_UPDATES = False

WHISPER_AUTOFLUSH = False

Thanks for any answer you might have.

Cheers,
Steph

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
FRLinux
Solved:
Last query:
Last reply:
Revision history for this message
Michael Leinartas (mleinartas) said : 		#1

Are you using 0.9.9? There were a few fixes to the cache full behavior that I think will help you. In 0.9.9 there was a bug where if carbon-cache's cache were full, it would have to drain the entire cache to disk before it'd start receiving again. There was also a similar problems in the carbon client (used when sending from one carbon daemon to another).

I suggest you start with applying at least this patch:
http://bazaar.launchpad.net/~graphite-dev/graphite/main/revision/673

and you may want these two as well:
http://bazaar.launchpad.net/~graphite-dev/graphite/main/revision/671
http://bazaar.launchpad.net/~graphite-dev/graphite/main/revision/672

Outside of that, the main setting to play with is the MAX_UPDATES_PER_SECOND - decreasing the value will cause more writes to be bunched together and result in more sequential writes and increasing it will result in more random writes. You'll want to tweak it to what performs best on your storage.

Revision history for this message
FRLinux (frlinux-frlinux) said : 		#2

Thanks for your response!