Graphite

Size of carbon-cache process only grows, never shrinks

Asked by spliff

We have a graphite system that is quite busy, 42k metrics updated every 30 seconds. We plan on adding about 250k metrics although they will not all be updated at 30 seconds. The system is fast enough to handle it, dual Xeon 5500, Raid5 15k drives etc..

When we start our metric loader process intraday it attempts to load the metrics from the beginning of the day as fast as it can. Carbon-cache grows in size as it is busy creating the whisper files, processing incoming data, etc. When carbon-cache catches up, the resident memory size of the process stays at about 1 gig, it never shrinks.

Is this the intended behavior of a very busy carbon-cache system? Should we expect the memory footprint to only grow and never shrink?

Question information

Language:
English Edit question
Status:
Answered
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Launchpad Janitor (janitor) said : 		#1

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Revision history for this message
chrismd (chrismd) said : 		#2

In my experience, most long-running python applications tend to grow larger over time at least to a certain point. Python is fairly liberal about memory usage, particularly because allocating objects is expensive. In many cases python opts to keep unused objects in memory for later reuse rather than to free the memory back to the OS. For more insight, take a peek at this article http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-a-large-object.htm

Carbon in particular deals with a *lot* of number objects, and Python never actually frees numbers. I've never run out of memory because of this myself. I have some very long running carbon processes that are at about 1G rss, that is a lot of memory but they don't seem to ever need more than that (at least on my system, YMMV).

Revision history for this message
chrismd (chrismd) said : 		#3

Sorry for the delay in getting back to you, I took a couple weeks break from Graphite

Can you help with this problem?

Provide an answer of your own, or ask spliff for more information if necessary.

To post a message you must log in.