Graphite

High CPU Usage for carbon-cache?

Asked by Matt

We are witnessing high CPU usage for carbon-cache on two of our systems, carbon-cache is maxing out one of the cpu cores.

Both machines are showing similar statistics.

MetricsRecieved is hovering around 21k
There's a very small number of creates (<10 day)
The cache.size is hovering around 2k
Cache.size is hovering around 2k
Memory usage is tiny, 20Mb or so
I/O utilisation is well under 20%, usually around 10%

Our config file is the default

The machines are Centos VMS, but the hardware they are running on (two different machines) have very little else on them and there is little contention.

The vms have 4 x 4 core processors (Intel(R) Xeon(R) CPU E7- 4830 @ 2.13GHz)
We haven't got to the point of looking at our disk set up yet, we're storing our wsp files in a /var directory which is mounted directly in / (ext4 defaults, 70% free)
The Vms have 4G memory

We've inherited this set up and are pretty new to graphite. My understanding is that carbon-cache is IO heavy rather than CPU heavy and this kind of load shouldn't hit the cpu this hard.

Is this CPU usage too high and if so what should we look at to reduce it?

Thanks,

Matt

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Matt
Solved:
Last query:
Last reply:
Revision history for this message
Wawrzyniec Niewodniczański (wawrzek) said : 		#1

What do you exactly mean by "We are witnessing high CPU usage for carbon-cache on two of our systems, carbon-cache is maxing out one of the cpu cores." What does CPU doing?

Revision history for this message
Jason Dixon (jason-dixongroup) said : 		#2

I've seen this when carbon-cache is blocking on writes. If you're using
relays look for the existence of carbon.relays.*.destinations.*.queuedUntilConnected
and carbon.relays.*.destinations.*.fullQueueDrops metrics.

Also, check your carbon.agents.*.cache.queues metrics.

Jason

On Fri, Jul 18, 2014 at 09:31:51PM -0000, Wawrzyniec Niewodnicza??ski wrote:
> Question #251723 on Graphite changed:
> https://answers.launchpad.net/graphite/+question/251723
>
> Status: Open => Needs information
>
> Wawrzyniec Niewodnicza??ski requested more information:
> What do you exactly mean by "We are witnessing high CPU usage for
> carbon-cache on two of our systems, carbon-cache is maxing out one of
> the cpu cores." What does CPU doing?
>
> --
> You received this question notification because you are a member of
> graphite-dev, which is an answer contact for Graphite.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~graphite-dev
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~graphite-dev
> More help : https://help.launchpad.net/ListHelp

--
Jason Dixon
http://obfuscurity.com/
https://twitter.com/obfuscurity

Revision history for this message
Matt (matt-slane) said : 		#3

We're not using carbon-relay here. the queue metrics are very low. These boxes are part of CI processes and don't have a lot of query traffic. I've just been told it's a problem with the host the vms are on. Initially we were told the host was fine.