Graphite

httpd process cpu usage high

Asked by Yousef F on 2018-01-27

i am using graphite as a data source in grafana, I have no issues with the server performance on all the different dashboards and queries but we are now testing a new map panel that can have the metrics plotted on it.

the panel have 5K+ metrics plotted and when it refreshes for the new metrics from graphite, the httpd process on graphite showing that it's demanding 130% %CPU via the top command ( running graphite on redhat 7.4 by the way)

what's interesting is that only one of the cores is being utilized at %60 percent at my server when the refresh happens.

so my question is there anyway to make that particulate query get spanned on multiple process/threads so it can utilize more of the 16 cores on my server ? that may also enhance the time the query take to refresh on grafana which is ~15 seconds.

any other suggestion is highly appreciated to go around this

thank you for the help

Question information

Language:
English Edit question
Status:
Answered
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Denis Zhdanov (deniszhdanov) said : 		#1

Hell,
It depends on your setup. If you're using apache + mod_wsgi then you need to increase number of processes and threads in line similar to https://github.com/graphite-project/graphite-web/blob/master/examples/example-graphite-vhost.conf#L31 (current number of threads in example is 5).
Then you need to restart Apache.

Revision history for this message
Yousef F (yousef.fattal) said : 		#2

Thank you for the suggestion, I didn't get a chance to try this out on my prod server.

is there any recommendations behind the numbers ? I have the default installation with the configuration looking like the link you sent. In my case i have 16 cpu cores server. The server has only graphite installed on it so does it make since to raise both numbers to 16 matching my cores ? or even more if my server can handle it?

Thank you for the help

Revision history for this message
Denis Zhdanov (deniszhdanov) said : 		#3

Rule of thumb here is workers=2xCPU, but please take note that if you're have hyperthreading enable real cores can be 8 and not 16.
So, unfortunately YMMV - it's need to be tested on your prticular hardware. Start with 16, and increase if needed.

Can you help with this problem?

Provide an answer of your own, or ask Yousef F for more information if necessary.

To post a message you must log in.