Graphite

Lots of droppedCreates with relatively few metrics

Asked by Gabe Mahoney on 2018-04-29

We are using the graphiteapp/graphite-statsd docker container on an amazon i3.large instance. We are trying to send about 10k metrics per minute from a handful of hosts. We noticed we were missing a lot of data points so I started investigating performance. I wasn't able to find documentation on carbon's stats but I am seeing about 5k "droppedCreates" per minute and "committedPoints" of about 150 per minute. Looking at the code it seems that we are hitting some performance issues, but its not clear to me what the bottleneck is. I tried setting MAX_UPDATES_PER_SECOND = inf but that didn't change the behavior (still getting lots of droppedCreates).

carbon.agents.*.cache.size never goes above 100.
carbon.agents.*.cache.overflow stays at 0.
carbon.agents.*.creates are at about 50 per minute.

I'm seeing barely any CPU or mem usage by the agents or the aggregator.

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Gabe Mahoney
Solved:
Last query:
Last reply:
Gabe Mahoney (gmahoney) said : #1

Nevermind, found the problem. We were creating a lot of metrics per minute and there is a default MAX_CREATES_PER_MINUTE = 50.