Graphite

Creates too slow, but I/O util is below 20%

Asked by Rodion Vynnychenko

I'm trying to verify that graphite will handle 100k metrics per minute properly, so I've made a script posting 1666 metrics per second to a single carbon-cache instance on localhost.

The problem is that creates are really slow, I started running this script every second "find /mnt/whisper/ -type f | wc -l >> carbon-counts" and it turned out that only about 10 new metrics are created. At the same time "iostat -dxk 10" shows:

ubuntu@ip-10-64-153-96:~$ iostat -dxk 10
Linux 3.2.0-23-virtual (ip-10-64-153-96) 06/27/2012 _x86_64_ (1 CPU)

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.32 0.59 1.01 8.11 29.78 47.25 0.07 43.87 23.65 55.72 3.33 0.53
xvdb 0.00 123.70 0.10 136.45 0.34 1040.59 15.25 27.98 204.87 6.85 205.01 1.24 17.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.21 0.00 0.62 0.00 3.32 10.67 0.00 1.33 0.00 1.33 0.67 0.04
xvdb 0.00 301.04 0.00 61.89 0.00 1451.71 46.91 7.12 114.97 0.00 114.97 1.27 7.85

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvdap1 0.00 0.52 0.00 1.14 0.00 12.84 22.55 0.00 1.09 0.00 1.09 0.73 0.08
xvdb 0.00 274.22 0.00 31.78 0.00 1224.02 77.03 1.16 36.56 0.00 36.56 0.47 1.49

So there doesn't seem to be any substantial disk activity.

MAX_CACHE_SIZE is set to 1000000, MAX_CREATES_PER_MINUTE = inf. Top shows this:

ubuntu@ip-10-64-153-96:~$ top | grep carbon-cache
 1398 root 20 0 460m 378m 1968 S 38.4 10.1 40:52.62 carbon-cache.py
15838 root 20 0 8 4 0 R 3.8 0.0 0:00.02 carbon-cache.py
 1398 root 20 0 460m 378m 1968 S 58.8 10.1 40:54.39 carbon-cache.py
 1398 root 20 0 460m 378m 1968 S 64.7 10.1 40:56.34 carbon-cache.py
 1398 root 20 0 460m 378m 1968 S 55.7 10.1 40:58.02 carbon-cache.py

This is running on a ec2.medium instance with 1 CPU and 4GB memory.

How do I increase the rate at which creates happen?

Question information

Language:
English Edit question
Status:
Answered
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Michael Leinartas (mleinartas) said : 		#1

The creates are done by writing zeros to the new whisper files to fill them in an effort to ensure the data is stored on disk contiguously. If your tests are mainly concerned with handling the volume of incoming writes, you might want to change your storage-schemas to something very short - a day or two max. For the most accuracy, use the same precision values you plan to use in a production setup. The smaller retentions will allow the files to be created much faster.

Just a stab in the dark, but you could also be running into the I/O ceiling of the medium instance type as these files are being created. Amazon describes it as having a "moderate" I/O cap. I'd recommend starting with an m1.large for any graphite install. That may get you to 100k/min ok. I'm a bit skeptical that a medium will perform well enough for it.

Can you help with this problem?

Provide an answer of your own, or ask Rodion Vynnychenko for more information if necessary.

To post a message you must log in.