OpenStack Object Storage (swift)

Dealing with "too much" async pending

Asked by David Kranz

I have had a running cluster for a while and a new user is bashing the system resulting in the number of async pending going >10. I had set notification warnings for that threshold. My question is what I do about that. Is there some configuration parameter that needs to be adjusted, or is this just OK? If it is just OK, what is a reasonable threshold to warn on this value?

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Solved by:
David Kranz
Solved:
Last query:
Last reply:
Revision history for this message
John Dickinson (notmyname) said : 		#1

"Too much" async pending is definitely something to monitor in your system. I don't know the size of your cluster, but only 10 is incredibly small. Once whatever issue is causing it (either hardware failure or simply load on your container servers), the system should handle them very quickly. I would guess that your threshold for alerts would be on the order of thousands to tens-of-thousands (for a relatively small cluster). One way to get a feel for how they work is to remove a container drive from the system (ie unmount it) and watch the asyncs climb. Remount the drive (ie fix the issue) and watch how quickly they come down. Based on your cluster size, your traffic patterns, and your hardware SLA, you will be able to determine an appropriate alert level.

You can use swift-recon to find the number of async pendings in the system.

Revision history for this message
David Kranz (david-kranz) said : 		#2

Thanks, John. That helped a lot. I think I got off track because 10 seems to be what zenoss says it will warn on by default.