OpenStack Object Storage (swift)

Performance Issue with Large Files

Asked by supp

I have installed Swift integrated with Keystone and when trying to retrieve large files (almost 3GB) it is taking 4 minutes approximately. I know that we should consider another aspects such as network bandwidth, etc. but I believe the problem comes from the fact that I use a single partition for storage (I followed "Swift - All In One" instructions) with a 30 GB disk containing 262144 partitions. My question is, can this generate a perfomance issue when trying to get objects from Swift? Is it not that a large number of partitions considering the disk size I used?
I can provide more details if needed.
Thanks in advance.

Question information

Language:
English Edit question
Status:
Answered
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
supp (suppldp2001) said : 		#1

What I basically would like to know is what is an acceptable data transfer bandwidth for Swift for files about 3 GB in size?

Revision history for this message
Chuck Thier (cthier) said : 		#2

For large files like that, swift will be limited by either your network throughput or your available disk io (whichever is smaller). A general rule of thumb is a 1G network and a single SATA drive can support on the order of magnitude of 100MB/s. This will be further limited if there is any other network traffic, or network shaping taking place, or if there is other disk io being done at the same time.

As a side note, if you are running everything on a single drive, then it is also likely that you have a lot of processes going on as well (like replication, etc.) that can eat up disk io if not properly tuned. You might try testing with the other services turned off to see if you need to tune those a bit further.

--
Chuck

Revision history for this message
supp (suppldp2001) said : 		#3

Thank you Chuck. I will take into consideration your advice.

I am using Swift as a storage backend for Glance. And every time I run a "glance delete" the image file is not deleted from Swift. Is it a common behaviour? Is this correct or the image file should also be deleted from Swift?

Thanks in advance.-

Revision history for this message
Chuck Thier (cthier) said : 		#4

I don't really have experience with glance sorry.

Can you help with this problem?

Provide an answer of your own, or ask supp for more information if necessary.

To post a message you must log in.