OpenStack Object Storage (swift)

Slow response to expiration time update

Asked by John Burke

Hi, I am using Swift as a cache for files that get up into the 10s of MB, and after fetching one I update its X-Delete-After time (in order to do approximate LRU by having unused stuff expire). I've noticed that for big files, this operation takes longer, which doesn't make sense since I am not sending any data, just headers, and I'm only getting back a small message, and on the server side this is (supposedly) handled asynchronously anyway. I don't have many files on swift since I am basically testing it out right now, so I don't think that is the problem. For example, updating the expiration time for a ~50MB file took min 2.21s, avg 7.5s, and max 28s (!). Here is the command I called 100 times to test this:
curl -silent -k -X POST -H 'X-Auth-Token: [my auth token]' -H 'X-Delete-After: 604800' [my storage url]/[container]/[filename]

Any help would be greatly appreciated, taking 7 seconds on average to do this is really not acceptable for our use case (also, for the record, ping times to the servers in question are sub-ms and they have gigabit connections, so I don't think its a network problem).

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Solved by:
David Goetz
Solved:
Last query:
Last reply:

This question was reopened

Revision history for this message
David Goetz (david-goetz) said : 		#1

By default all POSTs in swift actually turn into PUT/COPYs. This means that the larger your file is the longer the POST will take because it has to COPY the entire object. If this is not acceptable you can set this flag in the proxy-server.conf:

object_post_as_copy = false

(from docs in etc/proxy-server.conf-sample)

# Set object_post_as_copy = false to turn on fast posts where only the metadata
# changes are stored anew and the original data file is kept in place. This
# makes for quicker posts; but since the container metadata isn't updated in
# this mode, features like container sync won't be able to sync posts.
# object_post_as_copy = true

Revision history for this message
David Goetz (david-goetz) said : 		#2

oh- looks like you also can't update Content-Type with a POST if you have 'object_post_as_copy = false' . Hopefully thats not a problem...

Revision history for this message
John Burke (jcburke) said : 		#3

Thanks David Goetz, that solved my question.

Revision history for this message
John Burke (jcburke) said : 		#4

Do you think you could explain a little more why this is the default, and what the consequences are of setting it to 'false'? It seems strange to copy the entire object every time its metadata is updated by default, and container-to-container sync seems like a rather obscure feature to be worth such a performance-costly default setting. Is there some other functionality that would break if I change this setting? Also, can it be set on a per-container basis, or does it have to apply to the whole cluster?

Revision history for this message
Best David Goetz (david-goetz) said : 		#5

Both object_post_as_copy true and false settings are supported by the API so other than the mentioned exceptions POSTs will act as expected. As to why this this setting is needed it has to do with how swift stores its object metadata in the files on disk in conjunction with the metadata stored in the container servers. POST performance is not typically a large concern in swift clusters so the default is set for keeping content-type in sync and the having the ability to use container-sync.

Revision history for this message
John Burke (jcburke) said : 		#6

Thanks David Goetz, that solved my question.

Revision history for this message
John Burke (jcburke) said : 		#7

Also, thanks for responding so quickly! You guys are clearly doing a great job staying on top of this forum