Most reliable way to test uploading a 5GB object.

Asked by Travis W Kiel

Avoiding timeouts, what is the best method to send up a 5GB file (testing limits)?

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Solved by:
Travis W Kiel
Solved:
Last query:
Last reply:
Revision history for this message
gholt (gholt) said :
#1

Upload it on a network local to the cluster? I'm not sure I entirely understand the question though.

Revision history for this message
Travis W Kiel (twkiel) said :
#2

On the local network, with Swift installed with default settings, I occassionally get timeouts on files several GB in size. I assume this is due to the rate limiting settings. I was thinking as S3 does an alternate mechanism for uploading large files, possibly theres more than one way with Swift. If there is only the one way, I can just look into the rate limiting settings/network.

Revision history for this message
gholt (gholt) said :
#3

Ah gotcha, yeah there's just the one way right now. What are the optional ways with S3? We could investigate such features for Swift.

Revision history for this message
Travis W Kiel (twkiel) said :
#4

I take this back, Amazon has a service for S3 that lets you send in portable storage for them to manually load. Sorry for the confusion, and thanks for your help.

Revision history for this message
dimm (dmitry-pha) said :
#5

epochasset: Did you figure out the proper rate limiting settings?

Revision history for this message
Travis W Kiel (twkiel) said :
#6

Internal tests seemed to have enough bandwidth to where it wasnt a problem. When I ran tests from a remote office I believe their usage caused it.

Revision history for this message
dimm (dmitry-pha) said :
#7

I'm uploading to the localhost and getting this error (only 1 node in this zone, maybe that's a problem).
What setting should I change?

Revision history for this message
Travis W Kiel (twkiel) said :
#8

The example config appears to have ratelimiting off. But I also think the ratelimiting is by request, so a single operation is likely unaffected. At that point I would wonder what storage you are using, what size object times out in write, and how long it takes before failure.