OpenStack Object Storage (swift)

Problem with folders/directories while using S3 middletier

Asked by Lukasz Bachman

Hi all,
I'd like to use Swift as a S3 compatible interface. To do that, we have been using middletier written in Python, provided with swift. The problem I had encountered is happening, when I'm trying to upload an object with path, let's say: bucketname/a/b/c.txt
If directories 'a' and 'b' does not exist, the file will fail to upload with status message:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>InvalidURI</Code>
  <Message>Could not parse the specified URI</Message>
</Error>

On the other hand, if I upload file directly to the bucket: bucketname/c.txt everything works as expected.

I guess this is somehow related to this topic in documentation: http://docs.openstack.org/cactus/openstack-object-storage/developer/content/pseudo-hierarchical-folders-directories.html

But that bug itself puts compatibility with S3 in doubt, since S3 behaves properly in that case - forces creating directories.

Any help would be appreciated, since this is extremely important case for me.

Thanks.

P.S. I'm using S3curl for tests.

Question information

Language:
English Edit question
Status:
Answered
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Chuck Thier (cthier) said : 		#1

Hi Lukasz,

Does S3curl work for you in your swift cluster? Other's are having issues with getting it to work with the middleware, so I would be curious to hear if you had to do anything special to make it work?

As to the files that you are uploading. is it a file named "a/b/c.txt" or are a and b directories locally?

Revision history for this message
Lukasz Bachman (lukaszbachman) said : 		#2

Hi, thanks for a quick reply. I really appreciate it.

When it comes down to using s3curl - I think the only thing I configured was a parameter inside 's3curl.pl' script, which was listing 'known hosts'. The variable was called 'endopoints', if I recall. Check out this post: http://open.eucalyptus.com/wiki/s3curl
I'll try to contact my sys admin to ask if that was the only thing that he has configured.

I don't actually get the other question. I'm trying to send local file from my desktop to specified directory (path) on remote Swift. For instance:

s3curl.pl --id MY_USERNAME --key MY_SECRET_PASS --put some_file.txt -- -s -v https://mystorage.com/mybucket/c.txt
Response: 200

s3curl.pl --id MY_USERNAME --key MY_SECRET_PASS --put some_file.txt -- -s -v https://mystorage.com/mybucket/a/b/c.txt
Response: error - InvalidURI

As you can see the only thing that changes is the destination on my swift storage. When I upload straing to bucket 'mybucket' everything is cool, since this object exists. But when I upload to folder 'mybucket/a/b' - I get an error. S3 would accept that request, no biggie - it would create folders 'a' and 'b'. But Swift somehow does not cope with it.

The other interesting thing is - my sys admin contacted me and told me that he tried achieving the same WITHOUT S3Middletier and it worked! So I guess it now look like python script handling Middletier does it wrong...

Revision history for this message
Chuck Thier (cthier) said : 		#3

Ah yes, I just found the known hosts things a couple of hours ago :)

and now I can reproduce the error that you were seeing. Turns out there is a small bug in the swift3 middleware.

I will attach a bug to this question, and get the fix in soon. Thanks for reporting the issue!

--
Chuck

Revision history for this message
Lukasz Bachman (lukaszbachman) said : 		#4

That's truly great news! Could You please let me know when could I expect this bug to be fixed? The whole decision upon whether to use Swift or not depends on it :-)

Revision history for this message
Chuck Thier (cthier) said : 		#5

I'm working on it right now, so hopefully today or tomorrow :)

Revision history for this message
Lukasz Bachman (lukaszbachman) said : 		#6

Chuk, YOU ARE THE MAN :-) Thanks!

Revision history for this message
Lukasz Bachman (lukaszbachman) said : 		#7

Edit: Chuck, ofcourse :)

Revision history for this message
Chuck Thier (cthier) said : 		#8

The fix was just committed to trunk. Let me know if you run into any more issues.

Revision history for this message
Lukasz Bachman (lukaszbachman) said : 		#9

Ok, will do. I'll check it out first thing in the morning (in about 9h).

Can you help with this problem?

Provide an answer of your own, or ask Lukasz Bachman for more information if necessary.

To post a message you must log in.