How to work with S3 API using Swift3?

Asked by Shi Jin

Hi there,

I am wondering if there is any more documentation on how to get swift3 to work with S3 API.
Specifically, if I created a swift account by
seki@OS-CC:/var/log$ swift-auth-add-user -K devauth -a system root testpass
https://192.168.1.33:8080/v1/AUTH_365f77c9d523435dbcf12c9d2678d197
And get the following
seki@OS-CC:/var/log$ curl -k -v -H 'X-Storage-User: system:root' -H 'X-Storage-Pass: testpass' https://192.168.1.33:11000/v1.0
...
< X-Storage-Url: https://192.168.1.33:8080/v1/AUTH_365f77c9d523435dbcf12c9d2678d197
< X-Storage-Token: AUTH_tka3599de5039545809d181637d0f010a9
< X-Auth-Token: AUTH_tka3599de5039545809d181637d0f010a9
...

Should I set the following variables
export EC2_ACCESS_KEY=AUTH_tka3599de5039545809d181637d0f010a9
export EC2_SECRET_KEY=testpass
export S3_URL=https://192.168.1.33:8080/v1/AUTH_365f77c9d523435dbcf12c9d2678d197

I've added to the following to /etc/swift/proxy-server.conf
[filter:swift3]
use = egg:swift#swift3
log_facility = LOG_LOCAL1

But I am still getting 401 error:
seki@OS-CC:~/s3-curl$ ./s3curl.pl --id $EC2_ACCESS_KEY --key $EC2_SECRET_KEY --get -- -s -v $S3_URL -k
Unknown option: get
WARNING: It isn't safe to put your AWS secret access key on the
command line! The recommended key management system is to store
your AWS secret access keys in a file owned by, and only readable
by you.

For example:

%awsSecretAccessKeys = (
    # personal account
    personal => {
        id => '1ME55KNV6SBTR7EXG0R2',
        key => 'zyMrlZUKeG9UcYpwzlPko/+Ciu0K2co0duRM3fhi',
    },

    # corporate account
    company => {
        id => '1ATXQ3HHA59CYF1CVS02',
        key => 'WQY4SrSS95pJUT95V6zWea01gBKBCL6PI0cdxeH8',
    },
);

$ chmod 600 /home/seki/.s3curl

Will sleep and continue despite this problem.
Please set up /home/seki/.s3curl for future requests.
* About to connect() to 192.168.1.33 port 8080 (#0)
* Trying 192.168.1.33... connected
* Connected to 192.168.1.33 (192.168.1.33) port 8080 (#0)
* successfully set certificate verify locations:
* CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA
* Server certificate:
* subject: C=CA; ST=AB; L=Edmonton; O=VRS; OU=RD; CN=OS-CC; <email address hidden>
* start date: 2011-04-23 15:55:37 GMT
* expire date: 2011-05-23 15:55:37 GMT
* common name: OS-CC (does not match '192.168.1.33')
* issuer: C=CA; ST=AB; L=Edmonton; O=VRS; OU=RD; CN=OS-CC; <email address hidden>
* SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /v1/AUTH_365f77c9d523435dbcf12c9d2678d197 HTTP/1.1
> User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
> Host: 192.168.1.33:8080
> Accept: */*
> Date: Tue, 26 Apr 2011 16:42:05 +0000
> Authorization: AWS AUTH_tka3599de5039545809d181637d0f010a9:6L/VuKi4ZT5YQkI9JwnVIMT2TcI=
>
< HTTP/1.1 401 Unauthorized
< Content-Type: text/html; charset=UTF-8
< Content-Length: 364
< Date: Tue, 26 Apr 2011 16:42:05 GMT
<
<html>
 <head>
  <title>401 Unauthorized</title>
 </head>
 <body>
  <h1>401 Unauthorized</h1>
  This server could not verify that you are authorized to
access the document you requested. Either you supplied the
wrong credentials (e.g., bad password), or your browser
does not understand how to supply the credentials required.
<br /><br />

 </body>
* Connection #0 to host 192.168.1.33 left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):

Question information

Language:
English Edit question
Status:
Answered
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Lior Goikhburg (lgoikhburg) said :
#1

Do you have swift3 enabled in the auth pipeline ?
like:

[pipeline:main]
pipeline = healthcheck cache swift3 swauth proxy-server

Revision history for this message
Shi Jin (jinzishuai) said :
#2

Thanks.
I didn't have swift3 there.

Now i added it

seki@OS-CC:/var/log$ cat /etc/swift/proxy-server.conf
[DEFAULT]
cert_file = /etc/swift/cert.crt
key_file = /etc/swift/cert.key
bind_port = 8080
workers = 8
user = swift

[pipeline:main]
pipeline = healthcheck cache auth proxy-server swift3

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true

[filter:swift3]
use = egg:swift#swift3
log_facility = LOG_LOCAL1

[filter:auth]
use = egg:swift#auth
ssl = true

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:cache]
use = egg:swift#memcache
memcache_servers = 127.0.0.1:11211

Then I cannot start the proxy service. syslog shows the following error:

Apr 26 12:00:54 OS-CC proxy-server UNCAUGHT EXCEPTION#012Traceback (most recent call last):#012 File "/usr/bin/swift-proxy-server", line 22, in <module>#012 run_wsgi(conf_file, 'proxy-server', default_port=8080, **options)#012 File "/usr/lib/pymodules/python2.6/swift/common/wsgi.py", line 126, in run_wsgi#012 app = loadapp('config:%s' % conf_file, global_conf={'log_name': log_name})#012 File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 204, in loadapp#012 return loadobj(APP, uri, name=name, **kw)#012 File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 224, in loadobj#012 global_conf=global_conf)#012 File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 248, in loadcontext#012 global_conf=global_conf)#012 File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 278, in _loadconfig#012 return loader.get_context(object_type, name, global_conf)#012 File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 405, in get_context#012 global_additions=global_additions)#012 File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 500, in _pipeline_app_context#012 APP, pipeline[-1], global_conf)#012 File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 363, in get_context#012 object_type, name=name)#012 File "/usr/lib/pymodules/python2.6/paste/deploy/loadwsgi.py", line 528, in find_config_section#012 self.filename))#012LookupError: No section 'swift3' (prefixed by 'app' or 'application' or 'composite' or 'composit' or 'pipeline' or 'filter-app') found in config /etc/swift/proxy-server.conf

Should I call it filter-app instead of filter?
Where do I find the detailed reference on how to write this documentation?
I am running ubuntu-10.10 and swift-1.2.0-0ubuntu1~maverick0.
Thanks.

Shi

Revision history for this message
Chuck Thier (cthier) said :
#3

When using the s3 compatibility layer, the access key needs to be in the form of account_name:user_name, and the secret key used to sign the request is the user's password. So in the above example, EC2_ACCESS_KEY=system:root

and in your config, the pipeline should look something like:

pipeline = healthcheck cache swift3 auth proxy-server

http://swift.openstack.org/misc.html#module-swift.common.middleware.swift3

Is the only documentation that we have currently.

Revision history for this message
Force (force++) said :
#4

following worked for me after a lot of trying(Swift 1.4):

-->proxy-server.conf
[pipeline:main]
pipeline = healthcheck cache swift3 swauth proxy-server

[filter:swift3]
use = egg:swift#swift3

-->configure s3curl.pl
Add your Host to the @endpoints array in line 33
     my @endpoints = ( '1.2.3.4');

-->using s3curl.pl
    ./s3curl.pl --id 'myacc:myuser' --key mypw --get -- -s -v http://1.2.3.4:8080
Output:
...< HTTP/1.1 200 OK....
?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://doc.s3.amazonaws.com/2006-03-01">
    <Buckets>
        <Bucket>
            <Name>testdir</Name><CreationDate>2009-02-03T16:45:09.000Z</CreationDate>
        </Bucket>
    </Buckets>
</ListAllMyBucketsResult>

Revision history for this message
Graham Hemingway (graham-hemingway) said :
#5

I have setup my swift3 config per what you have here and want to test it out. You wouldn't have any idea perhaps about what a boto (python) connection script might look like would you?

Thanks for the great work on figuring this out.
Cheers,
   Graham

Can you help with this problem?

Provide an answer of your own, or ask Shi Jin for more information if necessary.

To post a message you must log in.