OpenStack Object Storage (swift)

Perform a GET and get http/ 1.1 503 Internal Server Error

Asked by jimmy on 2013-07-10

Hello,

I just setup swift env in my lab for testing. When i perform <curl -v -H 'X-Auth-Token: AUTH_tkadaffa35a3b4458f96a95c4377c3e496' http://127.0.0.1:8080/v1/AUTH_admin> and I am getting http/1.1 503 Internal Server Error. Can someone please help me with this?

[root@cleverloadgen11 swift]# curl -v -H 'X-Auth-Token: AUTH_tkadaffa35a3b4458f96a95c4377c3e496' http://127.0.0.1:8080/v1/AUTH_admin
* About to connect() to 127.0.0.1 port 8080 (#0)
* Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /v1/AUTH_admin HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
> Host: 127.0.0.1:8080
> Accept: */*
> X-Auth-Token: AUTH_tkadaffa35a3b4458f96a95c4377c3e496
>
< HTTP/1.1 503 Internal Server Error
< Content-Type: text/html; charset=UTF-8
< Content-Length: 0
< Date: Wed, 10 Jul 2013 17:59:03 GMT
<
* Connection #0 to host 127.0.0.1 left intact
* Closing connection #0
[root@cleverloadgen11 swift]#

Thank,
Jimmy

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Solved by:
jimmy
Solved:
2013-07-18
Last query:
2013-07-18
Last reply:
2013-07-11

This question was originally filed as bug #1199928.

Kun Huang (academicgareth) said : #1

There may be some config error during deployment, but bugs

jimmy (jlu) said : #2

I rebuilt the rings to the correct mount points and restart all services. When perform <curl -v -H 'X-Auth-Token: AUTH_tk95a0d54e054e40a28f2386bba26c76bd' http://10.218.36.74:8080/v1/AUTH_admin> and I am getting "Bad request with 400" error. The 503 error is no longer there. I am confused..:(

Thank you much!!

-Jimmy

<log from 1 of the storage node>

Jul 10 23:28:54 cleverloadgen16 object-expirer Unhandled exception: #012Traceback (most recent call last):#012 File "/usr/lib/python2.6/site-packages/swift/obj/expirer.py", line 91, in run_once#012 self.swift.get_account_info(self.expiring_objects_account)#012 File "/usr/lib/python2.6/site-packages/swift/common/internal_client.py", line 336, in get_account_info#012 resp = self.make_request('HEAD', path, {}, acceptable_statuses)#012 File "/usr/lib/python2.6/site-packages/swift/common/internal_client.py", line 162, in make_request#012 _('Unexpected response: %s' % (resp.status,)), resp)#012UnexpectedResponse: (UnexpectedResponse(...), 'Unexpected response: 400 Bad Request') (txn: tx57c71b791e794a31bcda7b3d462f9f55)

Jul 10 23:37:36 cleverloadgen15 account-replicator Exception dumping recon cache: #012Traceback (most recent call last):#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1284, in dump_recon_cache#012 with lock_file(cache_file, lock_timeout, unlink=False) as cf:#012 File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__#012 return self.gen.next()#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 852, in lock_file#012 fd = os.open(filename, flags)#012OSError: [Errno 2] No such file or directory: '/var/cache/swift/account.recon'
Jul 10 23:37:36 cleverloadgen15 account-replicator no_change:0 ts_repl:0 diff:0 rsync:0 diff_capped:0 hashmatch:0 empty:0
Jul 10 23:37:38 cleverloadgen15 object-server 10.218.36.76 - - [11/Jul/2013:06:37:38 +0000] "HEAD /sdb/231937/.expiring_objects" 400 43 "-" "tx2c7ac77aa8884fe0afc673851e616d41" "Swift Object Expirer" 0.0002
Jul 10 23:37:42 cleverloadgen15 object-server 10.218.36.76 - - [11/Jul/2013:06:37:42 +0000] "HEAD /sdb/231937/.expiring_objects" 400 43 "-" "tx6f12758f0342476eb9dfab996f9d278e" "Swift Object Expirer" 0.0002
Jul 10 23:37:51 cleverloadgen15 object-server 10.218.36.78 - - [11/Jul/2013:06:37:51 +0000] "HEAD /sdb/231937/.expiring_objects" 400 43 "-" "txee1462eb0ef44460bdb91c946fb3d4d7" "Swift Object Expirer" 0.0002

Samuel Merritt (torgomatic) said : #3

The answer is in the stack trace you posted.

Quoting: "OSError: [Errno 2] No such file or directory: '/var/cache/swift/account.recon'"

Please make sure /var/cache/swift exists and is writable by the user that the Swift processes run as (probably "swift", but YMMV).

jimmy (jlu) said : #4

Hi Samuel,

I had already created the file and changed the ownership to swift:swift and I don't see that error in the log.

This is my observation with the 400 bad request.

I have 5 storage nodes and 1 proxy node:

When I perform <curl -v -H "X-Auth-Token: AUTH_tk95a0d54e054e40a28f2386bba26c76bd" http://10.218.36.74:8080/v1/AUTH_admin>, I get a "400 Bad Request" in return.

What I noticed, on node3,5,6, i see these errors

node3:
Jul 11 13:44:15 cleverloadgen13 object-server 10.218.36.74 - - [11/Jul/2013:20:44:15 +0000] "GET /sdb/260881/AUTH_admin" 400 36 "-" "tx49b006fb16554e818d29e96ecaef4c53" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2" 0.0002

node5:
Jul 11 13:44:15 cleverloadgen15 object-server 10.218.36.74 - - [11/Jul/2013:20:44:15 +0000] "GET /sdb/260881/AUTH_admin" 400 36 "-" "tx49b006fb16554e818d29e96ecaef4c53" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2" 0.0002

node6:
Jul 11 13:44:15 cleverloadgen16 object-server 10.218.36.74 - - [11/Jul/2013:20:44:15 +0000] "GET /sdb/260881/AUTH_admin" 400 36 "-" "tx49b006fb16554e818d29e96ecaef4c53" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2" 0.0002

and node2 and node4 - i do not see such errors.

Thanks and appreciated for the help!!

-Jimmy

jimmy (jlu) said : #5

This issue is resolved. It was due to port mismatch when building the ring and in the rsyncd.conf for *-server.conf.

Thanks,
Jimmy