OpenStack Object Storage (swift)

Account PUT returning 503

Asked by lisha

swift failed to creare a new account when I　use tempauth

I　have deployed swift 1.4.3 on ubuntu 10.04 and it works fine
then when I　tried to switch to scientifc linux 5.5 which is similar to RHEL5.5 and CentOS5.5
something goes wrong

[root@fsc01 ~]# curl -k -v -H 'X-Storage-User: system:root' -H 'X-Storage-Pass: testpass' https:/mysite:8080/auth/v1.0
* About to connect() to fsc01.ihep.ac.cn port 8080
* Trying 192.168.61.83... connected
* Connected to mysite (192.168.61.83) port 8080
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSLv2, Client hello (1):
SSLv3, TLS handshake, Server hello (2):
SSLv3, TLS handshake, CERT (11):
SSLv3, TLS handshake, Server finished (14):
SSLv3, TLS handshake, Client key exchange (16):
SSLv3, TLS change cipher, Client hello (1):
SSLv3, TLS handshake, Finished (20):
SSLv3, TLS change cipher, Client hello (1):
SSLv3, TLS handshake, Finished (20):
SSL connection using AES256-SHA
* Server certificate:
* subject: /C=CN/ST=Beijing/L=Beijing/O=IHEP/OU=CC
* start date: 2011-12-23 15:53:44 GMT
* expire date: 2012-01-22 15:53:44 GMT
* common name: WARNING couldn't obtain
* issuer: /C=CN/ST=Beijing/L=Beijing/O=IHEP/OU=CC
* SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /auth/v1.0 HTTP/1.1
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: fsc01.ihep.ac.cn:8080
> Accept: */*
> X-Storage-User: system:root
> X-Storage-Pass: testpass
>
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain
< Content-Length: 789
< Date: Mon, 26 Dec 2011 09:14:45 GMT
< Connection: close
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/eventlet-0.9.16-py2.7.egg/eventlet/wsgi.py", line 336, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/swift-1.4.3-py2.7.egg/swift/common/middleware/healthcheck.py", line 38, in __call__
    return self.app(env, start_response)
  File "/usr/local/lib/python2.7/site-packages/swift-1.4.3-py2.7.egg/swift/common/middleware/memcache.py", line 32, in __call__
    return self.app(env, start_response)
  File "/usr/local/lib/python2.7/site-packages/swift-1.4.3-py2.7.egg/swift/common/middleware/tempauth.py", line 136, in __call__
    '%s' % (account_id, key))
Exception: Could not create account AUTH_system for user system:root
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):

then I　check 　proxy-server log
Dec 26 14:50:51 fsc01 proxy-server - - 26/Dec/2011/06/50/51 HEAD /v1/AUTH_system HTTP/1.0 404 - TempAuth - - - - - - 0.0092
Dec 26 14:50:51 fsc01 proxy-server Account PUT returning 503 for (503, 503, 503)
Dec 26 14:50:51 fsc01 proxy-server - - 26/Dec/2011/06/50/51 PUT /v1/AUTH_system HTTP/1.0 503 - TempAuth - - - - - - 0.0566

and the account server log

Dec 26 14:50:51 fsc02 account-server 192.168.61.83 - - [26/Dec/2011:06:50:51 +0000] "HEAD /sde1/205350/AUTH_system" 404 - "-" "-" "TempAuth" 0.0004 ""
Dec 26 14:50:51 fsc02 account-server ERROR __call__ error with PUT /sde1/205350/AUTH_system : Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/swift-1.4.3-py2.7.egg/swift/account/server.py", line 305, in __call__ res = getattr(self, req.method)(req) File "/usr/local/lib/python2.7/site-packages/swift-1.4.3-py2.7.egg/swift/account/server.py", line 106, in PUT broker.initialize(timestamp) File "/usr/local/lib/python2.7/site-packages/swift-1.4.3-py2.7.egg/swift/common/db.py", line 245, in initialize self.conn = get_db_connection(self.db_file, self.timeout) File "/usr/local/lib/python2.7/site-packages/swift-1.4.3-py2.7.egg/swift/common/db.py", line 148, in get_db_connection timeout=timeout) DatabaseConnectionError: DB connection error (/srv/node/sde1/accounts/205350/ca2/c8898f67451ad4a98fe3ec4b8273bca2/c8898f67451ad4a98fe3ec4b8273bca2.db, 25): Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/swift-1.4.3-py2.7.egg/swift/common/db.py", line 14
Dec 26 14:50:51 fsc02 account-server 192.168.61.83 - - [26/Dec/2011:06:50:51 +0000] "PUT /sde1/205350/AUTH_system" 500 1503 "-"

It seems to be a problem with database connnetion.

I　was wondering if there is requiement on the version of sqlite , since on sl5.5（or CENTOS 5.5 ）, it is 3.3.6

I donnt know how to get the database work

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Solved by:
lisha
Solved:
Last query:
Last reply:
Revision history for this message
lisha (lisha880730) said : 		#1

I also try to user swauth
I edited the proxy-server.conf and switch tempauth and then tried to do a swauth-prep to initialize it. But this takes very long and does not seem to complete in 30 minutes. When I do a ctrl+c to interrupt, I get the following message:Traceback (most recent call last):

 File "/usr/local/bin/swauth-prep", line 5, in <module>
    pkg_resources.run_script('swauth==1.0.2', 'swauth-prep')
  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in run_script
  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1214, in run_script
  File "/usr/local/lib/python2.7/site-packages/swauth-1.0.2-py2.7.egg/EGG-INFO/scripts/swauth-prep", line 57, in <module>

  File "/usr/local/lib/python2.7/site-packages/swift-1.4.3-py2.7.egg/swift/common/bufferedhttp.py", line 95, in getresponse
    response = HTTPConnection.getresponse(self)
  File "/usr/local/lib/python2.7/httplib.py", line 1027, in getresponse
    response.begin()
  File "/usr/local/lib/python2.7/httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline()
  File "/usr/local/lib/python2.7/socket.py", line 447, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/local/lib/python2.7/site-packages/eventlet-0.9.16-py2.7.egg/eventlet/greenio.py", line 238, in recv
    timeout_exc=socket.timeout("timed out"))
  File "/usr/local/lib/python2.7/site-packages/eventlet-0.9.16-py2.7.egg/eventlet/hubs/__init__.py", line 121, in trampoline
    return hub.switch()
  File "/usr/local/lib/python2.7/site-packages/eventlet-0.9.16-py2.7.egg/eventlet/hubs/hub.py", line 177, in switch
    return self.greenlet.switch()
  File "/usr/local/lib/python2.7/site-packages/eventlet-0.9.16-py2.7.egg/eventlet/hubs/hub.py", line 226, in run
    self.wait(sleep_time)
  File "/usr/local/lib/python2.7/site-packages/eventlet-0.9.16-py2.7.egg/eventlet/hubs/poll.py", line 84, in wait
    presult = self.do_poll(seconds)
  File "/usr/local/lib/python2.7/site-packages/eventlet-0.9.16-py2.7.egg/eventlet/hubs/epolls.py", line 59, in do_poll
    return self.poll.poll(seconds)

what' worse , when i check the syslog neither proxy server nor swauth generate any messages.

maybe the database connection error logged when I use tempauth also has something to do with this?

It's just so frastrating that I get no log message at all this time which makes it more difficult to diagnose the problem

Revision history for this message
lisha (lisha880730) said : 		#2

Now I know that I got no response Is because I use http rather than https,
I modify the parameter to

# swauth-prep -K swauthkey -A https://localhost:8080/auth/
Auth subsystem prep failed: 500 Server Error

I still get server error , similar error when I use tempauth

================
proxy -server log

Dec 26 21:47:36 fsc01 proxy-server Account PUT returning 503 for (503, 503, 503)
Dec 26 21:47:36 fsc01 proxy-server - - 26/Dec/2011/13/47/36 PUT /v1/AUTH_.auth HTTP/1.0 503 - Swauth - - - - - - 0.0285
Dec 26 21:47:36 fsc01 proxy-server STDOUT: EXCEPTION IN handle: Traceback (most recent call last): File "build/bdist.linux-x86_64/egg/swauth/middleware.py", line 401, in handle return self.handle_request(req)(env, start_response) File "build/bdist.linux-x86_64/egg/swauth/middleware.py", line 466, in handle_request req.response = handler(req) File "build/bdist.linux-x86_64/egg/swauth/middleware.py", line 485, in handle_prep (path, resp.status)) Exception: Could not create the main auth account: /v1/AUTH_.auth 503 Internal Server Error : {'SCRIPT_NAME': '/auth/v2', 'webob.adhoc_attrs': {'start_time': 1324907256.516816, 'bytes_transferred': '-', 'client_disconnect': False}, 'REQUEST_METHOD': 'POST', 'PATH_INFO': '/.prep', 'SERVER_PROTOCOL': 'HTTP/1.0', 'eventlet.posthooks': [(<bound method Swauth.posthooklogger of <swauth.middleware.Swauth object at 0x15956090>>, (<Request at 0x1596e710 POST https://localhost:8080/auth/v2/.prep>,), {})], 'SERVER_NAME': '127.0.0.1', 'REMOTE_ADDR': '127.0.0.1', 'eventlet.input'

============
account server log
Dec 26 21:47:36 fsc02 account-server ERROR __call__ error with PUT /sde1/175574/AUTH_.auth : Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/swift-1.4.3-py2.7.egg/swift/account/server.py", line 305, in __call__ res = getattr(self, req.method)(req) File "/usr/local/lib/python2.7/site-packages/swift-1.4.3-py2.7.egg/swift/account/server.py", line 106, in PUT broker.initialize(timestamp) File "/usr/local/lib/python2.7/site-packages/swift-1.4.3-py2.7.egg/swift/common/db.py", line 245, in initialize self.conn = get_db_connection(self.db_file, self.timeout) File "/usr/local/lib/python2.7/site-packages/swift-1.4.3-py2.7.egg/swift/common/db.py", line 148, in get_db_connection timeout=timeout) DatabaseConnectionError: DB connection error (/srv/node/sde1/accounts/175574/d96/ab75adda25674b71efe1ad5b7d562d96/ab75adda25674b71efe1ad5b7d562d96.db, 25): Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/swift-1.4.3-py2.7.egg/swift/common/db.py", line 143
Dec 26 21:47:36 fsc02 account-server 192.168.61.83 - - [26/Dec/2011:13:47:36 +0000] "PUT /sde1/175574/AUTH_.auth" 500 1503 "-" "-" "-" 0.0191 ""

I guess it really is the db connection problem

any idea how can I fix this?

Revision history for this message
lisha (lisha880730) said : 		#3

the version of sqlite is not right, I upgrade it from 3.4 to 3.6, now swift seems to work fine