OpenStack Object Storage (swift)

Update failed 503 service unavailable

Asked by Hugo Kou on 2010-11-05

Hi all
I'm facing a problem at below step ~~~

swift-auth-add-user -K [MYadmin-key] -a test2 tester2 testing2

↓↓I got a response↓↓

Update failed: 503 service unavailable

==================================================

I've readed https://answers.launchpad.net/swift/+question/127329 (503 service Unavaliable by Ryan Rodd) before,
and I tried to solve the problem by this article.....Unfortunately , I'm still in trouble.
Would someone plz take a look of my configuration?

====Using a loopback device for storage====================
[Ports of server]

Account-server
   1.conf 6012
   2.conf 6022
   3.conf 6032
   4.conf 6042

container-server
   1.conf 6011
   2.conf 6021
   3.conf 6031
   4.conf 6041

object-server
   1.conf 6010
   2.conf 6020
   3.conf 6030
   4.conf 6040

auth-server.conf
[DEFAULT]
user = hugo

[pipeline:main]
pipeline = auth-server

   [app:auth-server]
   use = egg:swift#auth
   default_cluster_url = http://127.0.0.1:8080/v1
   # Highly recommended to change this.
   super_admin_key = tony1020

proxy-server.conf
   [DEFAULT]
   bind_port = 8080
   user = hugo

[pipeline:main]
pipeline = healthcheck cache auth proxy-server

[app:proxy-server]
use = egg:swift#proxy

   [filter:auth]
   port = 11000
   use = egg:swift#auth

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:cache]
use = egg:swift#memcache
============================================

[IMG]http://i217.photobucket.com/albums/cc280/tonytkdk/NETS.jpg[/IMG]
http://i217.photobucket.com/albums/cc280/tonytkdk/NETS.jpg

All ports r listening............
As I know, when adding a new account/user, the auth server calls out to the proxy server based on the default_cluster_url config file value and it should match the bind_port value

is there any suggestion?

Question information

Language:: English Edit question

Status:: Answered

For:: OpenStack Object Storage (swift) Edit question

Assignee:: No assignee Edit question

Last query:: 2010-11-05

Last reply:: 2010-11-05

Link existing bug

Revision history for this message

gholt (gholt) said on 2010-11-05:

#1

Hmm. Can you find anything thing wrong in your /var/log/syslog or, if you happen to have configured up the log-stats-processing stuff, in /var/log/swift/*

Revision history for this message

Hugo Kou (tonytkdk) said on 2010-11-05:

#2

hello gholt...
in my /var/log/syslog as this

Nov 5 23:36:52 openstack auth-server ERROR Unhandled exception in ReST request: #012Traceback (most recent call last):#012 File "/home/hugo/swift/trunk/swift/auth/server.py", line 612, in handleREST#012 response = handler(req)#012 File "/home/hugo/swift/trunk/swift/auth/server.py", line 471, in handle_add_user#012 create_account_admin, create_reseller_admin)#012 File "/home/hugo/swift/trunk/swift/auth/server.py", line 334, in create_user#012 account_hash = self.add_storage_account()#012 File "/home/hugo/swift/trunk/swift/auth/server.py", line 198, in add_storage_account#012 (token, time()))#012 File "/home/hugo/swift/trunk/swift/common/db.py", line 81, in execute#012 return self._timeout(lambda: sqlite3.Connection.execute(#012 File "/home/hugo/swift/trunk/swift/common/db.py", line 74, in _timeout#012 return call()#012 File "/home/hugo/swift/trunk/swift/common/db.py", line 82, in <lambda>#012 self, *args, **kwargs))#012OperationalError: attempt to write a readonly database

Revision history for this message

clayg (clay-gerrard) said on 2010-11-05:

#3

see:
https://bugs.launchpad.net/swift/+bug/666957

Revision history for this message

gholt (gholt) said on 2010-11-05:

#4

chown hugo /etc/swift /etc/swift/auth.db

That should get you going again.

Revision history for this message

Hugo Kou (tonytkdk) said on 2010-11-06:

#5

Thanks for your help.... and I get into another big trouble
the "Update failed 503 service unavailable" appeared .....the same with this topic

BUT I got very different syslog information as below......
Should I re-install entire OS again....... because it looks like a big trouble for me
did I lost something .......

=============================================================
Nov 7 04:51:31 openstack auth-server validate_token('AUTH_tk035256ff2d9f4fc6ad9dc4eda4b79149', _, _) = (86399.939764976501, '.super_admin', '.single_use', '.reseller_admin') [0.06]

Nov 7 04:51:31 openstack auth-server 127.0.0.1 - - [06/Nov/2010:20:51:31 +0000] "GET /token/AUTH_tk035256ff2d9f4fc6ad9dc4eda4b79149 HTTP/1.0" 204 - "-" "-" - - - - - - - - - "-" "127.0.0.1" "-" 0.0569

Nov 7 04:51:31 openstack account-server ERROR __call__ error with PUT /sdb2/52482/AUTH_14f77faee139431896897f42f0249dba transaction tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e: #012Traceback (most recent call last):#012 File "/home/hugo/swift/trunk/swift/account/server.py", line 304, in __call__#012 res = getattr(self, req.method)(req)#012 File "/home/hugo/swift/trunk/swift/account/server.py", line 107, in PUT#012 broker.initialize(timestamp)#012 File "/home/hugo/swift/trunk/swift/common/db.py", line 178, in initialize#012 mkdirs(self.db_dir)#012 File "/home/hugo/swift/trunk/swift/common/utils.py", line 170, in mkdirs#012 os.makedirs(path)#012 File "/usr/lib/python2.6/os.py", line 150, in makedirs#012 makedirs(head, mode)#012 File "/usr/lib/python2.6/os.py", line 150, in makedirs#012 makedirs(head, mode)#012 File "/usr/lib/python2.6/os.py", line 150, in makedirs#012 makedirs(head, mode)#012 File "/usr/lib/python2.6/os.py", line 157, in makedirs#012 mkdir(name, mode)#012OSError: [Errno 13] Permission denied: '/srv/2/node/sdb2/accounts'

Nov 7 04:51:31 openstack account-server 127.0.0.1 - - [06/Nov/2010:20:51:31 +0000] "PUT /sdb2/52482/AUTH_14f77faee139431896897f42f0249dba" 500 851 "tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e" "-" "-" 0.0009 ""

Nov 7 04:51:31 openstack account-server ERROR __call__ error with PUT /sdb3/52482/AUTH_14f77faee139431896897f42f0249dba transaction tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e: #012Traceback (most recent call last):#012 File "/home/hugo/swift/trunk/swift/account/server.py", line 304, in __call__#012 res = getattr(self, req.method)(req)#012 File "/home/hugo/swift/trunk/swift/account/server.py", line 107, in PUT#012 broker.initialize(timestamp)#012 File "/home/hugo/swift/trunk/swift/common/db.py", line 178, in initialize#012 mkdirs(self.db_dir)#012 File "/home/hugo/swift/trunk/swift/common/utils.py", line 170, in mkdirs#012 os.makedirs(path)#012 File "/usr/lib/python2.6/os.py", line 150, in makedirs#012 makedirs(head, mode)#012 File "/usr/lib/python2.6/os.py", line 150, in makedirs#012 makedirs(head, mode)#012 File "/usr/lib/python2.6/os.py", line 150, in makedirs#012 makedirs(head, mode)#012 File "/usr/lib/python2.6/os.py", line 157, in makedirs#012 mkdir(name, mode)#012OSError: [Errno 13] Permission denied: '/srv/3/node/sdb3/accounts'

Nov 7 04:51:31 openstack account-server 127.0.0.1 - - [06/Nov/2010:20:51:31 +0000] "PUT /sdb3/52482/AUTH_14f77faee139431896897f42f0249dba" 500 851 "tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e" "-" "-" 0.0009 ""

Nov 7 04:51:31 openstack account-server ERROR __call__ error with PUT /sdb1/52482/AUTH_14f77faee139431896897f42f0249dba transaction tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e: #012Traceback (most recent call last):#012 File "/home/hugo/swift/trunk/swift/account/server.py", line 304, in __call__#012 res = getattr(self, req.method)(req)#012 File "/home/hugo/swift/trunk/swift/account/server.py", line 107, in PUT#012 broker.initialize(timestamp)#012 File "/home/hugo/swift/trunk/swift/common/db.py", line 178, in initialize#012 mkdirs(self.db_dir)#012 File "/home/hugo/swift/trunk/swift/common/utils.py", line 170, in mkdirs#012 os.makedirs(path)#012 File "/usr/lib/python2.6/os.py", line 150, in makedirs#012 makedirs(head, mode)#012 File "/usr/lib/python2.6/os.py", line 150, in makedirs#012 makedirs(head, mode)#012 File "/usr/lib/python2.6/os.py", line 150, in makedirs#012 makedirs(head, mode)#012 File "/usr/lib/python2.6/os.py", line 157, in makedirs#012 mkdir(name, mode)#012OSError: [Errno 13] Permission denied: '/srv/1/node/sdb1/accounts'

Nov 7 04:51:31 openstack account-server 127.0.0.1 - - [06/Nov/2010:20:51:31 +0000] "PUT /sdb1/52482/AUTH_14f77faee139431896897f42f0249dba" 500 851 "tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e" "-" "-" 0.0009 ""

Nov 7 04:51:31 openstack account-server ERROR __call__ error with PUT /sdb4/52482/AUTH_14f77faee139431896897f42f0249dba transaction tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e: #012Traceback (most recent call last):#012 File "/home/hugo/swift/trunk/swift/account/server.py", line 304, in __call__#012 res = getattr(self, req.method)(req)#012 File "/home/hugo/swift/trunk/swift/account/server.py", line 107, in PUT#012 broker.initialize(timestamp)#012 File "/home/hugo/swift/trunk/swift/common/db.py", line 178, in initialize#012 mkdirs(self.db_dir)#012 File "/home/hugo/swift/trunk/swift/common/utils.py", line 170, in mkdirs#012 os.makedirs(path)#012 File "/usr/lib/python2.6/os.py", line 150, in makedirs#012 makedirs(head, mode)#012 File "/usr/lib/python2.6/os.py", line 150, in makedirs#012 makedirs(head, mode)#012 File "/usr/lib/python2.6/os.py", line 150, in makedirs#012 makedirs(head, mode)#012 File "/usr/lib/python2.6/os.py", line 157, in makedirs#012 mkdir(name, mode)#012OSError: [Errno 13] Permission denied: '/srv/4/node/sdb4/accounts'

Nov 7 04:51:31 openstack account-server 127.0.0.1 - - [06/Nov/2010:20:51:31 +0000] "PUT /sdb4/52482/AUTH_14f77faee139431896897f42f0249dba" 500 851 "tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e" "-" "-" 0.0009 ""

Nov 7 04:51:31 openstack proxy-server Account PUT returning 503 for [503, 503, 503], transaction tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e

Nov 7 04:51:31 openstack proxy-server - 127.0.0.1 06/Nov/2010/20/51/31 PUT /v1/AUTH_14f77faee139431896897f42f0249dba HTTP/1.0 503 - - .super_admin%3A.single_use%2CAUTH_tk035256ff2d9f4fc6ad9dc4eda4b79149 - - - tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e - 0.0110

Nov 7 04:51:31 openstack auth-server ERROR attempting to create account http://127.0.0.1:8080/v1/AUTH_14f77faee139431896897f42f0249dba: 503 Internal Server Error

Nov 7 04:51:31 openstack auth-server FAILED create_user('test', 'tester', _, True, False) [0.13]

Nov 7 04:51:31 openstack auth-server 127.0.0.1 - - [06/Nov/2010:20:51:31 +0000] "PUT /account/test/tester HTTP/1.0" 503 - "-" "-" - - - - - - - - - "-" "127.0.0.1" "-" 0.1309

=============================================================

Thanks for your help....  and I get into  another big trouble
the "Update failed 503 service unavailable"  appeared .....the same with this topic

BUT I got very different syslog information as below......
Should I re-install entire OS again....... because it looks like a big trouble for me
did I lost something .......

=============================================================
Nov  7 04:51:31 openstack auth-server validate_token('AUTH_tk035256ff2d9f4fc6ad9dc4eda4b79149', _, _) = (86399.939764976501, '.super_admin', '.single_use', '.reseller_admin') [0.06]

Nov  7 04:51:31 openstack auth-server 127.0.0.1 - - [06/Nov/2010:20:51:31 +0000] "GET /token/AUTH_tk035256ff2d9f4fc6ad9dc4eda4b79149 HTTP/1.0" 204 - "-" "-" - - - - - - - - - "-" "127.0.0.1" "-" 0.0569

Nov  7 04:51:31 openstack account-server ERROR __call__ error with PUT /sdb2/52482/AUTH_14f77faee139431896897f42f0249dba transaction tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e: #012Traceback (most recent call last):#012  File "/home/hugo/swift/trunk/swift/account/server.py", line 304, in __call__#012    res = getattr(self, req.method)(req)#012  File "/home/hugo/swift/trunk/swift/account/server.py", line 107, in PUT#012    broker.initialize(timestamp)#012  File "/home/hugo/swift/trunk/swift/common/db.py", line 178, in initialize#012    mkdirs(self.db_dir)#012  File "/home/hugo/swift/trunk/swift/common/utils.py", line 170, in mkdirs#012    os.makedirs(path)#012  File "/usr/lib/python2.6/os.py", line 150, in makedirs#012    makedirs(head, mode)#012  File "/usr/lib/python2.6/os.py", line 150, in makedirs#012    makedirs(head, mode)#012  File "/usr/lib/python2.6/os.py", line 150, in makedirs#012    makedirs(head, mode)#012  File "/usr/lib/python2.6/os.py", line 157, in makedirs#012    mkdir(name, mode)#012OSError: [Errno 13] Permission denied: '/srv/2/node/sdb2/accounts'

Nov  7 04:51:31 openstack account-server 127.0.0.1 - - [06/Nov/2010:20:51:31 +0000] "PUT /sdb2/52482/AUTH_14f77faee139431896897f42f0249dba" 500 851 "tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e" "-" "-" 0.0009 ""

Nov  7 04:51:31 openstack account-server ERROR __call__ error with PUT /sdb3/52482/AUTH_14f77faee139431896897f42f0249dba transaction tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e: #012Traceback (most recent call last):#012  File "/home/hugo/swift/trunk/swift/account/server.py", line 304, in __call__#012    res = getattr(self, req.method)(req)#012  File "/home/hugo/swift/trunk/swift/account/server.py", line 107, in PUT#012    broker.initialize(timestamp)#012  File "/home/hugo/swift/trunk/swift/common/db.py", line 178, in initialize#012    mkdirs(self.db_dir)#012  File "/home/hugo/swift/trunk/swift/common/utils.py", line 170, in mkdirs#012    os.makedirs(path)#012  File "/usr/lib/python2.6/os.py", line 150, in makedirs#012    makedirs(head, mode)#012  File "/usr/lib/python2.6/os.py", line 150, in makedirs#012    makedirs(head, mode)#012  File "/usr/lib/python2.6/os.py", line 150, in makedirs#012    makedirs(head, mode)#012  File "/usr/lib/python2.6/os.py", line 157, in makedirs#012    mkdir(name, mode)#012OSError: [Errno 13] Permission denied: '/srv/3/node/sdb3/accounts'

Nov  7 04:51:31 openstack account-server 127.0.0.1 - - [06/Nov/2010:20:51:31 +0000] "PUT /sdb3/52482/AUTH_14f77faee139431896897f42f0249dba" 500 851 "tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e" "-" "-" 0.0009 ""

Nov  7 04:51:31 openstack account-server ERROR __call__ error with PUT /sdb1/52482/AUTH_14f77faee139431896897f42f0249dba transaction tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e: #012Traceback (most recent call last):#012  File "/home/hugo/swift/trunk/swift/account/server.py", line 304, in __call__#012    res = getattr(self, req.method)(req)#012  File "/home/hugo/swift/trunk/swift/account/server.py", line 107, in PUT#012    broker.initialize(timestamp)#012  File "/home/hugo/swift/trunk/swift/common/db.py", line 178, in initialize#012    mkdirs(self.db_dir)#012  File "/home/hugo/swift/trunk/swift/common/utils.py", line 170, in mkdirs#012    os.makedirs(path)#012  File "/usr/lib/python2.6/os.py", line 150, in makedirs#012    makedirs(head, mode)#012  File "/usr/lib/python2.6/os.py", line 150, in makedirs#012    makedirs(head, mode)#012  File "/usr/lib/python2.6/os.py", line 150, in makedirs#012    makedirs(head, mode)#012  File "/usr/lib/python2.6/os.py", line 157, in makedirs#012    mkdir(name, mode)#012OSError: [Errno 13] Permission denied: '/srv/1/node/sdb1/accounts'

Nov  7 04:51:31 openstack account-server 127.0.0.1 - - [06/Nov/2010:20:51:31 +0000] "PUT /sdb1/52482/AUTH_14f77faee139431896897f42f0249dba" 500 851 "tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e" "-" "-" 0.0009 ""

Nov  7 04:51:31 openstack account-server ERROR __call__ error with PUT /sdb4/52482/AUTH_14f77faee139431896897f42f0249dba transaction tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e: #012Traceback (most recent call last):#012  File "/home/hugo/swift/trunk/swift/account/server.py", line 304, in __call__#012    res = getattr(self, req.method)(req)#012  File "/home/hugo/swift/trunk/swift/account/server.py", line 107, in PUT#012    broker.initialize(timestamp)#012  File "/home/hugo/swift/trunk/swift/common/db.py", line 178, in initialize#012    mkdirs(self.db_dir)#012  File "/home/hugo/swift/trunk/swift/common/utils.py", line 170, in mkdirs#012    os.makedirs(path)#012  File "/usr/lib/python2.6/os.py", line 150, in makedirs#012    makedirs(head, mode)#012  File "/usr/lib/python2.6/os.py", line 150, in makedirs#012    makedirs(head, mode)#012  File "/usr/lib/python2.6/os.py", line 150, in makedirs#012    makedirs(head, mode)#012  File "/usr/lib/python2.6/os.py", line 157, in makedirs#012    mkdir(name, mode)#012OSError: [Errno 13] Permission denied: '/srv/4/node/sdb4/accounts'

Nov  7 04:51:31 openstack account-server 127.0.0.1 - - [06/Nov/2010:20:51:31 +0000] "PUT /sdb4/52482/AUTH_14f77faee139431896897f42f0249dba" 500 851 "tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e" "-" "-" 0.0009 ""

Nov  7 04:51:31 openstack proxy-server Account PUT returning 503 for [503, 503, 503], transaction tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e

Nov  7 04:51:31 openstack proxy-server - 127.0.0.1 06/Nov/2010/20/51/31 PUT /v1/AUTH_14f77faee139431896897f42f0249dba HTTP/1.0 503 - - .super_admin%3A.single_use%2CAUTH_tk035256ff2d9f4fc6ad9dc4eda4b79149 - - - tx3b74daff-65f1-4f7f-abb5-8650b45b4c3e - 0.0110

Nov  7 04:51:31 openstack auth-server ERROR attempting to create account http://127.0.0.1:8080/v1/AUTH_14f77faee139431896897f42f0249dba: 503 Internal Server Error

Nov  7 04:51:31 openstack auth-server FAILED create_user('test', 'tester', _, True, False) [0.13]

Nov  7 04:51:31 openstack auth-server 127.0.0.1 - - [06/Nov/2010:20:51:31 +0000] "PUT /account/test/tester HTTP/1.0" 503 - "-" "-" - - - - - - - - - "-" "127.0.0.1" "-" 0.1309

=============================================================

Can you help with this problem?

Provide an answer of your own, or ask Hugo Kou for more information if necessary.

To post a message you must log in.

Ask a question

Edit question

Subscribers