Swift 1.9.0: swauth prep error

Asked by Netmagic Solutions on 2013-07-25

Hi,

We are in the process of implementation of new "object storage" infra.

Infra:
1. Swift version: 1.9.0
2. Swauth : 1.0.9

Changes:
1. We have added each and every HDD present on the node/storage server as individual disk.
2. LABELed each HDD as "HDD001, HDD002.... HDD012 etc" using xfs_admin and used the same label to mount the partitions under "/srv/node/HDD001..." using fstab (to avoid inconsistent partition mounting after reboots)
3. The disks are added into the ring with LABEL names and not the device names
e.g.
swift-ring-builder object.builder add r2z1-XX.XX.XX.1:6010/HDD001 100

Below is the error we are getting when trying to prep the swauth:

Command:
#swauth-prep -A https://<domain>/auth/ -U .super_admin -K swauthkey

Error in proxys syslog:
===============
Jul 25 17:48:51 MUM-Swift-Proxy1 proxy-server STDOUT: EXCEPTION IN handle: Traceback (most recent call last):#012 File "/usr/local/lib/python2.7/dist-packages/swauth-1.0.9.dev-py2.7.egg/swauth/middleware.py", line 454, in handle#012 return self.handle_request(req)(env, start_response)#012 File "/usr/local/lib/python2.7/dist-packages/swauth-1.0.9.dev-py2.7.egg/swauth/middleware.py", line 521, in handle_request#012 req.response = handler(req)#012 File "/usr/local/lib/python2.7/dist-packages/swauth-1.0.9.dev-py2.7.egg/swauth/middleware.py", line 549, in handle_prep#012 (path, resp.status))#012Exception: Could not create the main auth account: /v1/AUTH_.auth 503 Internal Server Error#012: {'SCRIPT_NAME': '/auth/v2', 'REQUEST_METHOD': 'POST', 'PATH_INFO': '/.prep', 'SERVER_PROTOCOL': 'HTTP/1.0', 'HTTP_CONNECTION': 'close', 'REMOTE_PORT': '61806', 'SERVER_NAME': 'XX.XX.XX.12', 'REMOTE_ADDR': 'XX.XX.XX.10', 'eventlet.input': <eventlet.wsgi.Input object at 0x2a6f450>, 'HTTP_X_AUTH_ADMIN_KEY': '<PASSWORD>', 'wsgi.url_scheme': 'http', 'SERVER_PORT': '8080', 'HTTP_X_AUTH_ADMIN_USER': '.super_admin', 'wsgi.input': <eventlet.wsgi.Input object at 0x2a6f450>, 'HTTP_HOST': '<domian>', 'swift.cache': <swift.common.memcached.MemcacheRing object at 0x2d79410>, 'wsgi.multithread': True, 'eventlet.posthooks': [(<bound method Swauth.posthooklogger of <swauth.middleware.Swauth object at 0x2a69410>>, (<swift.common.swob.Request object at 0x2a6fa10>,), {})], 'wsgi.version': (1, 0), 'RAW_PATH_INFO': '/auth/v2/.prep', 'GATEWAY_INTERFACE': 'CGI/1.1', 'wsgi.run_once': False, 'wsgi.errors': <swift.common.utils.LoggerFileObject object at 0x2a61f10>, 'wsgi.multiprocess': False, 'swift.trans_id': 'tx8ddc3854d3d74cb68c2b9-0051f117aa', 'HTTP_X_FORWARDED_FOR': 'XX.XX.XX.12,127.1.0.15', 'CONTENT_TYPE': None, 'HTTP_ACCEPT_ENCODING': 'identity'}
Jul 25 17:48:51 MUM-Swift-Proxy1 proxy-server - - 25/Jul/2013/12/18/51 PUT /v1/AUTH_.auth HTTP/1.0 499 - Swauth - - 118 - tx8ddc3854d3d74cb68c2b9-0051f117aa - 0.9072 SWTH -
===============

Regards,

VIshal

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Solved by:
Netmagic Solutions
Solved:
2013-07-25
Last query:
2013-07-25
Last reply:
2013-07-25
clayg (clay-gerrard) said : #1

I think swauth 1.0.8 is the last stable release of swauth, you might try installing a tagged release; or you could try posing the question on gholt's github page for swauth:

https://github.com/gholt/swauth

But it looks like it's probably something weird with the swift cluster... The proxy-logger is showing the response as 499, which makes me think there's some more error responses from the backend servers that didn't get pasted here.

Looking at the swauth.middleware around 549 it looks like it uses a pre_auth_request, and then calls get_response with the app then checks the status code with out ever reading the body. I wonder if there's something interesting in this 503/499 resp.body?

Samuel Merritt (torgomatic) said : #2

Hint for troubleshooting: the proxy entry has a transaction id in it: "tx8ddc3854d3d74cb68c2b9-0051f117aa". Look for that string across all your logs.

Sorry Guys,

it was due to mis config of separate replication network.

Fixed it and things are working fine now.

Thanks for all the replies :)

Regards,

Vishal

We have started from scratch and reconfigured everything as per below link which worked for us.

http://docs.openstack.org/developer/swift/replication_network.html

Regards,

Vishal