1.3- Account creating fails: KeyError: 'x-container-meta-account-id'

If I remember correctly, this usually indicates a problem when the account was created. You may want to try creating the account again.

The account was created as part of swauth-prep.... so, presumably it should have been created correctly?

Folks from swauth, any idea on where this header is being set?

Ah sorry, I meant try doing a swauth-add-account before doing the swauth-add-user. I'm just guessing that maybe swauth-add-user thinks the account exists, but it only got half created or something.

To answer your other quest, the only place that header is set is in "handle_put_account" which is a PUT /auth/v2/<account> call (either by swauth-add-account or swauth-add-user).

Guys,

I am having a similar problem here. However I can't seem to execute any swift command successfully. I have 1 proxy, 3 storage nodes that use two disks each. When I execute:

swauth-add-user -A https://localhost/auth/ -K <swauthkey> test_a test_u testpass

I get from the CLI:

Account creation failed: 500 Server Error
User creation failed: 500 Server Error

and the storage PROXY node says:

Jun 30 05:46:20 localhost proxy-server - - 30/Jun/2011/09/46/20 GET /v1/AUTH_.auth/.token_3/AUTH_itkcf4cadae8c794da6ba04875f6139c1b3 HTTP/1.0 404 - Swauth - - - - tx6e77e7a1-6aa3-4a90-ac82-74a98379da37 - 0.0097
Jun 30 05:46:20 localhost proxy-server ERROR: Exception while trying to communicate with https://192.168.1.196:None//v1/AUTH_06c778a0-9008-4f9b-beec-69d513415db8 (txn: txcc521abe-ce0a-4934-a74b-243b38c4a840)
Jun 30 05:46:20 localhost proxy-server STDOUT: EXCEPTION IN handle: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/swift/common/middleware/swauth.py", line 321, in handle return self.handle_request(req)(env, start_response) File "/usr/lib/python2.6/site-packages/swift/common/middleware/swauth.py", line 384, in handle_request req.response = handler(req) File "/usr/lib/python2.6/site-packages/swift/common/middleware/swauth.py", line 637, in handle_put_account 'cluster: %s %s %s' % (path, resp.status, resp.reason)) Exception: Could not create account on the Swift cluster: /v1/AUTH_06c778a0-9008-4f9b-beec-69d513415db8 401 Unauthorized : {'SCRIPT_NAME': '/auth/v2/test_a', 'webob.adhoc_attrs': {'start_time': 1309427180.0651929, 'bytes_transferred': '-', 'client_disconnect': False}, 'REQUEST_METHOD': 'PUT', 'PATH_INFO': '', 'SERVER_PROTOCOL': 'HTTP/1.0', 'eventlet.posthooks': [(<bound method Swauth.posthooklogger of <swift.common.middleware.swauth.Swauth object at 0x964c4ac>>, (<Request at
Jun 30 05:46:20 localhost proxy-server - - 30/Jun/2011/09/46/20 HEAD /v1/AUTH_.auth/test_a HTTP/1.0 204 - Swauth - - - - tx79257e6a-cbe2-4360-b44d-43681189ac4f - 0.1308
Jun 30 05:46:20 localhost proxy-server STDOUT: EXCEPTION IN handle: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/swift/common/middleware/swauth.py", line 321, in handle return self.handle_request(req)(env, start_response) File "/usr/lib/python2.6/site-packages/swift/common/middleware/swauth.py", line 384, in handle_request req.response = handler(req) File "/usr/lib/python2.6/site-packages/swift/common/middleware/swauth.py", line 902, in handle_put_user resp.headers['x-container-meta-account-id']} File "build/bdist.linux-i686/egg/webob/headerdict.py", line 28, in __getitem__ raise KeyError(key) KeyError: 'x-container-meta-account-id' : {'SCRIPT_NAME': '/auth/v2/test_a/test_u', 'webob.adhoc_attrs': {'start_time': 1309427180.2822649, 'bytes_transferred': '-', 'client_disconnect': False}, 'REQUEST_METHOD': 'PUT', 'PATH_INFO': '', 'SERVER_PROTOCOL': 'HTTP/1.0', 'eventlet.posthooks': [(<bound method Swauth.posthooklogger of <swift.common.middleware.swauth.Swauth object at 0x964c4ac

One object service says:

Jun 30 05:49:50 os-vpx-26-1F-AC-BD-C7-FF container-replicator Beginning replication run
Jun 30 05:49:50 os-vpx-26-1F-AC-BD-C7-FF container-replicator Replication run OVER
Jun 30 05:49:50 os-vpx-26-1F-AC-BD-C7-FF container-replicator Attempted to replicate 35 dbs in 0.16845 seconds (207.77681/s)
Jun 30 05:49:50 os-vpx-26-1F-AC-BD-C7-FF container-replicator Removed 0 dbs
Jun 30 05:49:50 os-vpx-26-1F-AC-BD-C7-FF container-replicator 70 successes, 0 failures
Jun 30 05:49:50 os-vpx-26-1F-AC-BD-C7-FF container-replicator no_change:70 ts_repl:0 diff:0 rsync:0 hashmatch:0 empty:0
Jun 30 05:49:50 os-vpx-26-1F-AC-BD-C7-FF account-server 192.168.1.196 - - [30/Jun/2011:09:49:50 +0000] "HEAD /xvdg/255100/AUTH_.auth" 204 - "tx7f1d3b40-7663-4c8d-a1f8-eebe8fe61f34" "-" "-" 0.0025 ""
Jun 30 05:49:50 os-vpx-26-1F-AC-BD-C7-FF object-server 192.168.1.196 - - [30/Jun/2011:09:49:50 +0000] "GET /xvdf/149103/AUTH_.auth/.token_3/AUTH_itkcf4cadae8c794da6ba04875f6139c1b3" 404 - "-" "txaf9d392b-0972-405c-8a41-3744378a739b" "Swauth" 0.0003
Jun 30 05:49:50 os-vpx-26-1F-AC-BD-C7-FF account-server 192.168.1.196 - - [30/Jun/2011:09:49:50 +0000] "HEAD /xvdg/255100/AUTH_.auth" 204 - "txf27f343b-ee6d-4977-9ede-0c1bbd52f01d" "-" "-" 0.0024 ""
Jun 30 05:49:51 os-vpx-26-1F-AC-BD-C7-FF object-auditor Begin object audit "forever" mode (ALL)
Jun 30 05:49:51 os-vpx-26-1F-AC-BD-C7-FF object-auditor Object audit (ALL) "forever" mode completed: 0.00s. Total files/sec: 0.00 , Total bytes/sec: 0.00
Jun 30 05:49:51 os-vpx-26-1F-AC-BD-C7-FF object-auditor Begin object audit "forever" mode (ZBF)
Jun 30 05:49:51 os-vpx-26-1F-AC-BD-C7-FF object-auditor Object audit (ZBF) "forever" mode completed: 0.00s. Total files/sec: 0.00 , Total bytes/sec: 0.00

When I do swauth-list, I get:

{"accounts": [{"name": "test_a"}]}

But if I try to get info on the account I get 'List failed: 404 Not found'

It seems I cannot do anything, swauth-delete-account, swauth-delete-user, etc...I keep getting 500 Internal Error.

This is the proxy output for the account deletion

Jun 30 05:52:52 localhost proxy-server - - 30/Jun/2011/09/52/52 GET /v1/AUTH_.auth/test_a%3Fformat%3Djson%26marker%3D HTTP/1.0 200 - Swauth - - - - txd36cd55c-f844-4141-965d-ba3eef7c7273 - 0.0117
Jun 30 05:52:53 localhost proxy-server STDOUT: EXCEPTION IN handle: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/swift/common/middleware/swauth.py", line 321, in handle return self.handle_request(req)(env, start_response) File "/usr/lib/python2.6/site-packages/swift/common/middleware/swauth.py", line 384, in handle_request req.response = handler(req) File "/usr/lib/python2.6/site-packages/swift/common/middleware/swauth.py", line 699, in handle_delete_account account_id = resp.headers['x-container-meta-account-id'] File "build/bdist.linux-i686/egg/webob/headerdict.py", line 28, in __getitem__ raise KeyError(key) KeyError: 'x-container-meta-account-id' : {'SCRIPT_NAME': '/auth/v2/test_a', 'webob.adhoc_attrs': {'start_time': 1309427572.959549, 'bytes_transferred': '-', 'client_disconnect': False}, 'REQUEST_METHOD': 'DELETE', 'PATH_INFO': '', 'SERVER_PROTOCOL': 'HTTP/1.0', 'eventlet.posthooks': [(<bound method Swauth.posthooklogger of <swift.common.middleware.swauth.Swauth object

this is the proxy output for the user deletion:

Jun 30 05:53:28 localhost proxy-server - - 30/Jun/2011/09/53/28 HEAD /v1/AUTH_.auth/test_a/test_u HTTP/1.0 404 - Swauth - - - - tx5cbf6cbb-6749-44fd-89d4-96760e4d92cf - 0.0137
Jun 30 05:53:28 localhost proxy-server - 192.168.1.195 30/Jun/2011/09/53/28 DELETE /auth/v2/test_a/test_u HTTP/1.0 404 - - .super_admin - - - tx5cbf6cbb-6749-44fd-89d4-96760e4d92cf - 0.0151

So, I am stuck. I there are a bunch of 404 error code that I see, but I don't understand what is causing them. Is there a chance there is something wrong with the disks? They look correctly mounted and accessible.

Any clue, on where to look to further troubleshoot this problem?

I am running Swift 1.3

Many thanks,
Armando

In my case, it ended up that the swift cluster wasn't 100% configured correctly (forget the details, but the ports/services/ip addresses in the ring and in the actual deployment didn't agree).
So a) check that.

another approach that helped was to remove swauth from the proxy config, and exercise the cluster using curl (see http://docs.openstack.org/cactus/openstack-object-storage/developer/content/ch_object-storage-dev-troubleshooting.html )

Once I was able to reliably create accounts containers and files, I turned authentication back on, and things seemed much happier.

To remove authentication, edit proxy.conf and replace the line:
pipeline = healthcheck cache swauth proxy-server

with:
pipeline = healthcheck cache proxy-server

hope this helps.

Hi there, thanks for your input. I tried to upload a storage object after having disabled swauth. I played with the system a bit and everything seemed fine. Then I put swauth back but still same result...I couldn't create/deleet account etc.

Is there anything else I could look into?

Many thanks!

Sorry for resurrecting a very old issue, but I experienced precisely the same issue on a fresh deployment of Swift 1.7.4 (where tempauth and upload/download operations were working prior to plugging in swauth).

The initial requests to create the account were failing as the request body was not being set due to a change in webob 1.2.

I was able to resolve by downgrading webob from 1.2.x down to 1.1.1 via `pip install -Iv webob==1.1.1`.

Good news: commit 5e3e9a8, just merged today, removes Swift's dependency on webob. It's not in any released version yet, but an end to the pain is on the way.