503 Service Unavailable

When adding a new account/user, the auth server will call out to the proxy server based on the default_cluster_url config file value (defaults to http://127.0.0.1:8080/v1 I think). Might double-check that value in /etc/swift/auth-server.conf and that you're running the proxy server on that same port (bind_port value, I think that defaults to port 80).

Thanks for the response! I've gone back and checked the bound ports for each service. They are:

account-server.conf: 6003
container-server.conf: 6002
object-server.conf: 6001
and for the important ones:
auth-server.conf: 80
 - default_cluster_url = http://127.0.0.1:8080/v1
proxy-server.conf: 8080

This has gotten me past the 503 error (thanks) but has created another one. Now when issuing the same command (swift-auth-add-user -K cumulo -a test tester testing) the terminal hangs and I'm forced after a minute or so to interrupt (command-c) it. The terminal spits out a traceback:

Traceback (most recent call last):
File "/usr/bin/swift-auth-add-user", line 69, in <module>
  resp = conn.getresponse()
File "/usr/lib/pymodules/python2.6/swift/common/bufferedhttp.py", line 101, in getresponse
  response = HTTPConnection.getresponse(self)
File "/usr/lib/python2.6/httplib.py", line 986, in getresponse
  response.begin()
File "/usr/lib/python2.6/httplib.py", line 391, in begin
  version, status, reason = self._read_status()
File "/usr/lib/python2.6/httplib.py", line 349, in _read_status
  line = self.fp.readline()
File "/usr/lib/python2.6/socket.py", line 406, in readline
  data = self._sock.recv(self._rbufsize)
File "/usr/lib/pymodules/python2.6/eventlet/greenio.py", line 228, in recv
  timeout_exc=socket.timeout("timed out"))
...

And this is recorded in the system log:

STDOUT: Traceback (most recent call last):
File "/usr/lib/pymodules/python2.6/eventlet/wsgi.py", line 335, in handle_one_response
  result = self.application(self.environ, start_response)
File "/usr/lib/pymodules/python2.6/swift/common/middleware/healthcheck.py", line 38, in __call__
  return self.app(env, start_response)
File "/usr/lib/pymodules/python2.6/swift/common/middleware/memcache.py", line 31, in __call__
  return self.app(env, start_response)
File "/usr/lib/pymodules/python2.6/swift/common/middleware/auth.py", line 61, in __call__
  self.get_groups(token, memcache_client=cache_from_env(env))
File "/usr/lib/pymodules/python2.6/swift/common/middleware/auth.py", line 131, in get_groups
  '/token/%s' % token, ssl=self.ssl)
File "/usr/lib/pymodules/python2.6/swift/common/bufferedhttp.py", line 169, in http_connect_raw
  conn.endheaders()
File "/usr/lib/python2.6/httplib.py", line 904, in endheaders
  self._send_output()#012 File "/usr/lib/python2.6/httplib.py", line 776, in _send_output
  self.send(msg)#012 File "/usr/lib/python2.6/httplib.py", line 735, in send
  self.connect()
File "/usr/lib/pymodules/python2.6/swift/common/bufferedhttp.py", line 80, in connect
  return HTTPConnection.connect(self)#012 File "/usr/lib/python2.6/httplib.py", line 716, in connect
  self.timeout)
File "/usr/lib/pymodules/python2.6/eventlet/green/socket.py", line 92, in create_connection
    raise error, msg
error: [Errno 111] ECONNREFUSED

I will admit this could have been caused by my endless tinkering to settings and the such so I'm tempted to just go back and reinstall swift and ubuntu (without xubuntu-desktop). Thanks again.

Ah, okay. That error is now the proxy server trying to connect back to the auth server for token validation. If you're auth server is configured to run on port 80, you'll need to set port=80 in your proxy-server.conf under the [filter:auth] section. It defaults to 11000, which I think is what the auth server defaults to.

Thanks for setting me straight on this. This has definitely helped me go groom all my config files. I'm going to post one more problem at the expense of looking like a total noob, but the 503 error has re-spawn in a different form. The syslog now returns:

auth-server validate_token('AUTH_tk93fb6c4577bd4432ad43666a75044b29', _, _) = (86399.916866064072, '.super_admin', '.single_use', '.reseller_admin') [0.05]
auth-server 127.0.0.1 - - [01/Oct/2010:17:04:10 +0000] "GET /token/AUTH_tk93fb6c4577bd4432ad43666a75044b29 HTTP/1.0" 204 - "-" "-" - - - - - - - - - "-" "127.0.0.1" "-" 0.0488
account-server 127.0.0.1 - - [01/Oct/2010:17:04:10 +0000] "PUT /sdb1/80201/AUTH_f4748dba45464d2f9730e1c81174d887" 507 - "tx71ec71de-ff88-4bc1-a9df-593a72d257fc" "-" "-" 0.0012 ""
proxy-server Account PUT returning 503 for [503], transaction tx71ec71de-ff88-4bc1-a9df-593a72d257fc
proxy-server - 127.0.0.1 01/Oct/2010/17/04/10 PUT /v1/AUTH_f4748dba45464d2f9730e1c81174d887 HTTP/1.0 503 - - .super_admin%3A.single_use%2CAUTH_tk93fb6c4577bd4432ad43666a75044b29 - - - tx71ec71de-ff88-4bc1-a9df-593a72d257fc - 0.0085
auth-server ERROR attempting to create account http://127.0.0.1:80/v1/AUTH_f4748dba45464d2f9730e1c81174d887: 503 Internal Server Error
auth-server FAILED create_user('test', 'tester', _, True, False) [0.15]
auth-server 127.0.0.1 - - [01/Oct/2010:17:04:10 +0000] "PUT /account/test/tester HTTP/1.0" 503 - "-" "-" - - - - - - - - - "-" "127.0.0.1" "-" 0.1477

It looks like the error is sent by the account-server to the proxy server in response to the PUT. Shed any light on this one? Thanks again. You guys rock.

Ah, the 507 returned by the account server in there indicates the account server thinks the device is not mounted.

If you're using "pretend" devices as with the standard SAIO, you'll want to ensure mount_check = false is in your account-server.conf file (or your account-server/*.conf files) under the [DEFAULT] section(s).

If that's not set right in your account server configurations, you'll want to double check your container and object server configurations too.

Thanks for that. You're right in that I was using steps from SAIO for virtual devices. Turning mount_check=false has seemed to have fixed things.

The thing is, I actually am mounting a separate device (hard drive) whose path I am specifying (/dev/sdb1 is mounted at /mnt/sdb1). However when I had mount_check=true and devices=/dev/sdb1 the services were telling me the device was not mounted. Why was that?

And FYI I'm adding devices to the rings like this:
...
swift-ring-builder object.builder add z1-127.0.0.1:6000/sdb1 1
...

Thanks again

Ah, devices= is supposed to be set to the directory where a set of mount points can be found. The system is set up to allow many drives per server. For instance, we often have systems with devices=/srv/node and if you did an ls /srv/node you might see:

sda sdc sde sdg sdi sdk sdm sdo sdq sds sdu sdw
sdb sdd sdf sdh sdj sdl sdn sdp sdr sdt sdv sdx

Each one of those being a separate mount point and therefore device.

I'm pretty sure you don't want devices=/dev/sdb1 but you probably don't want devices=/mnt either. By convention, you might want to create an /srv/node directory and mount sdb1 at /srv/node/sdb1 and set devices=/srv/node

With devices=/dev/sdb1 I'm not sure what might be happening on your system. It might actually be trying to write files to /dev/sdb1/sdb1/* If you're running as root, it might actually be succeeding! Best to double check things aren't wonky in that respect; you may need to do a bit of cleanup.