Not able to start SAIO: cannot bind to ports

Asked by Chris Almond

I have carefully followed the all-in-one instructions here for setting up a development swift VM: http://swift.openstack.org/development_saio.html

Everything is fine until I reach step #8 in this section: http://swift.openstack.org/development_saio.html#setting-up-scripts-for-running-swift

When I run startmain I get the following output:

saio@swift-saio-0:~$ swift-init main start
WARNING: Unable to increase file descriptor limit. Running as non-root?
Starting proxy-server...(/etc/swift/proxy-server.conf)
Starting container-server...(/etc/swift/container-server/b.conf)
Starting container-server...(/etc/swift/container-server/c.conf)
Starting container-server...(/etc/swift/container-server/d.conf)
Starting container-server...(/etc/swift/container-server/e.conf)
Starting account-server...(/etc/swift/account-server/b.conf)
Starting account-server...(/etc/swift/account-server/c.conf)
Starting account-server...(/etc/swift/account-server/d.conf)
Starting account-server...(/etc/swift/account-server/e.conf)
Starting object-server...(/etc/swift/object-server/b.conf)
Starting object-server...(/etc/swift/object-server/c.conf)
Starting object-server...(/etc/swift/object-server/d.conf)
Starting object-server...(/etc/swift/object-server/e.conf)

Traceback (most recent call last):
  File "/usr/local/bin/swift-container-server", line 7, in <module>
    execfile(__file__)
  File "/home/saio/swift/bin/swift-container-server", line 22, in <module>
    run_wsgi(conf_file, 'container-server', default_port=6001, **options)
  File "/home/saio/swift/swift/common/wsgi.py", line 128, in run_wsgi
    sock = get_socket(conf, default_port=kwargs.get('default_port', 8080))
  File "/home/saio/swift/swift/common/wsgi.py", line 88, in get_socket
    bind_addr)
Exception: Could not bind to 0.0.0.0:6011 after trying for 30 seconds

(The above traceback stanza repeats two more times, then this...)

Traceback (most recent call last):
  File "/usr/local/bin/swift-account-server", line 7, in <module>
    execfile(__file__)
  File "/home/saio/swift/bin/swift-account-server", line 22, in <module>
    run_wsgi(conf_file, 'account-server', default_port=6002, **options)
  File "/home/saio/swift/swift/common/wsgi.py", line 128, in run_wsgi
    sock = get_socket(conf, default_port=kwargs.get('default_port', 8080))
  File "/home/saio/swift/swift/common/wsgi.py", line 88, in get_socket
    bind_addr)
Exception: Could not bind to 0.0.0.0:6012 after trying for 30 seconds

(...two more times, then this...)

Traceback (most recent call last):
  File "/usr/local/bin/swift-object-server", line 7, in <module>
    execfile(__file__)
  File "/home/saio/swift/bin/swift-object-server", line 22, in <module>
    run_wsgi(conf_file, 'object-server', default_port=6000, **options)
  File "/home/saio/swift/swift/common/wsgi.py", line 128, in run_wsgi
    sock = get_socket(conf, default_port=kwargs.get('default_port', 8080))
  File "/home/saio/swift/swift/common/wsgi.py", line 88, in get_socket
    bind_addr)
Exception: Could not bind to 0.0.0.0:6010 after trying for 30 seconds

(...repeats two more times).

Note:
before I run startmain, output of `sudo netstat -ntlp` looks like this:

Active Internet connections (only servers)
   Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
   tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 864/sshd
   tcp 0 0 127.0.0.1:873 0.0.0.0:* LISTEN 7915/rsync
   tcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN 6811/memcached
   tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 938/apache2
   tcp6 0 0 :::22 :::* LISTEN 864/sshd

After running startmain it looks like this:

   Active Internet connections (only servers)
   Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
   tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 864/sshd
   tcp 0 0 0.0.0.0:6010 0.0.0.0:* LISTEN 10622/python
   tcp 0 0 0.0.0.0:6011 0.0.0.0:* LISTEN 10614/python
   tcp 0 0 0.0.0.0:6012 0.0.0.0:* LISTEN 10616/python
   tcp 0 0 127.0.0.1:873 0.0.0.0:* LISTEN 7915/rsync
   tcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN 6811/memcached
   tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 10611/python
   tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 938/apache2
   tcp6 0 0 :::22 :::* LISTEN 864/sshd

I tried adding "bind_ip = 127.0.0.1" into the DEFAULT stanza in /etc/swift/proxy-server.conf and each of the .conf files inside /etc/swift/account-server, /etc/swift/object-server and /etc/swift/container-server. Same failure mode appears after stopping/restarting swift main.

BTW – when I run `swift-init rest start` I get no errors in stdout.

Pressing on in the procedure, I find that the test in step #9 seems to work ok:

saio@swift-saio-0:~$ curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://127.0.0.1:8080/auth/v1.0
* About to connect() to 127.0.0.1 port 8080 (#0)
* Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /auth/v1.0 HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: 127.0.0.1:8080
> Accept: */*
> X-Storage-User: test:tester
> X-Storage-Pass: testing
>
< HTTP/1.1 200 OK
< X-Storage-Url: http://127.0.0.1:8080/v1/AUTH_test
< X-Storage-Token: AUTH_tk89a9393d9ef4402cbd7602a43bcbae26
< X-Auth-Token: AUTH_tk89a9393d9ef4402cbd7602a43bcbae26
< Content-Length: 0
< Date: Thu, 21 Jun 2012 15:54:03 GMT
<
* Connection #0 to host 127.0.0.1 left intact
* Closing connection #0

But the output from the test in step #10 is not as promising:

saio@swift-saio-0:~$ curl -v -H 'X-Auth-Token: AUTH_tk89a9393d9ef4402cbd7602a43bcbae26' http://127.0.0.1:8080/v1/AUTH_test
* About to connect() to 127.0.0.1 port 8080 (#0)
* Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /v1/AUTH_test HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: 127.0.0.1:8080
> Accept: */*
> X-Auth-Token: AUTH_tk89a9393d9ef4402cbd7602a43bcbae26
>
< HTTP/1.1 500 Internal Server Error
< Content-Length: 228
< Content-Type: text/html; charset=UTF-8
< Date: Thu, 21 Jun 2012 15:57:18 GMT
<
<html>
 <head>
  <title>500 Internal Server Error</title>
 </head>
 <body>
  <h1>500 Internal Server Error</h1>
  The server has either erred or is incapable of performing the requested operation.<br /><br />
 </body>
* Connection #0 to host 127.0.0.1 left intact
* Closing connection #0

I get no errors in the output of the "Check that Swift works" test in step #11:

saio@swift-saio-0:~$ swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat
   Account: AUTH_test
Containers: 0
   Objects: 0
     Bytes: 0
Accept-Ranges: bytes
X-Timestamp: 1340294238.20513

The function test (steps 12-13) throws a ton of errors.

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Solved by:
Chris Almond
Solved:
Last query:
Last reply:
Revision history for this message
Chris Almond (kcalmond) said :
#1

The "Could not bind..." errors were caused by multiple .conf files for the account, container and object services using the same bind_port value. My SAIO config is slightly different. I used 4 different physical disk devices ( /dev/sd[b-e] ) instead of a single disk device partitioned to behave like four ( /dev/sdb[1-4] ) like the SAIO procedure documents. For this to work you need to use a unique bind_port value for each of the account, container and object services using each of the emulated SAIO nodes. Swift started up fine after I set up bind_port values like this:

saio@swift-saio-0:~$ grep bind /etc/swift/account-server/*.conf
/etc/swift/account-server/b.conf:bind_port = 6012
/etc/swift/account-server/c.conf:bind_port = 6013
/etc/swift/account-server/d.conf:bind_port = 6014
/etc/swift/account-server/e.conf:bind_port = 6015
saio@swift-saio-0:~$ grep bind /etc/swift/container-server/*.conf
/etc/swift/container-server/b.conf:bind_port = 6016
/etc/swift/container-server/c.conf:bind_port = 6017
/etc/swift/container-server/d.conf:bind_port = 6018
/etc/swift/container-server/e.conf:bind_port = 6019
saio@swift-saio-0:~$ grep bind /etc/swift/object-server/*.conf
/etc/swift/object-server/b.conf:bind_port = 6020
/etc/swift/object-server/c.conf:bind_port = 6021
/etc/swift/object-server/d.conf:bind_port = 6022
/etc/swift/object-server/e.conf:bind_port = 6023

A big thank you to Jonathan Bryce (~jbryce) for ID'ing the problem!

Revision history for this message
ouyangmy (373467963-t) said :
#2

@Chris Almond (kcalmond)
hi, i'm sorry to interupt u, but i've confronted the same problem as u describbed above. You think it is the bind_port value issue, and after u set up the new bind_port values, it works well. the question is that i don't understand :
1. what the difference between the former bind_port values and the later, which is also the ones listed aforementioned?
2. besides the new bind_port values being changed, what files else do you have made a slight change???

thank you for your answer. i am so eager to hear from u.

Revision history for this message
ouyangmy (373467963-t) said :
#3

i have solved the problem. thanks anyway.