OpenStack Object Storage (swift)

[glance.api.v1.images] Error uploading image: [Errno 111] ECONNREFUSED

Asked by Diego Lalo

Hi guys I'm writing to you because I'm having a problem using glance with swift backed storage, the machine that is running the glance-api and glance-registry services is has ubuntu 10.10 64Bits and the machine that is running all swift services is also ubuntu 10.10 64Bits, here is the trace of glance-api.log:

2011-07-19 17:41:43 DEBUG [glance.api.middleware.version_negotiation] Processing request: POST /v1/images Accept:
2011-07-19 17:41:43 DEBUG [glance.api.middleware.version_negotiation] Matched versioned URI. Version: 1.0
2011-07-19 17:41:43 DEBUG [routes.middleware] Matched POST /images
2011-07-19 17:41:43 DEBUG [routes.middleware] Route path: '/images', defaults: {'action': u'create', 'controller': <glance.common.wsgi.Resource object at 0x31536d0>}
2011-07-19 17:41:43 DEBUG [routes.middleware] Match dict: {'action': u'create', 'controller': <glance.common.wsgi.Resource object at 0x31536d0>}
2011-07-19 17:41:43 DEBUG [glance.registry] Adding image metadata...
2011-07-19 17:41:43 DEBUG [glance.registry] container_format: ami
2011-07-19 17:41:43 DEBUG [glance.registry] disk_format: ami
2011-07-19 17:41:43 DEBUG [glance.registry] is_public: True
2011-07-19 17:41:43 DEBUG [glance.registry] name: Centos 5.5 32Bits Image
2011-07-19 17:41:43 DEBUG [glance.registry] size: 0
2011-07-19 17:41:43 DEBUG [glance.registry] status: queued
2011-07-19 17:41:43 DEBUG [glance.registry] Returned image metadata from call to RegistryClient.add_image():
2011-07-19 17:41:43 DEBUG [glance.registry] checksum: None
2011-07-19 17:41:43 DEBUG [glance.registry] container_format: ami
2011-07-19 17:41:43 DEBUG [glance.registry] created_at: 2011-07-19T16:41:43.453634
2011-07-19 17:41:43 DEBUG [glance.registry] deleted: False
2011-07-19 17:41:43 DEBUG [glance.registry] deleted_at: None
2011-07-19 17:41:43 DEBUG [glance.registry] disk_format: ami
2011-07-19 17:41:43 DEBUG [glance.registry] id: 44
2011-07-19 17:41:43 DEBUG [glance.registry] is_public: True
2011-07-19 17:41:43 DEBUG [glance.registry] location: None
2011-07-19 17:41:43 DEBUG [glance.registry] name: Centos 5.5 32Bits Image
2011-07-19 17:41:43 DEBUG [glance.registry] size: 0
2011-07-19 17:41:43 DEBUG [glance.registry] status: queued
2011-07-19 17:41:43 DEBUG [glance.registry] updated_at: None
2011-07-19 17:41:43 DEBUG [glance.api.v1.images] Setting image 44 to status 'saving'
2011-07-19 17:41:43 DEBUG [glance.registry] Updating image metadata for image 44...
2011-07-19 17:41:43 DEBUG [glance.registry] status: saving
2011-07-19 17:41:43 DEBUG [glance.registry] Returned image metadata from call to RegistryClient.update_image():
2011-07-19 17:41:43 DEBUG [glance.registry] checksum: None
2011-07-19 17:41:43 DEBUG [glance.registry] container_format: ami
2011-07-19 17:41:43 DEBUG [glance.registry] created_at: 2011-07-19T16:41:43.453634
2011-07-19 17:41:43 DEBUG [glance.registry] deleted: False
2011-07-19 17:41:43 DEBUG [glance.registry] deleted_at: None
2011-07-19 17:41:43 DEBUG [glance.registry] disk_format: ami
2011-07-19 17:41:43 DEBUG [glance.registry] id: 44
2011-07-19 17:41:43 DEBUG [glance.registry] is_public: True
2011-07-19 17:41:43 DEBUG [glance.registry] location: None
2011-07-19 17:41:43 DEBUG [glance.registry] name: Centos 5.5 32Bits Image
2011-07-19 17:41:43 DEBUG [glance.registry] size: 0
2011-07-19 17:41:43 DEBUG [glance.registry] status: saving
2011-07-19 17:41:43 DEBUG [glance.registry] updated_at: 2011-07-19T16:41:43.521586
2011-07-19 17:41:43 DEBUG [glance.api.v1.images] Uploading image data for image 44 to swift store
2011-07-19 17:41:43 DEBUG [glance.store.swift] Adding image object to Swift using (auth_address=10.0.20.226:8080/auth/v1.0, user=test:tester, key=testing)
2011-07-19 17:42:14 ERROR [glance.api.v1.images] Error uploading image: [Errno 111] ECONNREFUSED
2011-07-19 17:42:14 DEBUG [glance.registry] Updating image metadata for image 44...
2011-07-19 17:42:14 DEBUG [glance.registry] status: killed
2011-07-19 17:42:14 DEBUG [glance.registry] Returned image metadata from call to RegistryClient.update_image():
2011-07-19 17:42:14 DEBUG [glance.registry] checksum: None
2011-07-19 17:42:14 DEBUG [glance.registry] container_format: ami
2011-07-19 17:42:14 DEBUG [glance.registry] created_at: 2011-07-19T16:41:43.453634
2011-07-19 17:42:14 DEBUG [glance.registry] deleted: False
2011-07-19 17:42:14 DEBUG [glance.registry] deleted_at: None
2011-07-19 17:42:14 DEBUG [glance.registry] disk_format: ami
2011-07-19 17:42:14 DEBUG [glance.registry] id: 44
2011-07-19 17:42:14 DEBUG [glance.registry] is_public: True
2011-07-19 17:42:14 DEBUG [glance.registry] location: None
2011-07-19 17:42:14 DEBUG [glance.registry] name: Centos 5.5 32Bits Image
2011-07-19 17:42:14 DEBUG [glance.registry] size: 0
2011-07-19 17:42:14 DEBUG [glance.registry] status: killed
2011-07-19 17:42:14 DEBUG [glance.registry] updated_at: 2011-07-19T16:42:14.624355
2011-07-19 17:42:35 DEBUG [eventlet.wsgi.server] 10.0.20.30 - - [19/Jul/2011 17:42:35] "POST /v1/images HTTP/1.1" 400 293 52.391712

Here is the /etc/glance/glance-api.conf file:

[DEFAULT]
# Show more verbose log output (sets INFO log level output)
verbose = True

# Show debugging output in logs (sets DEBUG log level output)
debug = True

# Which backend store should Glance use by default is not specified
# in a request to add a new image to Glance? Default: 'file'
# Available choices are 'file', 'swift', and 's3'
default_store = swift

# Address to bind the API server
bind_host = 10.0.20.243

# Port the bind the API server to
bind_port = 9292

# Address to find the registry server
registry_host = 10.0.20.243

# Port the registry server is listening on
registry_port = 9191

# Log to this file. Make sure you do not set the same log
# file for both the API and registry servers!
log_file = /var/log/glance/api.log

# ============ Filesystem Store Options ========================

# Directory that the Filesystem backend store
# writes image data to
filesystem_store_datadir = /var/lib/glance/images

# ============ Swift Store Options =============================

# Address where the Swift authentication service lives
swift_store_auth_address = 10.0.20.226:8080/auth/v1.0

# User to authenticate against the Swift authentication service
swift_store_user = test:tester

# Auth key for the user authenticating against the
# Swift authentication service
swift_store_key = testing

# Container within the account that the account should use
# for storing images in Swift
swift_store_container = glance

# Do we create the container if it does not exist?
swift_store_create_container_on_put = True

[pipeline:glance-api]
pipeline = versionnegotiation apiv1app

[pipeline:versions]
pipeline = versionsapp

[app:versionsapp]
paste.app_factory = glance.api.versions:app_factory

[app:apiv1app]
paste.app_factory = glance.api.v1:app_factory

[filter:versionnegotiation]
paste.filter_factory = glance.api.middleware.version_negotiation:filter_factory

And here is the configuration of /etc/swift/proxy-server.conf

[DEFAULT]
cert_file = /etc/swift/cert.crt
key_file = /etc/swift/cert.key
bind_port = 8080
user = swift
log_facility = LOG_LOCAL1

[pipeline:main]
pipeline = healthcheck cache tempauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true

[filter:tempauth]
use = egg:swift#tempauth
user_admin_admin = admin .admin .reseller_admin
user_test_tester = testing .admin
user_test2_tester2 = testing2 .admin
user_test_tester3 = testing3

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:cache]
use = egg:swift#memcache/swift/proxy-server.conf

This is the version of swift I have installed:

root@swift-in-a-box-1-0rev10:/etc/swift# dpkg -l | grep swift
ii python-swift 1.4.1-0ubuntu0~ppa1~maverick1 A distributed virtual object store (python libraries)
ii swift 1.4.1-0ubuntu0~ppa1~maverick1 A distributed virtual object store (common files)
ii swift-account 1.4.1-0ubuntu0~ppa1~maverick1 The swift account server
ii swift-container 1.4.1-0ubuntu0~ppa1~maverick1 The swift container server
ii swift-doc 1.4.1-0ubuntu0~ppa1~maverick1 The swift documentation
ii swift-object 1.4.1-0ubuntu0~ppa1~maverick1 The swift object server
ii swift-proxy 1.4.1-0ubuntu0~ppa1~maverick1 The swift proxy server

As I said before the swift machine is running all the swift services inside it, I used a partition of 24Gb as storage and I used SAIO instructions to set it up but I used the packages of the diablo release of swift. Can anyone help me with this issue?

Cheers
Diego

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Solved by:
Diego Lalo
Solved:
Last query:
Last reply:
Revision history for this message
clayg (clay-gerrard) said : 		#1

Well, the relevant error from glance-api is:
2011-07-19 17:42:14 ERROR [glance.api.v1.images] Error uploading image: [Errno 111] ECONNREFUSED

os.strerror(111) = 'Connection refused'

so I guess I'd get on the glance box and make sure it can hit the swift auth.

Quick test:

curl -v https://10.0.20.226:8080/auth/v1.0 -H 'x-auth-user: test:tester' -H 'x-auth-key: testing'

If not, try and get that working...

Revision history for this message
Diego Lalo (diego-lalo) said : 		#2

Hi clayg, I executed the command you suggested on the glance machine, I had to add -k option because the self-signed certificate is signed by an untrusted user (me lol!!), here is the output of curl:

root@openstack-in-a-box-1rev87:~# curl -v -k https://10.0.20.226:8080/auth/v1.0 -H 'x-auth-user: test:tester' -H 'x-auth-key: testing'
* About to connect() to 10.0.20.226 port 8080 (#0)
* Trying 10.0.20.226... connected
* Connected to 10.0.20.226 (10.0.20.226) port 8080 (#0)
* successfully set certificate verify locations:
* CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA
* Server certificate:
* subject: C=AR; ST=Buenos Aires; L=Capital Federal; O=UshareSoft; CN=Diego; <email address hidden>
* start date: 2011-07-19 16:11:52 GMT
* expire date: 2011-08-18 16:11:52 GMT
* common name: Diego (does not match '10.0.20.226')
* issuer: C=AR; ST=Buenos Aires; L=Capital Federal; O=UshareSoft; CN=Diego; <email address hidden>
* SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /auth/v1.0 HTTP/1.1
> User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
> Host: 10.0.20.226:8080
> Accept: */*
> x-auth-user: test:tester
> x-auth-key: testing
>
 < HTTP/1.1 200 OK
< X-Storage-Url: https://127.0.0.1:8080/v1/AUTH_test
< X-Storage-Token: AUTH_tkcce28150dfa045c7ba0fdc06fe624f17
< X-Auth-Token: AUTH_tkcce28150dfa045c7ba0fdc06fe624f17
< Content-Length: 0
< Date: Tue, 19 Jul 2011 18:56:58 GMT
<
* Connection #0 to host 10.0.20.226 left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):

Cheers
Diego

Revision history for this message
clayg (clay-gerrard) said : 		#3

oic, the X-Storage-Url is wrong.

after auth when glance turns around and tries to acctually connect to the storage server it's trying to connect to the proxy at https://127.0.0.1:8080/v1/AUTH_test.

Two options:

1) In your proxy-server.conf add 'bind_ip = 10.0.20.226' to the [DEFAULT] section. This will cause swift to only listen on the primary interface and tempauth will just return that ip as the storage url. But you won't be able to connect swift via 'localhost' anymore.

2) In your [filter:tempauth] section add an explicit storage url as a fourth parameter to each user, which will override the default url constructed from the bind_ip.

e.g.
[filter:tempauth]
use = egg:swift#tempauth
user_admin_admin = admin .admin .reseller_admin https://10.0.20.226:8080/v1/AUTH_admin
user_test_tester = testing .admin https://10.0.20.226:8080/v1/AUTH_test
user_test2_tester2 = testing2 .admin https://10.0.20.226:8080/v1/AUTH_test2
user_test_tester3 = testing3 https://10.0.20.226:8080/v1/AUTH_test

Option 2 is a little more work, and probably more error prone... simpler solution is probably to just bind to primary interface. You'll need to restart/reload your proxy server for changes to take effect.

good catch on the -k ;)

Revision history for this message
Diego Lalo (diego-lalo) said : 		#4

Thank you very much Clayg!!!! I tested option 1) and worked fine so the image was uploaded to the swift storage :)

Cheers
Diego