Multi-Zone finally working on ESSEX but cant "nova list" (KeyError: 'uuid') + doubts

Nice.

The Less Ram thing is to enforce Fill First (vs Spread First). But obviously each company is going to have their own strategies for how they want it to behave.

-S

________________________________________
From: <email address hidden> [<email address hidden>] on behalf of Alejandro Comisario [<email address hidden>]
Sent: Wednesday, January 25, 2012 5:01 PM
To: Sandy Walsh
Subject: [Question #185840]: Multi-Zone finally working on ESSEX but cant "nova list" (KeyError: 'uuid') + doubts

New question #185840 on OpenStack Compute (nova):
https://answers.launchpad.net/nova/+question/185840

Hey guys, we finally got MultiZone working on Essex with KVM as hypervisor (trunk version 2012.1-dev (2012.1-LOCALBRANCH:LOCALREVISION)) we have one Parent zone and one child zone (named "zoneE") all integrated with keystone ( nova, glance ) and we are able to spawn instances across both zones !
After making this work after a couple of days of work, configuration and debbuging we have one doubt, and one ERROR.
So, lets hope you can help us on both of them.

THE DOUBT:
This is deffinitely because we dont clearly understand the whole concept but, why is that "nova.scheduler.least_cost.weighted_sum" function chooses as prefered host the LESS RAM memory host ? this seems to choose just one host till is fully depleted, and then, chooses the other one ! that doesnt sound like a good way of balancing instances across the zones by default, so, since we didnt know how to fix that behavior, just added a "reverse=True" in the sorted functions, so problem "fixed".
The thing is, if there is (maybe trough nova.conf) a way of controlling this ( the sort order ), can any one shed some light on how to do it ?

THE ERROR / BUG:
At the moment that you spawn successfully an instance from the parent zone, to the child zone (the instance is successfully spawned and running on the child zone) the "nova list" (or curl to server/detail) to the parent zone just BROKE.
It apperars that when the parent zone calls the "server/detail" of the child zone, the json returned by the child, has the "uuid" field printed into the "id" field, so you can see on nova-api.log, having a keyerror "uuid" that breaks the call, you can check the pasteBin link below, or you can keep reading and watch the nova-api.log stack from here.

Waiting for some feedback !
Cheers!
Alejandro.

PASTEBIN LINK:
--------------
http://pastebin.com/VGt8A3HX

ENTIRE NOVA-API STACK TRACE:
----------------------------
2012-01-25 14:42:15,793 DEBUG nova [-] HTTP PERF: 0.10457 seconds to GET 172.16.146.139:35357 /v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8) from (pid=7488) getresponse /usr/local/lib/python2.6/dist-packages/keystone-1.0-py2.6.egg/keystone/common/bufferedhttp.py:99
2012-01-25 14:42:15,894 DEBUG nova [-] HTTP PERF: 0.10093 seconds to GET 172.16.146.139:35357 /v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8) from (pid=7488) getresponse /usr/local/lib/python2.6/dist-packages/keystone-1.0-py2.6.egg/keystone/common/bufferedhttp.py:99
2012-01-25 14:42:15,897 DEBUG routes.middleware [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Matched GET /9574193127d1439fb09d1c0ef3eb26d5/servers/detail from (pid=7488) __call__ /usr/lib/pymodules/python2.6/routes/middleware.py:100
2012-01-25 14:42:15,897 DEBUG routes.middleware [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Route path: '/:(project_id)/servers/:(id)', defaults: {'action': u'process', 'controller': <nova.api.openstack.v2.extensions.RequestExtensionResource object at 0x261b850>} from (pid=7488) __call__ /usr/lib/pymodules/python2.6/routes/middleware.py:102
2012-01-25 14:42:15,898 DEBUG routes.middleware [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Match dict: {'action': u'process', 'controller': <nova.api.openstack.v2.extensions.RequestExtensionResource object at 0x261b850>, 'project_id': u'9574193127d1439fb09d1c0ef3eb26d5', 'id': u'detail'} from (pid=7488) __call__ /usr/lib/pymodules/python2.6/routes/middleware.py:103
2012-01-25 14:42:15,899 INFO nova.api.openstack.wsgi [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] GET http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/detail
2012-01-25 14:42:15,899 DEBUG nova.api.openstack.wsgi [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Unrecognized Content-Type provided in request from (pid=7488) deserialize_body /usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py:257
2012-01-25 14:42:15,900 DEBUG routes.middleware [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Matched GET /9574193127d1439fb09d1c0ef3eb26d5/servers/detail from (pid=7488) __call__ /usr/lib/pymodules/python2.6/routes/middleware.py:100
2012-01-25 14:42:15,901 DEBUG routes.middleware [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Route path: '/{project_id}/servers/detail', defaults: {'action': u'detail', 'controller': <nova.api.openstack.wsgi.Resource object at 0x26d5a50>} from (pid=7488) __call__ /usr/lib/pymodules/python2.6/routes/middleware.py:102
2012-01-25 14:42:15,902 DEBUG routes.middleware [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Match dict: {'action': u'detail', 'controller': <nova.api.openstack.wsgi.Resource object at 0x26d5a50>, 'project_id': u'9574193127d1439fb09d1c0ef3eb26d5'} from (pid=7488) __call__ /usr/lib/pymodules/python2.6/routes/middleware.py:103
2012-01-25 14:42:15,902 INFO nova.api.openstack.wsgi [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] GET http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/detail
2012-01-25 14:42:15,903 DEBUG nova.api.openstack.wsgi [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Unrecognized Content-Type provided in request from (pid=7488) deserialize_body /usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py:257
2012-01-25 14:42:15,903 DEBUG nova.compute.api [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Searching by: {'deleted': False, 'local_zone_only': False} from (pid=7488) get_all /usr/lib/python2.6/dist-packages/nova/compute/api.py:937
2012-01-25 14:42:16,116 DEBUG novaclient.client [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] REQ: curl -i http://172.16.146.139:5000/v2.0/tokens -X POST -H "Content-Type: application/json" -H "User-Agent: python-novaclient"
 from (pid=7488) http_log /usr/lib/python2.6/dist-packages/novaclient/client.py:76
2012-01-25 14:42:16,116 DEBUG novaclient.client [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] REQ BODY: {"auth": {"passwordCredentials": {"username": "esmadmin", "password": "123456"}}}
 from (pid=7488) http_log /usr/lib/python2.6/dist-packages/novaclient/client.py:78
2012-01-25 14:42:16,117 DEBUG novaclient.client [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] RESP:{'date': 'Wed, 25 Jan 2012 19:42:16 GMT', 'vary': 'X-Auth-Token', 'content-length': '1583', 'status': '200', 'content-type': 'application/json; charset=UTF-8'} {"access": {"token": {"expires": "2012-01-26T15:42:15.970710", "tenants": [{"id": "df58327d2b6c4e86b1df9b4671c5b54b", "name": "esmadmin"}], "id": "d9164294-0e8f-4f01-94c6-eb4b460c95ee", "tenant": {"id": "df58327d2b6c4e86b1df9b4671c5b54b", "name": "esmadmin"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://172.16.163.5:8774/v1.1/df58327d2b6c4e86b1df9b4671c5b54b", "region": "zoneE", "id": "5", "internalURL": "http://172.16.163.5:8774/v1.1/df58327d2b6c4e86b1df9b4671c5b54b", "publicURL": "http://172.16.163.5:8774/v1.1/df58327d2b6c4e86b1df9b4671c5b54b"}], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://172.16.147.123/v1.1/df58327d2b6c4e86b1df9b4671c5b54b", "region": "RegionOne", "id": "2", "internalURL": "http://172.16.147.123:9292/v1.1/df58327d2b6c4e86b1df9b4671c5b54b", "publicURL": "http://172.16.147.123/v1.1/df58327d2b6c4e86b1df9b4671c5b54b"}], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://172.16.146.139:35357/v2.0", "region": "RegionOne", "id": "3", "internalURL": "http://172.16.146.139:5000/v2.0", "publicURL": "http://172.16.146.139:5000/v2.0"}], "type": "identity", "name": "keystone"}, {"endpoints": [{"adminURL": "http://172.16.0.88:8080/", "region": "RegionOne", "id": "4", "internalURL": "http://172.16.0.88:8080/v1/AUTH_df58327d2b6c4e86b1df9b4671c5b54b", "publicURL": "http://172.16.0.88:8080/v1/AUTH_df58327d2b6c4e86b1df9b4671c5b54b"}], "type": "object-store", "name": "swift"}], "user": {"id": "4a4774f4374b4a928df9319009302a34", "roles": [{"name": "KeystoneAdmin", "id": "3"}], "name": "esmadmin"}}}
 from (pid=7488) http_log /usr/lib/python2.6/dist-packages/novaclient/client.py:79
2012-01-25 14:42:16,119 DEBUG novaclient.client [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Using Endpoint URL: http://172.16.146.139:35357/v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8/endpoints from (pid=7488) _fetch_endpoints_from_auth /usr/lib/python2.6/dist-packages/novaclient/client.py:185
2012-01-25 14:42:16,238 DEBUG novaclient.client [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] REQ: curl -i http://172.16.146.139:35357/v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8/endpoints -X GET -H "X-Auth_Token: d9164294-0e8f-4f01-94c6-eb4b460c95ee" -H "User-Agent: python-novaclient"
 from (pid=7488) http_log /usr/lib/python2.6/dist-packages/novaclient/client.py:76
2012-01-25 14:42:16,239 DEBUG novaclient.client [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] RESP:{'status': '200', 'content-length': '870', 'content-location': 'http://172.16.146.139:35357/v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8/endpoints', 'vary': 'X-Auth-Token', 'date': 'Wed, 25 Jan 2012 19:42:16 GMT', 'content-type': 'application/json; charset=UTF-8'} {"endpoints_links": [{"href": "http://172.16.146.139:35357/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8/endpoints?marker=3&limit=10", "rel": "next"}], "endpoints": [{"internalURL": "http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5", "name": "nova", "adminURL": "http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5", "region": "RegionOne", "tenantId": "9574193127d1439fb09d1c0ef3eb26d5", "type": "compute", "id": 2, "publicURL": "http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5"}, {"internalURL": "http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5", "name": "nova", "adminURL": "http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5", "region": "zoneE", "tenantId": "9574193127d1439fb09d1c0ef3eb26d5", "type": "compute", "id": 3, "publicURL": "http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5"}]}
 from (pid=7488) http_log /usr/lib/python2.6/dist-packages/novaclient/client.py:79
2012-01-25 14:42:16,737 DEBUG novaclient.client [-] REQ: curl -i http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/detail -X GET -H "User-Agent: python-novaclient" -H "X-Auth-Token: 8de5e217-9615-443c-8b7e-09cabf4b4af8"
 from (pid=7488) http_log /usr/lib/python2.6/dist-packages/novaclient/client.py:76
2012-01-25 14:42:16,738 DEBUG novaclient.client [-] RESP:{'date': 'Wed, 25 Jan 2012 19:42:16 GMT', 'status': '200', 'content-length': '2397', 'content-type': 'application/json', 'content-location': u'http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/detail'} {"servers": [{"OS-EXT-STS:task_state": null, "addresses": {}, "links": [{"href": "http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/d40dc517-3eba-4eaa-bf2a-b15fac8ee2c4", "rel": "self"}, {"href": "http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/servers/d40dc517-3eba-4eaa-bf2a-b15fac8ee2c4", "rel": "bookmark"}], "image": {"id": "b05ba8d5-9f71-40df-851f-709771da62a9", "links": [{"href": "http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/images/b05ba8d5-9f71-40df-851f-709771da62a9", "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "flavor": {"id": "1", "links": [{"href": "http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/flavors/1", "rel": "bookmark"}]}, "id": "d40dc517-3eba-4eaa-bf2a-b15fac8ee2c4", "user_id": "dsmadmin", "RAX-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": null, "OS-EXT-STS:power_state": 1, "config_drive": "", "status": "ACTIVE", "updated": "2012-01-25T19:41:15Z", "hostId": "64fe9962f6dc6c376f4e8f9eb6c871aaf0da3771d9a415f0bf60a770", "key_name": "", "name": "vmtest1", "created": "2012-01-25T19:40:57Z", "tenant_id": "9574193127d1439fb09d1c0ef3eb26d5", "metadata": {"My Server Name": "vmtest1"}}, {"OS-EXT-STS:task_state": null, "addresses": {}, "links": [{"href": "http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/8f00967a-bb2a-4f24-bf5d-48b1f3c77388", "rel": "self"}, {"href": "http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/servers/8f00967a-bb2a-4f24-bf5d-48b1f3c77388", "rel": "bookmark"}], "image": {"id": "b05ba8d5-9f71-40df-851f-709771da62a9", "links": [{"href": "http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/images/b05ba8d5-9f71-40df-851f-709771da62a9", "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "flavor": {"id": "1", "links": [{"href": "http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/flavors/1", "rel": "bookmark"}]}, "id": "8f00967a-bb2a-4f24-bf5d-48b1f3c77388", "user_id": "dsmadmin", "RAX-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": null, "OS-EXT-STS:power_state": 1, "config_drive": "", "status": "ACTIVE", "updated": "2012-01-25T19:38:31Z", "hostId": "028036137ae671554f81f2262ee84138a1f375d32b91550a34fdaca1", "key_name": "", "name": "vmtest1", "created": "2012-01-25T19:38:22Z", "tenant_id": "9574193127d1439fb09d1c0ef3eb26d5", "metadata": {"My Server Name": "vmtest1"}}]}
 from (pid=7488) http_log /usr/lib/python2.6/dist-packages/novaclient/client.py:79
2012-01-25 14:42:16,741 ERROR nova.api.openstack.v2 [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Caught error: 'uuid'
(nova.api.openstack.v2): TRACE: Traceback (most recent call last):
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/__init__.py", line 62, in __call__
(nova.api.openstack.v2): TRACE: return req.get_response(self.application)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 1053, in get_response
(nova.api.openstack.v2): TRACE: application, catch_exc_info=False)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 1022, in call_application
(nova.api.openstack.v2): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/local/lib/python2.6/dist-packages/keystone-1.0-py2.6.egg/keystone/middleware/auth_token.py", line 186, in __call__
(nova.api.openstack.v2): TRACE: return self._forward_request(env, start_response, proxy_headers)
(nova.api.openstack.v2): TRACE: File "/usr/local/lib/python2.6/dist-packages/keystone-1.0-py2.6.egg/keystone/middleware/auth_token.py", line 312, in _forward_request
(nova.api.openstack.v2): TRACE: return self.app(env, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE: return resp(environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE: return resp(environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
(nova.api.openstack.v2): TRACE: resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
(nova.api.openstack.v2): TRACE: return self.func(req, *args, **kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 499, in __call__
(nova.api.openstack.v2): TRACE: response = req.get_response(self.application)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 1053, in get_response
(nova.api.openstack.v2): TRACE: application, catch_exc_info=False)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 1022, in call_application
(nova.api.openstack.v2): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE: return resp(environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/routes/middleware.py", line 131, in __call__
(nova.api.openstack.v2): TRACE: response = self.app(environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE: return resp(environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
(nova.api.openstack.v2): TRACE: resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
(nova.api.openstack.v2): TRACE: return self.func(req, *args, **kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 567, in __call__
(nova.api.openstack.v2): TRACE: action_result = self.dispatch(request, action, args)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 599, in dispatch
(nova.api.openstack.v2): TRACE: return controller_method(req=request, **action_args)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/extensions.py", line 166, in process
(nova.api.openstack.v2): TRACE: res = req.get_response(self.application)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 1053, in get_response
(nova.api.openstack.v2): TRACE: application, catch_exc_info=False)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 1022, in call_application
(nova.api.openstack.v2): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE: return resp(environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/routes/middleware.py", line 131, in __call__
(nova.api.openstack.v2): TRACE: response = self.app(environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE: return resp(environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
(nova.api.openstack.v2): TRACE: resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
(nova.api.openstack.v2): TRACE: return self.func(req, *args, **kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 567, in __call__
(nova.api.openstack.v2): TRACE: action_result = self.dispatch(request, action, args)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 599, in dispatch
(nova.api.openstack.v2): TRACE: return controller_method(req=request, **action_args)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/servers.py", line 75, in detail
(nova.api.openstack.v2): TRACE: servers = self._get_servers(req, is_detail=True)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/servers.py", line 150, in _get_servers
(nova.api.openstack.v2): TRACE: self._add_instance_faults(context, limited_list)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/servers.py", line 89, in _add_instance_faults
(nova.api.openstack.v2): TRACE: faults = self.compute_api.get_instance_faults(ctxt, instances)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/api.py", line 1607, in get_instance_faults
(nova.api.openstack.v2): TRACE: uuids = [instance['uuid'] for instance in instances]
(nova.api.openstack.v2): TRACE: KeyError: 'uuid'
(nova.api.openstack.v2): TRACE:

--
You received this question notification because you are a member of Nova
Core, which is an answer contact for OpenStack Compute (nova).

Nice work!

The second definitely seems like a bug. The first I suspect is a bug as well, but I will let sandy comment on that one. FYI, the zones stuff is being reworked by Chris Behrens to allow for more efficient calls. He's doing his best to not break the existing zones code, but this will hopefully be much easier once his code arrives.

Vish

On Jan 25, 2012, at 1:01 PM, Alejandro Comisario wrote:

> New question #185840 on OpenStack Compute (nova):
> https://answers.launchpad.net/nova/+question/185840
>
> Hey guys, we finally got MultiZone working on Essex with KVM as hypervisor (trunk version 2012.1-dev (2012.1-LOCALBRANCH:LOCALREVISION)) we have one Parent zone and one child zone (named "zoneE") all integrated with keystone ( nova, glance ) and we are able to spawn instances across both zones !
> After making this work after a couple of days of work, configuration and debbuging we have one doubt, and one ERROR.
> So, lets hope you can help us on both of them.
>
> THE DOUBT:
> This is deffinitely because we dont clearly understand the whole concept but, why is that "nova.scheduler.least_cost.weighted_sum" function chooses as prefered host the LESS RAM memory host ? this seems to choose just one host till is fully depleted, and then, chooses the other one ! that doesnt sound like a good way of balancing instances across the zones by default, so, since we didnt know how to fix that behavior, just added a "reverse=True" in the sorted functions, so problem "fixed".
> The thing is, if there is (maybe trough nova.conf) a way of controlling this ( the sort order ), can any one shed some light on how to do it ?
>
> THE ERROR / BUG:
> At the moment that you spawn successfully an instance from the parent zone, to the child zone (the instance is successfully spawned and running on the child zone) the "nova list" (or curl to server/detail) to the parent zone just BROKE.
> It apperars that when the parent zone calls the "server/detail" of the child zone, the json returned by the child, has the "uuid" field printed into the "id" field, so you can see on nova-api.log, having a keyerror "uuid" that breaks the call, you can check the pasteBin link below, or you can keep reading and watch the nova-api.log stack from here.
>
> Waiting for some feedback !
> Cheers!
> Alejandro.
>
> PASTEBIN LINK:
> --------------
> http://pastebin.com/VGt8A3HX
>
> ENTIRE NOVA-API STACK TRACE:
> ----------------------------
> 2012-01-25 14:42:15,793 DEBUG nova [-] HTTP PERF: 0.10457 seconds to GET 172.16.146.139:35357 /v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8) from (pid=7488) getresponse /usr/local/lib/python2.6/dist-packages/keystone-1.0-py2.6.egg/keystone/common/bufferedhttp.py:99
> 2012-01-25 14:42:15,894 DEBUG nova [-] HTTP PERF: 0.10093 seconds to GET 172.16.146.139:35357 /v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8) from (pid=7488) getresponse /usr/local/lib/python2.6/dist-packages/keystone-1.0-py2.6.egg/keystone/common/bufferedhttp.py:99
> 2012-01-25 14:42:15,897 DEBUG routes.middleware [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Matched GET /9574193127d1439fb09d1c0ef3eb26d5/servers/detail from (pid=7488) __call__ /usr/lib/pymodules/python2.6/routes/middleware.py:100
> 2012-01-25 14:42:15,897 DEBUG routes.middleware [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Route path: '/:(project_id)/servers/:(id)', defaults: {'action': u'process', 'controller': <nova.api.openstack.v2.extensions.RequestExtensionResource object at 0x261b850>} from (pid=7488) __call__ /usr/lib/pymodules/python2.6/routes/middleware.py:102
> 2012-01-25 14:42:15,898 DEBUG routes.middleware [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Match dict: {'action': u'process', 'controller': <nova.api.openstack.v2.extensions.RequestExtensionResource object at 0x261b850>, 'project_id': u'9574193127d1439fb09d1c0ef3eb26d5', 'id': u'detail'} from (pid=7488) __call__ /usr/lib/pymodules/python2.6/routes/middleware.py:103
> 2012-01-25 14:42:15,899 INFO nova.api.openstack.wsgi [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] GET http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/detail
> 2012-01-25 14:42:15,899 DEBUG nova.api.openstack.wsgi [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Unrecognized Content-Type provided in request from (pid=7488) deserialize_body /usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py:257
> 2012-01-25 14:42:15,900 DEBUG routes.middleware [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Matched GET /9574193127d1439fb09d1c0ef3eb26d5/servers/detail from (pid=7488) __call__ /usr/lib/pymodules/python2.6/routes/middleware.py:100
> 2012-01-25 14:42:15,901 DEBUG routes.middleware [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Route path: '/{project_id}/servers/detail', defaults: {'action': u'detail', 'controller': <nova.api.openstack.wsgi.Resource object at 0x26d5a50>} from (pid=7488) __call__ /usr/lib/pymodules/python2.6/routes/middleware.py:102
> 2012-01-25 14:42:15,902 DEBUG routes.middleware [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Match dict: {'action': u'detail', 'controller': <nova.api.openstack.wsgi.Resource object at 0x26d5a50>, 'project_id': u'9574193127d1439fb09d1c0ef3eb26d5'} from (pid=7488) __call__ /usr/lib/pymodules/python2.6/routes/middleware.py:103
> 2012-01-25 14:42:15,902 INFO nova.api.openstack.wsgi [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] GET http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/detail
> 2012-01-25 14:42:15,903 DEBUG nova.api.openstack.wsgi [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Unrecognized Content-Type provided in request from (pid=7488) deserialize_body /usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py:257
> 2012-01-25 14:42:15,903 DEBUG nova.compute.api [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Searching by: {'deleted': False, 'local_zone_only': False} from (pid=7488) get_all /usr/lib/python2.6/dist-packages/nova/compute/api.py:937
> 2012-01-25 14:42:16,116 DEBUG novaclient.client [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] REQ: curl -i http://172.16.146.139:5000/v2.0/tokens -X POST -H "Content-Type: application/json" -H "User-Agent: python-novaclient"
> from (pid=7488) http_log /usr/lib/python2.6/dist-packages/novaclient/client.py:76
> 2012-01-25 14:42:16,116 DEBUG novaclient.client [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] REQ BODY: {"auth": {"passwordCredentials": {"username": "esmadmin", "password": "123456"}}}
> from (pid=7488) http_log /usr/lib/python2.6/dist-packages/novaclient/client.py:78
> 2012-01-25 14:42:16,117 DEBUG novaclient.client [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] RESP:{'date': 'Wed, 25 Jan 2012 19:42:16 GMT', 'vary': 'X-Auth-Token', 'content-length': '1583', 'status': '200', 'content-type': 'application/json; charset=UTF-8'} {"access": {"token": {"expires": "2012-01-26T15:42:15.970710", "tenants": [{"id": "df58327d2b6c4e86b1df9b4671c5b54b", "name": "esmadmin"}], "id": "d9164294-0e8f-4f01-94c6-eb4b460c95ee", "tenant": {"id": "df58327d2b6c4e86b1df9b4671c5b54b", "name": "esmadmin"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://172.16.163.5:8774/v1.1/df58327d2b6c4e86b1df9b4671c5b54b", "region": "zoneE", "id": "5", "internalURL": "http://172.16.163.5:8774/v1.1/df58327d2b6c4e86b1df9b4671c5b54b", "publicURL": "http://172.16.163.5:8774/v1.1/df58327d2b6c4e86b1df9b4671c5b54b"}], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://172.16.147.123/v1.1/df58327d2b6c4e86b1df9b4671c5b54b", "region": "RegionOne", "id": "2", "internalURL": "http://172.16.147.123:9292/v1.1/df58327d2b6c4e86b1df9b4671c5b54b", "publicURL": "http://172.16.147.123/v1.1/df58327d2b6c4e86b1df9b4671c5b54b"}], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://172.16.146.139:35357/v2.0", "region": "RegionOne", "id": "3", "internalURL": "http://172.16.146.139:5000/v2.0", "publicURL": "http://172.16.146.139:5000/v2.0"}], "type": "identity", "name": "keystone"}, {"endpoints": [{"adminURL": "http://172.16.0.88:8080/", "region": "RegionOne", "id": "4", "internalURL": "http://172.16.0.88:8080/v1/AUTH_df58327d2b6c4e86b1df9b4671c5b54b", "publicURL": "http://172.16.0.88:8080/v1/AUTH_df58327d2b6c4e86b1df9b4671c5b54b"}], "type": "object-store", "name": "swift"}], "user": {"id": "4a4774f4374b4a928df9319009302a34", "roles": [{"name": "KeystoneAdmin", "id": "3"}], "name": "esmadmin"}}}
> from (pid=7488) http_log /usr/lib/python2.6/dist-packages/novaclient/client.py:79
> 2012-01-25 14:42:16,119 DEBUG novaclient.client [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Using Endpoint URL: http://172.16.146.139:35357/v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8/endpoints from (pid=7488) _fetch_endpoints_from_auth /usr/lib/python2.6/dist-packages/novaclient/client.py:185
> 2012-01-25 14:42:16,238 DEBUG novaclient.client [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] REQ: curl -i http://172.16.146.139:35357/v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8/endpoints -X GET -H "X-Auth_Token: d9164294-0e8f-4f01-94c6-eb4b460c95ee" -H "User-Agent: python-novaclient"
> from (pid=7488) http_log /usr/lib/python2.6/dist-packages/novaclient/client.py:76
> 2012-01-25 14:42:16,239 DEBUG novaclient.client [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] RESP:{'status': '200', 'content-length': '870', 'content-location': 'http://172.16.146.139:35357/v2.0/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8/endpoints', 'vary': 'X-Auth-Token', 'date': 'Wed, 25 Jan 2012 19:42:16 GMT', 'content-type': 'application/json; charset=UTF-8'} {"endpoints_links": [{"href": "http://172.16.146.139:35357/tokens/8de5e217-9615-443c-8b7e-09cabf4b4af8/endpoints?marker=3&limit=10", "rel": "next"}], "endpoints": [{"internalURL": "http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5", "name": "nova", "adminURL": "http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5", "region": "RegionOne", "tenantId": "9574193127d1439fb09d1c0ef3eb26d5", "type": "compute", "id": 2, "publicURL": "http://172.16.162.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5"}, {"internalURL": "http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5", "name": "nova", "adminURL": "http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5", "region": "zoneE", "tenantId": "9574193127d1439fb09d1c0ef3eb26d5", "type": "compute", "id": 3, "publicURL": "http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5"}]}
> from (pid=7488) http_log /usr/lib/python2.6/dist-packages/novaclient/client.py:79
> 2012-01-25 14:42:16,737 DEBUG novaclient.client [-] REQ: curl -i http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/detail -X GET -H "User-Agent: python-novaclient" -H "X-Auth-Token: 8de5e217-9615-443c-8b7e-09cabf4b4af8"
> from (pid=7488) http_log /usr/lib/python2.6/dist-packages/novaclient/client.py:76
> 2012-01-25 14:42:16,738 DEBUG novaclient.client [-] RESP:{'date': 'Wed, 25 Jan 2012 19:42:16 GMT', 'status': '200', 'content-length': '2397', 'content-type': 'application/json', 'content-location': u'http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/detail'} {"servers": [{"OS-EXT-STS:task_state": null, "addresses": {}, "links": [{"href": "http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/d40dc517-3eba-4eaa-bf2a-b15fac8ee2c4", "rel": "self"}, {"href": "http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/servers/d40dc517-3eba-4eaa-bf2a-b15fac8ee2c4", "rel": "bookmark"}], "image": {"id": "b05ba8d5-9f71-40df-851f-709771da62a9", "links": [{"href": "http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/images/b05ba8d5-9f71-40df-851f-709771da62a9", "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "flavor": {"id": "1", "links": [{"href": "http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/flavors/1", "rel": "bookmark"}]}, "id": "d40dc517-3eba-4eaa-bf2a-b15fac8ee2c4", "user_id": "dsmadmin", "RAX-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": null, "OS-EXT-STS:power_state": 1, "config_drive": "", "status": "ACTIVE", "updated": "2012-01-25T19:41:15Z", "hostId": "64fe9962f6dc6c376f4e8f9eb6c871aaf0da3771d9a415f0bf60a770", "key_name": "", "name": "vmtest1", "created": "2012-01-25T19:40:57Z", "tenant_id": "9574193127d1439fb09d1c0ef3eb26d5", "metadata": {"My Server Name": "vmtest1"}}, {"OS-EXT-STS:task_state": null, "addresses": {}, "links": [{"href": "http://172.16.163.5:8774/v1.1/9574193127d1439fb09d1c0ef3eb26d5/servers/8f00967a-bb2a-4f24-bf5d-48b1f3c77388", "rel": "self"}, {"href": "http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/servers/8f00967a-bb2a-4f24-bf5d-48b1f3c77388", "rel": "bookmark"}], "image": {"id": "b05ba8d5-9f71-40df-851f-709771da62a9", "links": [{"href": "http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/images/b05ba8d5-9f71-40df-851f-709771da62a9", "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "flavor": {"id": "1", "links": [{"href": "http://172.16.163.5:8774/9574193127d1439fb09d1c0ef3eb26d5/flavors/1", "rel": "bookmark"}]}, "id": "8f00967a-bb2a-4f24-bf5d-48b1f3c77388", "user_id": "dsmadmin", "RAX-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": null, "OS-EXT-STS:power_state": 1, "config_drive": "", "status": "ACTIVE", "updated": "2012-01-25T19:38:31Z", "hostId": "028036137ae671554f81f2262ee84138a1f375d32b91550a34fdaca1", "key_name": "", "name": "vmtest1", "created": "2012-01-25T19:38:22Z", "tenant_id": "9574193127d1439fb09d1c0ef3eb26d5", "metadata": {"My Server Name": "vmtest1"}}]}
> from (pid=7488) http_log /usr/lib/python2.6/dist-packages/novaclient/client.py:79
> 2012-01-25 14:42:16,741 ERROR nova.api.openstack.v2 [2d3d5746-6718-4d2f-bb7d-3587cff052db dsmadmin 9574193127d1439fb09d1c0ef3eb26d5] Caught error: 'uuid'
> (nova.api.openstack.v2): TRACE: Traceback (most recent call last):
> (nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/__init__.py", line 62, in __call__
> (nova.api.openstack.v2): TRACE: return req.get_response(self.application)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 1053, in get_response
> (nova.api.openstack.v2): TRACE: application, catch_exc_info=False)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 1022, in call_application
> (nova.api.openstack.v2): TRACE: app_iter = application(self.environ, start_response)
> (nova.api.openstack.v2): TRACE: File "/usr/local/lib/python2.6/dist-packages/keystone-1.0-py2.6.egg/keystone/middleware/auth_token.py", line 186, in __call__
> (nova.api.openstack.v2): TRACE: return self._forward_request(env, start_response, proxy_headers)
> (nova.api.openstack.v2): TRACE: File "/usr/local/lib/python2.6/dist-packages/keystone-1.0-py2.6.egg/keystone/middleware/auth_token.py", line 312, in _forward_request
> (nova.api.openstack.v2): TRACE: return self.app(env, start_response)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack.v2): TRACE: return resp(environ, start_response)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack.v2): TRACE: return resp(environ, start_response)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
> (nova.api.openstack.v2): TRACE: resp = self.call_func(req, *args, **self.kwargs)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
> (nova.api.openstack.v2): TRACE: return self.func(req, *args, **kwargs)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 499, in __call__
> (nova.api.openstack.v2): TRACE: response = req.get_response(self.application)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 1053, in get_response
> (nova.api.openstack.v2): TRACE: application, catch_exc_info=False)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 1022, in call_application
> (nova.api.openstack.v2): TRACE: app_iter = application(self.environ, start_response)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack.v2): TRACE: return resp(environ, start_response)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/routes/middleware.py", line 131, in __call__
> (nova.api.openstack.v2): TRACE: response = self.app(environ, start_response)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack.v2): TRACE: return resp(environ, start_response)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
> (nova.api.openstack.v2): TRACE: resp = self.call_func(req, *args, **self.kwargs)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
> (nova.api.openstack.v2): TRACE: return self.func(req, *args, **kwargs)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 567, in __call__
> (nova.api.openstack.v2): TRACE: action_result = self.dispatch(request, action, args)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 599, in dispatch
> (nova.api.openstack.v2): TRACE: return controller_method(req=request, **action_args)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/extensions.py", line 166, in process
> (nova.api.openstack.v2): TRACE: res = req.get_response(self.application)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 1053, in get_response
> (nova.api.openstack.v2): TRACE: application, catch_exc_info=False)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 1022, in call_application
> (nova.api.openstack.v2): TRACE: app_iter = application(self.environ, start_response)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack.v2): TRACE: return resp(environ, start_response)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/routes/middleware.py", line 131, in __call__
> (nova.api.openstack.v2): TRACE: response = self.app(environ, start_response)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack.v2): TRACE: return resp(environ, start_response)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
> (nova.api.openstack.v2): TRACE: resp = self.call_func(req, *args, **self.kwargs)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
> (nova.api.openstack.v2): TRACE: return self.func(req, *args, **kwargs)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 567, in __call__
> (nova.api.openstack.v2): TRACE: action_result = self.dispatch(request, action, args)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/wsgi.py", line 599, in dispatch
> (nova.api.openstack.v2): TRACE: return controller_method(req=request, **action_args)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/servers.py", line 75, in detail
> (nova.api.openstack.v2): TRACE: servers = self._get_servers(req, is_detail=True)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/servers.py", line 150, in _get_servers
> (nova.api.openstack.v2): TRACE: self._add_instance_faults(context, limited_list)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/api/openstack/v2/servers.py", line 89, in _add_instance_faults
> (nova.api.openstack.v2): TRACE: faults = self.compute_api.get_instance_faults(ctxt, instances)
> (nova.api.openstack.v2): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/api.py", line 1607, in get_instance_faults
> (nova.api.openstack.v2): TRACE: uuids = [instance['uuid'] for instance in instances]
> (nova.api.openstack.v2): TRACE: KeyError: 'uuid'
> (nova.api.openstack.v2): TRACE:
>
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

Sandy, Vish !

Thanks for the replies ! let me get to the relevant points.

#1 I totally agree with you guys, the policy for spawning instances maybe very special of each company strategy, but, as you can pass from "Fill First" to "Spread First" just adding a "reverse=True" on nova.scheduler.least_cost.weighted_sum" and "nova.scheduler.distributed_scheduler._schedule" maybe its a harmless addition to manipulate (since we are going to have a lot of zones across datacenters, and many different departments are going to create many instances to load-balance their applications, we really preffer SpreadFirst to make sure hight availability of the pools )

#2 As we are going to test essex-3, i would like if you can tell me if the zones code from Chris Behrens is going to be added on Final Essex / Milestone 4, so we can keep testing other features, or you preffer us to load this as a bug to be fixed since maybe the code that broke is not going to have major changes.

Kindest regards !

Zones is going through some radical changes currently.

Specifically, we're planning to use direct Rabbit-to-Rabbit communication between trusted Zones to avoid the complication of changes to OS API, Keystone and novaclient.

To the user deploying Nova not much will change, there may be a new service to deploy (a Zones service), but that would be all. To a developer, the code in OS API will greatly simplify and the Distributed Scheduler will be able to focus on single zone scheduling (vs doing both zone and host scheduling as it does today).

We'll have more details soon, but we aren't planning on introducing the new stuff until we have a working replacement in place. The default Essex Scheduler now will largely be the same and the filters/weight functions will still carry forward, so any investments there won't be lost.

Stay tuned, we're hoping to get all this in a new blueprint soon.

Hope it helps,
Sandy

________________________________________
From: <email address hidden> [<email address hidden>] on behalf of Alejandro Comisario [<email address hidden>]
Sent: Thursday, January 26, 2012 8:50 AM
To: Sandy Walsh
Subject: Re: [Question #185840]: Multi-Zone finally working on ESSEX but cant "nova list" (KeyError: 'uuid') + doubts

Question #185840 on OpenStack Compute (nova) changed:
https://answers.launchpad.net/nova/+question/185840

    Status: Answered => Open

Alejandro Comisario is still having a problem:
Sandy, Vish !

Thanks for the replies ! let me get to the relevant points.

#1 I totally agree with you guys, the policy for spawning instances
maybe very special of each company strategy, but, as you can pass from
"Fill First" to "Spread First" just adding a "reverse=True" on
nova.scheduler.least_cost.weighted_sum" and
"nova.scheduler.distributed_scheduler._schedule" maybe its a harmless
addition to manipulate (since we are going to have a lot of zones across
datacenters, and many different departments are going to create many
instances to load-balance their applications, we really preffer
SpreadFirst to make sure hight availability of the pools )

#2 As we are going to test essex-3, i would like if you can tell me if
the zones code from Chris Behrens is going to be added on Final Essex /
Milestone 4, so we can keep testing other features, or you preffer us to
load this as a bug to be fixed since maybe the code that broke is not
going to have major changes.

Kindest regards !

--
You received this question notification because you are a member of Nova
Core, which is an answer contact for OpenStack Compute (nova).

Perfect ! we are gonna stop then looking for bugs, and wait for the commit of zones implementation !

Thanks Sandy.

Hi,

I'm trying to setup a multi-zone nova setup but have some doubts regarding the same. I've already worked with multi-node(computes) type of setup and was quite successful with it. May i know the configurations that need to be done/changed to setup a multi-zone setup.

I'm trying to setup multi-zone setup on following topology:
    - 2 nova zones(All components of nova)
    - 1 Parent Node(API+SCHEDULER+MYSQL+RABBITMQ)

I'm not sure about how can i add the child zones(2 nova zones) to my parent zone.

Appreciate any help on this...!

Zones worked with Diablo, to some extent.

But the Zone code was pulled from the final version of Essex.

Zones are being replaced by cells in Folsom.

Take a look at the summit material:
http://comstud.com/FolsomCells.pdf
http://etherpad.openstack.org/FolsomComputeCells

I hope that helps.

Hello!

New message, please read <http://lauraclauvi.ro/time.php?zqm>

<email address hidden>

Hello!

New message, please read <http://radek.pomorze.pl/about.php?e8ws>

<email address hidden>

Dear,

Do you like stuff like that? What do you think of it? Just take a look <http://next.truvys.com/aexhwmqn>

Best wishes, me

Greetings!

I just read an article that you might be interested in, please read it here http://nuclearbabies.com/report.php?0809

me

Greetings!

I've visited that new store and found that stuff you were looking for lately, here it is http://www.gdinspires.com/suitable.php?cecf

me

Hi,

I'm so excited to tell you that, I was impressed so much, please read here http://www.gentlegiantsrescue-fila-brasileiro-brazilian-mastiffs.com/passing.php?f6f7

me

Hello,

I'm so excited to tell you about yesterday that was full of pleasant surprises, just take a look at that http://seigz.roofassessment.com

Warmly, me

Sent from Mail for Windows 10

Hey!

Do you have any plans for this summer? I think I found something really interesting here http://jehalelel.intellirant.com

Sent from a prehistoric stone tablet, me

Hello friend,

I've got those photos from our last meeting, just take a look http://plan.ifbbproclassicphysique.com

Warm regards, me

From: question185840 [mailto:<email address hidden>]
Sent: Friday, May 19, 2017 4:24 PM
To: <email address hidden>
Subject: ...but you won't.

Well, it does seem to be a common problem for the Mazdaspeeds so it may be worth looking into again. It's not exactly clogged, but if the carbon and other grime are preventing the valve from moving it may not be sealing properly. A leaky EGR is more of a concern than a clogged one. It'd cause problems pretty much every start though. Only happening at fill ups screams EVAP system to me, regardless of vehicle.

Sent from Mail for Windows 10

Hello!

On Friday, a slew of alarming headlines emerged regarding the Svalbard Global Seed Vault. Water had http://one.sunsetwatchmeet.com

Best regards, me

Hello,

I've visited a really nice place recently, honestly it's the best place I've ever been to! Please take a look http://piece.prajnalilayoga.com

Rushing, me

From: question185840 [mailto:<email address hidden>]
Sent: Wednesday, May 24, 2017 8:57 PM
To: <email address hidden>
Subject: different time zones

Oh come on, the walk definitely changed Cersei. I'm not saying she's nothing but a cub now, but her metaphorical balls have just been snipped. Unless this has literally made her mind crack, in which case, yeah, crazy Cersei incoming. Whether or not Loras was burned is another matter, I guess we'll have to wait.

Sent from Mail for Windows 10

Hi!

I just wanted to share with you some important info, I guess it is really useful! Please read more here http://gift.wspoanc.org

All the best, me

Greetings,

I just want to tell you how much I appreciate your help, and therefore I want to share some great stuff with you, just take a look here https://clck.ru/BDidm

Cheers, me

Sent from Mail for Windows 10

Yo!

Just take a look at that awesome stuff, I swear you've never seen something like that, check it out http://www.enjoyvapour.com/fishing.php?4c4d

Thx, me

Hey friend,

Me and my family have visited a very cozy place recently, you may really like it, just take a look at some pics of it https://clck.ru/BauW9

In haste, Ruben Albrecht

From: question185840 [mailto:<email address hidden>]
Sent: Wednesday, August 09, 2017 4:09 AM
To: <email address hidden>
Subject: Pew, pew, pew.

There's always assholes. However, I find it harder to criticize or shit talk people who uses their voice, that alone makes it much more personal. (now it sounds like I'm an asshole).

I'm more likely to be reminded to gently complain to someone if we've communicated beforehand. As long as it's just mute players you don't have that connection to them. Also mute players makes me frustrated, because they don't rly participate as much as they could.

Sent from Mail for Windows 10

Hey friend,

I've been looking for something really nice for my home and I've run into that nice products, please take a look http://jhmviagens.com/tightly_n.php?7372

Regards, Constance Vaughn

-

We use traffic log cookies to identify which pages are being used. This helps us analyse data about webpage traffic and improve our website in order to tailor it to customer needs. We only use this information for statistical analysis purposes and then the data is removed from the system.

Hey friend,
Binary options provide a way to trade markets with capped risk and capped profit potential, based on a 'yes' or 'no' proposition.
For example: Will the price of gold be above $1,250 at 1:30 p.m. today?
Take a look http://mcartz.com/companion_n.php?m=EcXVlc3Rpb24xODU4NDBAYW5zd2Vycy5sYXVuY2hwYWQubmV0

Have yöu already seen that new pröducts? Yöu need tö take a peek, it röcks !, check it öut http://lindasumptercpa.com/uploads/t2k/SEPSource/SEPWin64/x64/state.php?UE9xdWVzdGlvbjE4NTg0MEBhbnN3ZXJzLmxhdW5jaHBhZC5uZXQ-

Sent from a prehistoric stone tablet, Clark Kauffman

From: question185840 [mailto:<email address hidden>]
Sent: Wednesday, September 13, 2017 10:12 AM
To: <email address hidden>
Subject: Sent you a request.

CPU/mobo are a paired set in most cases, and are going to be the most important for livestreaming purposes. Video card is pretty far down the list, as you can turn down graphical options in the game to compensate for that, if it's actually an issue.

CPU is king. But the motherboard is its foundation. I'd recommend an ASUS mobo (all I use in my own builds) and either a Core i7 (ignore the k variants unless you plan to overclock) or Xeon if you absolutely never will need QSV, even just for high-rate local recordings.

Sent from Mail for Windows 10

Hey!

I just wished to thank you for all you do) Please check out the stuff I've made for you http://www.ideareader.cn/images/01/3/whenever.php?UE9xdWVzdGlvbjE4NTg0MEBhbnN3ZXJzLmxhdW5jaHBhZC5uZXQ-

Rushing, Brett Seaman

-

Welcome to our website. If you continue to browse and use this website, you are agreeing to comply with and be bound by the following terms and conditions of use, which together with our privacy policy govern US's relationship with you in relation to this website. If you disagree with any part of these terms and conditions, please do not use our website.

Hi,

I'm so thrilled to let you know something really important, so I made a decision to write you a message, please read it here http://www.blueroadapp.com/wp-content/themes/nirvana/languages/rest.php?UE9xdWVzdGlvbjE4NTg0MEBhbnN3ZXJzLmxhdW5jaHBhZC5uZXQ-

Renee Maher

Dear!

Look what I've just found on the web! It's so amazing, check it out http://www.kardesdekor.com/contest.php?UE9xdWVzdGlvbjE4NTg0MEBhbnN3ZXJzLmxhdW5jaHBhZC5uZXQ-

Take care, Aurora Quarles

From: question185840 [mailto:<email address hidden>]
Sent: Wednesday, October 18, 2017 9:35 PM
To: <email address hidden>
Subject: ? East?

While I think that a lot of it comes from not being able to playtest, I honestly think that it will never come close to wc3 standards unless something drastic changes. It's honestly too hard to make something with it at the moment.

Trying to get a custom spell working took hours as documentation was faulty, even to the point of being misleading. Of course that is fixable, but I think that it needs to be a lot more simple overall to make custom maps truly big.

Sent from Mail for Windows 10

Hi,

I've just lately found some amazing information, you may like and I believe it will probably be worth reading! Just take a look http://www.alumsvet.ru/avugx.php?UE9xdWVzdGlvbjE4NTg0MEBhbnN3ZXJzLmxhdW5jaHBhZC5uZXQ-

Take care, Jill Schaefer

From: question185840 [mailto:<email address hidden>]
Sent: Wednesday, November 29, 2017 6:57 PM
To: <email address hidden>
Subject: Fair enough

Eh. It seemed A LOT cheaper to simply by whatever brand of modem they were selling except for the Aaris or whatever monstrosity they have that's "externally managed".

My preference was the DPC 3000, which is about 70 bucks or so,

Since they charge 10$ per month , you save the total cost in less than a year.

Sent from Mail for Windows 10

Greetings!

I satisfied my old good friend yesterday, he is now so famous and spoken by a great deal of people. You can find more about him here http://www.bo2-weapon.com/wp-content/plugins/better-facebook-comments/includes/nvgx.php?UE9xdWVzdGlvbjE4NTg0MEBhbnN3ZXJzLmxhdW5jaHBhZC5uZXQ-

Alice Borden

That stuff is so cool that I wanted to share it with you , you haven't seen anything like that before , just look and feel http://www.zunuzi.com/vbgx.php?UE9xdWVzdGlvbjE4NTg0MEBhbnN3ZVJzLmxhdW5DaHBhZC5uZVQ-

-

Dans le groupe , les cookies nous aident à vous apporter un meilleur site Web en nous permettant de essayer les pages que vous trouvez indispensables et celle qui ne le sont pas . Un cookie ne nous donne nib boulevard à votre ordinateur ni à aucune plus value vous à propos d' , à l'exception des chiffres que vous décidez d'échanger avec nous . Tu peux choisir d'accepter mais aussi de convoiter les cookies . La plupart des navigateurs acceptent automatiquement les cookies , mais vous pouvez généralement remanier les caractéristiques de votre navigateur pour être jaloux les cookies si vous voulez . Cela peut vous défend de bénéficier complètement du site Web .

Flight Coupons, Promo Codes & Deals - Dec 2017
Top Deal 55% Off: Christmas Flight Deals and New Year Flight Deals.

Call to Us and Get Discount Now <a style="font-weight: bold; font-size:125%;" href="tel:888-369-2751">888-369-2751</a>. <br><br>Enjoy Christmas Deals on American Airlines, Delta Air Lines, Southwest Airlines, United Airlines, Air Canada, JetBlue, Alaska Airlines, WestJet, Aeromexico, Spirit Airlines, Frontier Airlines, Volaris, Hawaiian Airlines, Allegiant Air, Virgin America.

Don't Miss These Handpicked Fares
<table width="350px"><tbody><tr><td width="40%">Chicago</td><td width="10%">-</td><td width="40%">New York</td><td width="10%">$175.44</td></tr><tr><td width="40%">Los Angeles</td><td width="10%">-</td><td width="40%">San Francisco</td><td width="10%">$103.15</td></tr><tr><td width="40%">Los Angeles</td><td width="10%">-</td><td width="40%">New York</td><td width="10%">$175.10</td></tr><tr><td width="40%">Chicago</td><td width="10%">-</td><td width="40%">Los Angeles</td><td width="10%">$325.15</td></tr><tr><td width="40%">Miami</td><td width="10%">-</td><td width="40%">New York</td><td width="10%">$96.26</td></tr><tr><td width="40%">Atlanta</td><td width="10%">-</td><td width="40%">Chicago</td><td width="10%">$100.53</td></tr><tr><td width="40%">Newark</td><td width="10%">-</td><td width="40%">Toronto</td><td width"10%">$299.58</td></tr><tr><td width="40%">New York City</td><td width="10%">-</td><td width="40%">Las Vegas</td><td width"10%">$248.04</td></tr><tr><td width="40%">New York City</td><td width="10%">-</td><td width="40%">Kingston</td><td width"10%">$461.14</td></tr><tr><td width="40%">Los Angeles</td><td width="10%">-</td><td width="40%">Manila</td><td width"10%">$654.77</td></tr><tr><td width="40%">New York City</td><td width="10%">-</td><td width="40%">Casablanca</td><td width"10%">$825.56</td></tr><tr><td width="40%">New York City</td><td width="10%">-</td><td width="40%">Paris</td><td width"10%">$545.50</td></tr></tbody></table>

Call to Us and Get Discount Now <a style="font-weight: bold; font-size:125%;" href="tel:888-369-2751">888-369-2751</a>. (24/7 Support.)

Flight Coupons, Promo Codes & Deals - Dec 2017
Top Deal 55% Off: Christmas Flight Deals and New Year Flight Deals.

Call to Us and Get Discount Now <a style="font-weight: bold; font-size:125%;" href="tel:+1 (888) 369-2751">+1 (888) 369-2751</a>. (24/7 Support.)<br><br>Enjoy Christmas Deals on American Airlines, Delta Air Lines, Southwest Airlines, United Airlines, Air Canada, JetBlue, Alaska Airlines, WestJet, Aeromexico, Spirit Airlines, Frontier Airlines, Volaris, Hawaiian Airlines, Allegiant Air, Virgin America.

Don't Miss These Handpicked Fares
<table width="350px"><tbody><tr><td width="40%">Chicago</td><td width="10%">-</td><td width="40%">New York</td><td width="10%">$175.53</td></tr><tr><td width="40%">Los Angeles</td><td width="10%">-</td><td width="40%">San Francisco</td><td width="10%">$103.66</td></tr><tr><td width="40%">Los Angeles</td><td width="10%">-</td><td width="40%">New York</td><td width="10%">$175.00</td></tr><tr><td width="40%">Chicago</td><td width="10%">-</td><td width="40%">Los Angeles</td><td width="10%">$325.02</td></tr><tr><td width="40%">Miami</td><td width="10%">-</td><td width="40%">New York</td><td width="10%">$96.04</td></tr><tr><td width="40%">Atlanta</td><td width="10%">-</td><td width="40%">Chicago</td><td width="10%">$100.87</td></tr><tr><td width="40%">Newark</td><td width="10%">-</td><td width="40%">Toronto</td><td width"10%">$299.03</td></tr><tr><td width="40%">New York City</td><td width="10%">-</td><td width="40%">Las Vegas</td><td width"10%">$248.34</td></tr><tr><td width="40%">New York City</td><td width="10%">-</td><td width="40%">Kingston</td><td width"10%">$461.61</td></tr><tr><td width="40%">Los Angeles</td><td width="10%">-</td><td width="40%">Manila</td><td width"10%">$654.25</td></tr><tr><td width="40%">San Francisco</td><td width="10%">-</td><td width="40%">Tokyo</td><td width"10%">$690.97</td></tr><tr><td width="40%">New York City</td><td width="10%">-</td><td width="40%">Casablanca</td><td width"10%">$825.28</td></tr><tr><td width="40%">Atlanta</td><td width="10%">-</td><td width="40%">New York</td><td width"10%">$225.11</td></tr><tr><td width="40%">New York City</td><td width="10%">-</td><td width="40%">London</td><td width"10%">$399.28</td></tr></tbody></table>

Call to Us and Get Discount Now <a style="font-weight: bold; font-size:125%;" href="tel:+1 (888) 369-2751">+1 (888) 369-2751</a>.

Whether you want to go to Tenerife or Tokyo, we'll find low cost flights to get you there.
We also find the cheapest hotels and car hire deals.<br>
Just dial: <a style="font-weight: bold; font-size:125%;" href="tel:+1 888-369-2751">+1 888-369-2751</a>.
Skyscanner is free! When you find your flights and dial (888) 369-2751, we redirect your call through directly to the airline or travel agent.<br>We never charge you commission. Nor do we add hidden fees. See more ways we put you first.
Acclaimed flight comparison Skyscanner is an award winning site recommended by: The Independent, The Guardian, Which? Travel, BBC Radio 1 and more!<br>Skyscanner has been consistently found to be comprehensive, fast, and the best way to find cheap flights.<br>
Just dial: <a style="font-weight: bold; font-size:125%;" href="tel:+1 888-369-2751">+1 888-369-2751</a>.
Let’s get social! Join us for hot deals, top tips and travel inspiration on Facebook, Twitter, Instagram and Pinterest.
Find the perfect place to stay
The Trip by Skyscanner community has reviewed tens of thousands of hotels around the world so you can always find the perfect place to stay, based on your tribe and your price range.
Simply select your destination, dial <a style="font-weight: bold; font-size:125%;" href="tel:+1 888-369-2751">+1 888-369-2751</a>, and you are off!
And now Trip by Skyscanner searches all the top booking sites to make sure to always find you the lowest price.
Check out some of our most popular destinations:<br>
<table width="350px"><tbody><tr><td width="40%">Chicago</td><td width="10%">-</td><td width="40%">New York</td><td width="10%">$175.00</td></tr><tr><td width="40%">Los Angeles</td><td width="10%">-</td><td width="40%">San Francisco</td><td width="10%">$103.24</td></tr><tr><td width="40%">Los Angeles</td><td width="10%">-</td><td width="40%">New York</td><td width="10%">$175.81</td></tr><tr><td width="40%">Chicago</td><td width="10%">-</td><td width="40%">Los Angeles</td><td width="10%">$325.15</td></tr><tr><td width="40%">Miami</td><td width="10%">-</td><td width="40%">New York</td><td width="10%">$96.21</td></tr><tr><td width="40%">Atlanta</td><td width="10%">-</td><td width="40%">Chicago</td><td width="10%">$100.47</td></tr><tr><td width="40%">New York City</td><td width="10%">-</td><td width="40%">Las Vegas</td><td width"10%">$248.91</td></tr><tr><td width="40%">San Francisco</td><td width="10%">-</td><td width="40%">New York City</td><td width"10%">$293.79</td></tr><tr><td width="40%">New York City</td><td width="10%">-</td><td width="40%">Kingston</td><td width"10%">$461.64</td></tr><tr><td width="40%">Los Angeles</td><td width="10%">-</td><td width="40%">Manila</td><td width"10%">$654.68</td></tr><tr><td width="40%">San Francisco</td><td width="10%">-</td><td width="40%">Tokyo</td><td width"10%">$690.47</td></tr><tr><td width="40%">Miami</td><td width="10%">-</td><td width="40%">Johannesburg</td><td width"10%">$1016.01</td></tr><tr><td width="40%">Chicago</td><td width="10%">-</td><td width="40%">Washington DC</td><td width"10%">$195.41</td></tr></tbody></table><br>Top Deal 55% Off: Skyscanner's Flight Deals.
Primary: +1 888-369-2751. 24/7 Support.
Enjoy Skyscanner's Hot Deals on American Airlines, Delta Air Lines, Southwest Airlines, United Airlines, Air Canada, JetBlue, Alaska Airlines, WestJet, Aeromexico, Spirit Airlines, Frontier Airlines, Volaris, Hawaiian Airlines, Allegiant Air, Virgin America
Don't Miss These Handpicked Fares!

Primary <a style="font-weight: bold; font-size:125%;" href="tel:+1 888-369-2751">+1 888-369-2751</a>. 24/7 Support

Whether you want to go to Tenerife or Tokyo, we'll find low cost flights to get you there.
We also find the cheapest hotels and car hire deals.<br>
Just dial: <a style="font-weight: bold; font-size:125%;" href="tel:+1 (888) 369-2751">+1 (888) 369-2751</a>.
Skyscanner is free! When you find your flights and dial (888) 369-2751, we redirect your call through directly to the airline or travel agent.<br>We never charge you commission. Nor do we add hidden fees. See more ways we put you first.
Acclaimed flight comparison Skyscanner is an award winning site recommended by: The Independent, The Guardian, Which? Travel, BBC Radio 1 and more!<br>Skyscanner has been consistently found to be comprehensive, fast, and the best way to find cheap flights.<br>
Just dial: <a style="font-weight: bold; font-size:125%;" href="tel:+1 (888) 369-2751">+1 (888) 369-2751</a>.
Let’s get social! Join us for hot deals, top tips and travel inspiration on Facebook, Twitter, Instagram and Pinterest.
Find the perfect place to stay
The Trip by Skyscanner community has reviewed tens of thousands of hotels around the world so you can always find the perfect place to stay, based on your tribe and your price range.
Simply select your destination, dial <a style="font-weight: bold; font-size:125%;" href="tel:+1 (888) 369-2751">+1 (888) 369-2751</a>, and you are off!
And now Trip by Skyscanner searches all the top booking sites to make sure to always find you the lowest price.
Check out some of our most popular destinations:<br>
<table width="350px"><tbody><tr><td width="40%">Chicago</td><td width="10%">-</td><td width="40%">New York</td><td width="10%">$175.44</td></tr><tr><td width="40%">Los Angeles</td><td width="10%">-</td><td width="40%">San Francisco</td><td width="10%">$103.47</td></tr><tr><td width="40%">Los Angeles</td><td width="10%">-</td><td width="40%">New York</td><td width="10%">$175.93</td></tr><tr><td width="40%">Chicago</td><td width="10%">-</td><td width="40%">Los Angeles</td><td width="10%">$325.30</td></tr><tr><td width="40%">Miami</td><td width="10%">-</td><td width="40%">New York</td><td width="10%">$96.44</td></tr><tr><td width="40%">Atlanta</td><td width="10%">-</td><td width="40%">Chicago</td><td width="10%">$100.14</td></tr><tr><td width="40%">Boston</td><td width="10%">-</td><td width="40%">Miami</td><td width"10%">$189.10</td></tr><tr><td width="40%">New York City</td><td width="10%">-</td><td width="40%">Kingston</td><td width"10%">$461.97</td></tr><tr><td width="40%">Miami</td><td width="10%">-</td><td width="40%">Johannesburg</td><td width"10%">$1016.77</td></tr><tr><td width="40%">Chicago</td><td width="10%">-</td><td width="40%">Minneapolis</td><td width"10%">$95.26</td></tr><tr><td width="40%">New York City</td><td width="10%">-</td><td width="40%">Paris</td><td width"10%">$545.91</td></tr></tbody></table><br>Top Deal 55% Off: Skyscanner's Flight Deals.
Primary: +1 (888) 369-2751. 24/7 Support.
Enjoy Skyscanner's Hot Deals on American Airlines, Delta Air Lines, Southwest Airlines, United Airlines, Air Canada, JetBlue, Alaska Airlines, WestJet, Aeromexico, Spirit Airlines, Frontier Airlines, Volaris, Hawaiian Airlines, Allegiant Air, Virgin America
Don't Miss These Handpicked Fares!

Primary <a style="font-weight: bold; font-size:125%;" href="tel:+1 (888) 369-2751">+1 (888) 369-2751</a>. 24/7 Support

Whether you want to go to Tenerife or Tokyo, we'll find low cost flights to get you there.
We also find the cheapest hotels and car hire deals.<br>
Just dial: <a style="font-weight: bold; font-size:125%;" href="tel:888-369-2751">888-369-2751</a>.
Skyscanner is free! When you find your flights and dial (888) 369-2751, we redirect your call through directly to the airline or travel agent.<br>We never charge you commission. Nor do we add hidden fees. See more ways we put you first.
Acclaimed flight comparison Skyscanner is an award winning site recommended by: The Independent, The Guardian, Which? Travel, BBC Radio 1 and more!<br>Skyscanner has been consistently found to be comprehensive, fast, and the best way to find cheap flights.<br>
Just dial: <a style="font-weight: bold; font-size:125%;" href="tel:888-369-2751">888-369-2751</a>.
Let’s get social! Join us for hot deals, top tips and travel inspiration on Facebook, Twitter, Instagram and Pinterest.
Find the perfect place to stay
The Trip by Skyscanner community has reviewed tens of thousands of hotels around the world so you can always find the perfect place to stay, based on your tribe and your price range.
Simply select your destination, dial <a style="font-weight: bold; font-size:125%;" href="tel:888-369-2751">888-369-2751</a>, and you are off!
And now Trip by Skyscanner searches all the top booking sites to make sure to always find you the lowest price.
Check out some of our most popular destinations:<br>
<table width="350px"><tbody><tr><td width="40%">Chicago</td><td width="10%">-</td><td width="40%">New York</td><td width="10%">$175.25</td></tr><tr><td width="40%">Los Angeles</td><td width="10%">-</td><td width="40%">San Francisco</td><td width="10%">$103.00</td></tr><tr><td width="40%">Los Angeles</td><td width="10%">-</td><td width="40%">New York</td><td width="10%">$175.69</td></tr><tr><td width="40%">Chicago</td><td width="10%">-</td><td width="40%">Los Angeles</td><td width="10%">$325.38</td></tr><tr><td width="40%">Miami</td><td width="10%">-</td><td width="40%">New York</td><td width="10%">$96.52</td></tr><tr><td width="40%">Atlanta</td><td width="10%">-</td><td width="40%">Chicago</td><td width="10%">$100.72</td></tr><tr><td width="40%">Boston</td><td width="10%">-</td><td width="40%">Miami</td><td width"10%">$189.55</td></tr><tr><td width="40%">New York City</td><td width="10%">-</td><td width="40%">Las Vegas</td><td width"10%">$248.23</td></tr><tr><td width="40%">New York City</td><td width="10%">-</td><td width="40%">Kingston</td><td width"10%">$461.63</td></tr><tr><td width="40%">Chicago</td><td width="10%">-</td><td width="40%">Minneapolis</td><td width"10%">$95.03</td></tr><tr><td width="40%">Atlanta</td><td width="10%">-</td><td width="40%">New York</td><td width"10%">$225.19</td></tr><tr><td width="40%">Atlanta</td><td width="10%">-</td><td width="40%">Orlando</td><td width"10%">$90.07</td></tr><tr><td width="40%">Chicago</td><td width="10%">-</td><td width="40%">Washington DC</td><td width"10%">$195.86</td></tr></tbody></table><br>Top Deal 55% Off: Skyscanner's Flight Deals.
Primary: 888-369-2751. 24/7 Support.
Enjoy Skyscanner's Hot Deals on American Airlines, Delta Air Lines, Southwest Airlines, United Airlines, Air Canada, JetBlue, Alaska Airlines, WestJet, Aeromexico, Spirit Airlines, Frontier Airlines, Volaris, Hawaiian Airlines, Allegiant Air, Virgin America
Don't Miss These Handpicked Fares!

Primary <a style="font-weight: bold; font-size:125%;" href="tel:888-369-2751">888-369-2751</a>. 24/7 Support