Hello All,
I am running Havana on a Fedora 19 system and I am running into some issues in nova-compute when booting an instance. I am getting a "neutron client authentication failed: Could not find Service or Region in Service Catalog" error in the nova-compute log file (note 1). However, the endpoints do exist when run from the controller node (note 2). I am also able to use the neutron client to print previously created networks and all other related information (note 3), so I am not quite sure if it's the endpoint that is causing issues. Nova.conf appears to have the right settings to indicate where the neutron is running (note 4) while the api-paste.ini file also the right information (note 5). To verify, I ran keystone commands from the compute host and I am able to see the endpoints (note 6).
It does not appear to be the issue described in https://bugs.launchpad.net/nova/+bug/1241275 since the stack traces are different. From the looks of it, in my case, there is an attempt to get the token and it fails to locate even the endpoint. Any thoughts or suggestions in this regard are greatly appreciated.
Notes:
1. nova-compute log
2013-11-26 14:02:56.921 2493 ERROR nova.network.neutronv2 [req-662e2363-cdcb-495e-96c3-d9852fcfea59 None None] Neutron client authentication failed: Could not find Service or Region in Service Catalog.
2013-11-26 14:02:56.922 2493 ERROR nova.compute.manager [req-662e2363-cdcb-495e-96c3-d9852fcfea59 None None] Instance failed network setup after 1 attempt(s)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager Traceback (most recent call last):
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1228, in _allocate_network_async
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager dhcp_options=dhcp_options)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 49, in wrapper
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager res = f(self, context, *args, **kwargs)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 358, in allocate_for_instance
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager LOG.exception(msg, port_id)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 323, in allocate_for_instance
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager port_req_body)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 392, in _populate_neutron_extension_values
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager self._refresh_neutron_extensions_cache()
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 375, in _refresh_neutron_extensions_cache
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager neutron = neutronv2.get_client(context.get_admin_context())
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 71, in get_client
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager return _get_client(token=token)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 52, in _get_client
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager token = _get_auth_token()
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 47, in _get_auth_token
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager LOG.error(_('Neutron client authentication failed: %s'), e)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 43, in _get_auth_token
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager httpclient.authenticate()
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 245, in authenticate
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager self._extract_service_catalog(body)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 208, in _extract_service_catalog
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager endpoint_type=self.endpoint_type)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 82, in url_for
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager raise exceptions.EndpointNotFound()
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager EndpointNotFound: Could not find Service or Region in Service Catalog.
Note 2:
[root@drogon utils]# keystone endpoint-list
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+
| id | region | publicurl | internalurl | adminurl | service_id |
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+
| 44e10585f1e44d7f8d77d4ea54594517 | regionOne | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | 37ca06fa667a4148b0532566be4d6e77 |
| 7ee8df0d51204c27815fa36e9c95ac1d | regionOne | http://192.168.122.61:9292 | http://192.168.122.61:9292 | http://192.168.122.61:9292 | 3e7da61c8cd54c45bddc4ce09709c5bf |
| 95ecd68ab73643f1b90dc838f01b9d86 | regionOne | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | de825bed415b43fda2ef2af83c51a146 |
| d4282404b7674be69dee9143e1d4dcdd | regionOne | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Admin | c18c7e1643f241359b53c281b6609444 |
| d68b2d16f6624df3a0893d040a395ea7 | regionOne | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | f839d5dbbe7d4a30bba17e4ac7b62b22 |
| f91f74def62e4ee28900515ae7acf312 | regionOne | http://192.168.122.61:9696 | http://192.168.122.61:9696 | http://192.168.122.61:9696 | dab8d63d7341498e8414ae50f74d0aac |
| fd7405a77f804cf289a197a52a72b922 | regionOne | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:35357/v2.0 | 28fb944cbc154b02950fdccd4b912827 |
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+
Note 3: Printing net-list as both admin and demo users
[root@viserion utils]# neutron net-list
+--------------------------------------+----------+--------------------------------------------------+
| id | name | subnets |
+--------------------------------------+----------+--------------------------------------------------+
| 04676255-320d-4600-84b8-0e69b03928ac | demo-net | 537fe95c-1722-4007-af3e-3c5e91b80592 10.5.5.0/24 |
| eb0ddf1e-0251-4e2d-a7f8-edeef177da30 | ext-net | 86924ec1-84c5-4c20-85fe-05fcb2dbef41 7.7.7.0/24 |
+--------------------------------------+----------+--------------------------------------------------+
[root@viserion utils]# . ./keystone_demo_remote.bash
[root@viserion utils]# neutron net-list
+--------------------------------------+----------+--------------------------------------------------+
| id | name | subnets |
+--------------------------------------+----------+--------------------------------------------------+
| 04676255-320d-4600-84b8-0e69b03928ac | demo-net | 537fe95c-1722-4007-af3e-3c5e91b80592 10.5.5.0/24 |
| eb0ddf1e-0251-4e2d-a7f8-edeef177da30 | ext-net | 86924ec1-84c5-4c20-85fe-05fcb2dbef41 |
+--------------------------------------+----------+--------------------------------------------------+
Note 4: excerpt from nova.conf file on compute host
network_api_class = nova.network.neutronv2.api.API
neutron_url = http://192.168.122.61:9696
neutron_auth_strategy = keystone
neutron_admin_tenant = service
neutron_admin_username = neutron
neutron_admin_password = neutron
neutron_admin_auth_url = http://192.168.122.61:35357/v2.0
firewall_driver = nova.virt.firewall.IptablesFirewallDriver
security_group_api = neutron
compute_driver = nova.virt.libvirt.LibvirtDriver
Note 5: excerpt from api-paste.ini file on the compute host
[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
auth_host = 192.168.122.61
auth_protocol = http
auth_port = 35357
admin_user = nova
admin_tenant_name = service
admin_password = nova
Note 6: output of keystone command from compute host
[root@viserion utils]# keystone endpoint-list
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+
| id | region | publicurl | internalurl | adminurl | service_id |
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+
| 44e10585f1e44d7f8d77d4ea54594517 | regionOne | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | 37ca06fa667a4148b0532566be4d6e77 |
| 7ee8df0d51204c27815fa36e9c95ac1d | regionOne | http://192.168.122.61:9292 | http://192.168.122.61:9292 | http://192.168.122.61:9292 | 3e7da61c8cd54c45bddc4ce09709c5bf |
| 95ecd68ab73643f1b90dc838f01b9d86 | regionOne | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | de825bed415b43fda2ef2af83c51a146 |
| d4282404b7674be69dee9143e1d4dcdd | regionOne | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Admin | c18c7e1643f241359b53c281b6609444 |
| d68b2d16f6624df3a0893d040a395ea7 | regionOne | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | f839d5dbbe7d4a30bba17e4ac7b62b22 |
| f91f74def62e4ee28900515ae7acf312 | regionOne | http://192.168.122.61:9696 | http://192.168.122.61:9696 | http://192.168.122.61:9696 | dab8d63d7341498e8414ae50f74d0aac |
| fd7405a77f804cf289a197a52a72b922 | regionOne | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:35357/v2.0 | 28fb944cbc154b02950fdccd4b912827 |
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+
[root@viserion nova]# keystone endpoint-get --service network
+-------------------+----------------------------+
| Property | Value |
+-------------------+----------------------------+
| network.publicURL | http://192.168.122.61:9696 |
+-------------------+----------------------------+