Openstack Dashboard Error : network list cannot be retrieved

Asked by Chattar Singh Rathore on 2019-12-26

Hello Team,
I have installed openstack packstack in multinodes [four node].
OS : CentOS 7 (Fresh OS) and done all prerequisite as per openstack packstack guidelines.
====Openstack installed successfully. ====

After login dashboard giving some repeatedly errors such as:
Error: Unable to retrieve instances.
Error: Unable to retrieve the network list.

After restarting nova and network services trying to click on like delete instance then its repeating the same errors and nothing showing into dashboard.

Kindly help me to resolve this issue.
Thank you in advance.

Question information

Language:
English Edit question
Status:
Open
For:
OpenStack Dashboard (Horizon) Edit question
Assignee:
No assignee Edit question
Last query:
2020-01-11
Last reply:
2020-01-11

here i am giving nova-compute logs
var/log/nova/nova-compute.log

2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-0b3fb936-3d99-47fa-a83a-68667da6361c']
2019-12-26 12:51:28.393 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-41359b10-b792-4ccc-93cd-7fa3fb9b6be6']
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-41359b10-b792-4ccc-93cd-7fa3fb9b6be6']
2019-12-26 12:52:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-b409002a-1824-48d8-9d66-7cf431f6d982']
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-b409002a-1824-48d8-9d66-7cf431f6d982']
2019-12-26 12:53:30.569 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-55e6f301-ecd3-467c-8237-ef34d4f7f8e0']
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-55e6f301-ecd3-467c-8237-ef34d4f7f8e0']
2019-12-26 12:54:30.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-ccfa976b-086d-4f84-bfe9-c5aec2830d97']
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-ccfa976b-086d-4f84-bfe9-c5aec2830d97']
2019-12-26 12:55:31.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-a7f0335b-1887-48fd-b1cd-afbc27af36a7']
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-a7f0335b-1887-48fd-b1cd-afbc27af36a7']
2019-12-26 12:56:33.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-0f48d1c9-da73-464f-bd63-3c6695440464']
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-0f48d1c9-da73-464f-bd63-3c6695440464']
2019-12-26 12:57:33.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-9a0207d3-5de1-4329-955e-51e5fd03368f']
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-9a0207d3-5de1-4329-955e-51e5fd03368f']
2019-12-26 12:58:34.871 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-a02a48f1-181d-4766-9322-ee01db865cd8']
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-a02a48f1-181d-4766-9322-ee01db865cd8']
2019-12-26 12:59:34.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-b6a5dd47-7d60-45de-908c-f100195e07b0']
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-b6a5dd47-7d60-45de-908c-f100195e07b0']
2019-12-26 13:00:36.382 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:00:46.798 6141 ERROR nova.scheduler.client.report [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [None] Failed to retrieve traits from placement API for resource provider with UUID 91acf39e-074a-4270-9b54-48fd05c2489e. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
.
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] Error updating resources for node dc.openstack.controller.: ResourceProviderTraitRetrievalFailed: Failed to get traits for resource provider with UUID 91acf39e-074a-4270-9b54-48fd05c2489e
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager Traceback (most recent call last):
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8148, in _update_available_resource_for_node
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager startup=startup)
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 748, in update_available_resource
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup)
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager return f(*args, **kwargs)
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 829, in _update_available_resource
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager self._update(context, cn, startup=startup)
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 1036, in _update
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup)
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 68, in wrapped_f
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw)
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 223, in call
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager return attempt.get(self._wrap_exception)
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2])
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 962, in _update_to_placement
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname)
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 873, in get_provider_tree_and_ensure_root
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 681, in _ensure_resource_provider
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager self._refresh_associations(context, uuid_to_refresh, force=True)
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 793, in _refresh_associations
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager trait_info = self.get_provider_traits(context, rp_uuid)
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 401, in get_provider_traits
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager raise exception.ResourceProviderTraitRetrievalFailed(uuid=rp_uuid)
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager ResourceProviderTraitRetrievalFailed: Failed to get traits for resource provider with UUID 91acf39e-074a-4270-9b54-48fd05c2489e
2019-12-26 13:00:46.799 6141 ERROR nova.compute.manager
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-6d009738-6cfa-4397-b7d5-de82b0e63acd']
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-6d009738-6cfa-4397-b7d5-de82b0e63acd']
2019-12-26 13:01:37.388 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:01:44.986 6141 ERROR nova.scheduler.client.report [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 91acf39e-074a-4270-9b54-48fd05c2489e. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
.
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] Error updating resources for node dc.openstack.controller.: ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 91acf39e-074a-4270-9b54-48fd05c2489e
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager Traceback (most recent call last):
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8148, in _update_available_resource_for_node
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager startup=startup)
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 748, in update_available_resource
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup)
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager return f(*args, **kwargs)
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 829, in _update_available_resource
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager self._update(context, cn, startup=startup)
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 1036, in _update
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup)
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 68, in wrapped_f
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw)
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 223, in call
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager return attempt.get(self._wrap_exception)
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2])
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 962, in _update_to_placement
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname)
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 873, in get_provider_tree_and_ensure_root
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 655, in _ensure_resource_provider
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager rps_to_refresh = self._get_providers_in_tree(context, uuid)
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 71, in wrapper
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager return f(self, *a, **k)
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 522, in _get_providers_in_tree
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 91acf39e-074a-4270-9b54-48fd05c2489e
2019-12-26 13:01:44.987 6141 ERROR nova.compute.manager
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-0419cb7b-4f86-4f8d-bc12-7436fb0a5cc9']
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-0419cb7b-4f86-4f8d-bc12-7436fb0a5cc9']
2019-12-26 13:02:38.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-300ce412-0884-4fa0-b16b-7d7ccb3fc002']
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-300ce412-0884-4fa0-b16b-7d7ccb3fc002']
2019-12-26 13:03:39.582 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-a8f5a4cc-4442-413a-a7e6-1ab01d2eb6cd']
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-a8f5a4cc-4442-413a-a7e6-1ab01d2eb6cd']
2019-12-26 13:04:40.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-4e74c669-9912-4b6e-95a1-2fe4ec40132e']
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-4e74c669-9912-4b6e-95a1-2fe4ec40132e']
2019-12-26 13:05:42.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:05:47.825 6141 ERROR nova.compute.resource_tracker [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] Skipping removal of allocations for deleted instances: Failed to retrieve allocations for resource provider 91acf39e-074a-4270-9b54-48fd05c2489e: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
: ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 91acf39e-074a-4270-9b54-48fd05c2489e: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-9b296b4e-8e63-4f13-adc1-2f0036cdda61']
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-9b296b4e-8e63-4f13-adc1-2f0036cdda61']
2019-12-26 13:06:42.627 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-b5d8a54d-0898-4c35-b002-99fc26d6ec7d']
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-b5d8a54d-0898-4c35-b002-99fc26d6ec7d']
2019-12-26 13:07:44.386 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-6bd1782a-9ca1-4a21-8567-9ed254fc491a']
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-6bd1782a-9ca1-4a21-8567-9ed254fc491a']
2019-12-26 13:08:44.579 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-15559106-1548-44bc-9f39-aebe0878e4e3']
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-15559106-1548-44bc-9f39-aebe0878e4e3']
2019-12-26 13:09:45.391 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-3636dba3-f89d-4651-aa2c-19c7a4b18632']
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-3636dba3-f89d-4651-aa2c-19c7a4b18632']
2019-12-26 13:10:45.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-22a54259-3043-409d-adf3-5f98afcfbef0']
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-22a54259-3043-409d-adf3-5f98afcfbef0']
2019-12-26 13:11:45.390 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-9346ef78-85a0-4417-91c8-b500f64e2c78']
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-9346ef78-85a0-4417-91c8-b500f64e2c78']
2019-12-26 13:12:47.392 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:12:52.049 6141 ERROR nova.scheduler.client.report [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 91acf39e-074a-4270-9b54-48fd05c2489e. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
.
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] Error updating resources for node dc.openstack.controller.: ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 91acf39e-074a-4270-9b54-48fd05c2489e
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager Traceback (most recent call last):
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8148, in _update_available_resource_for_node
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager startup=startup)
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 748, in update_available_resource
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup)
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager return f(*args, **kwargs)
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 829, in _update_available_resource
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager self._update(context, cn, startup=startup)
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 1036, in _update
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup)
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 68, in wrapped_f
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw)
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 223, in call
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager return attempt.get(self._wrap_exception)
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2])
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 962, in _update_to_placement
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname)
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 873, in get_provider_tree_and_ensure_root
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 655, in _ensure_resource_provider
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager rps_to_refresh = self._get_providers_in_tree(context, uuid)
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 71, in wrapper
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager return f(self, *a, **k)
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 522, in _get_providers_in_tree
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 91acf39e-074a-4270-9b54-48fd05c2489e
2019-12-26 13:12:52.050 6141 ERROR nova.compute.manager
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-ffa6393b-6e80-42b6-98b3-5d4219eb3f76']
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-ffa6393b-6e80-42b6-98b3-5d4219eb3f76']
2019-12-26 13:13:47.559 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:13:53.856 6141 ERROR nova.compute.resource_tracker [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] Skipping removal of allocations for deleted instances: Failed to retrieve allocations for resource provider 91acf39e-074a-4270-9b54-48fd05c2489e: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
: ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 91acf39e-074a-4270-9b54-48fd05c2489e: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-f719c311-49a0-49e1-ba3c-2c4391dc230b']
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-f719c311-49a0-49e1-ba3c-2c4391dc230b']
2019-12-26 13:14:47.384 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] Error updating resources for node dc.openstack.controller.: RemoteError: Remote error: OperationalError (pymysql.err.OperationalError) (1040, u'Too many connections') (Background on this error at: http://sqlalche.me/e/e3q8)
[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 135, in _object_dispatch\n return getattr(target, method)(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper\n result = fn(cls, context, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1353, in get_by_filters\n use_slave=use_slave, sort_keys=sort_keys, sort_dirs=sort_dirs)\n', u' File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 221, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1342, in _get_by_filters_impl\n marker=marker, columns_to_join=_expected_cols(expected_attrs))\n', u' File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 788, in instance_get_all_by_filters\n columns_to_join=columns_to_join)\n', u' File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 179, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 266, in wrapped\n return f(context, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2058, in instance_get_all_by_filters\n sort_dirs=[sort_dir])\n', u' File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 179, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 266, in wrapped\n return f(context, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2295, in instance_get_all_by_filters_sort\n return _instances_fill_metadata(context, query_prefix.all(), manual_joins)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, in all\n return list(self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3081, in __iter__\n return self._execute_and_instances(context)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3103, in _execute_and_instances\n querycontext, self._connection_from_session, close_with_result=True\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3111, in _get_bind_args\n mapper=self._bind_mapper(), clause=querycontext.statement, **kw\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3096, in _connection_from_session\n conn = self.session.connection(**kw)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1120, in connection\n execution_options=execution_options,\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1126, in _connection_for_bind\n engine, execution_options\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 424, in _connection_for_bind\n conn = bind.contextual_connect()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2192, in contextual_connect\n self._wrap_pool_connect(self.pool.connect, None),\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2232, in _wrap_pool_connect\n e, dialect, self\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1526, in _handle_dbapi_exception_noconnection\n util.raise_from_cause(newraise, exc_info)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause\n reraise(type(exception), exception, tb=exc_tb, cause=cause)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect\n return fn()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 425, in connect\n return _ConnectionFairy._checkout(self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 822, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 554, in checkout\n rec = pool._do_get()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1250, in _do_get\n self._dec_overflow()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1247, in _do_get\n return self._create_connection()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 370, in _create_connection\n return _ConnectionRecord(self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 499, in __init__\n self.__connect(first_connect_check=True)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 701, in __connect\n connection = pool._invoke_creator(self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect\n return dialect.connect(*cargs, **cparams)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 437, in connect\n return self.dbapi.connect(*cargs, **cparams)\n', u' File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 94, in Connect\n return Connection(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 327, in __init__\n self.connect()\n', u' File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 598, in connect\n self._request_authentication()\n', u' File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 862, in _request_authentication\n auth_packet = self._process_auth(plugin_name, auth_packet)\n', u' File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 933, in _process_auth\n pkt = self._read_packet()\n', u' File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, in _read_packet\n packet.check_error()\n', u' File "/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in check_error\n err.raise_mysql_exception(self._data)\n', u' File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception\n raise errorclass(errno, errval)\n', u"OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections') (Background on this error at: http://sqlalche.me/e/e3q8)\n"].
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager Traceback (most recent call last):
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8148, in _update_available_resource_for_node
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager startup=startup)
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 732, in update_available_resource
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager resources = self.driver.get_available_resource(nodename)
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7117, in get_available_resource
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager disk_over_committed = self._get_disk_over_committed_size_total()
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8635, in _get_disk_over_committed_size_total
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager ctx, filters, use_slave=True)
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager args, kwargs)
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 241, in object_class_action_versions
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager args=args, kwargs=kwargs)
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 178, in call
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager retry=self.retry)
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 128, in _send
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager retry=retry)
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager call_monitor_timeout, retry=retry)
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager raise result
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager RemoteError: Remote error: OperationalError (pymysql.err.OperationalError) (1040, u'Too many connections') (Background on this error at: http://sqlalche.me/e/e3q8)
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 135, in _object_dispatch\n return getattr(target, method)(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper\n result = fn(cls, context, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1353, in get_by_filters\n use_slave=use_slave, sort_keys=sort_keys, sort_dirs=sort_dirs)\n', u' File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 221, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1342, in _get_by_filters_impl\n marker=marker, columns_to_join=_expected_cols(expected_attrs))\n', u' File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 788, in instance_get_all_by_filters\n columns_to_join=columns_to_join)\n', u' File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 179, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 266, in wrapped\n return f(context, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2058, in instance_get_all_by_filters\n sort_dirs=[sort_dir])\n', u' File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 179, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 266, in wrapped\n return f(context, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2295, in instance_get_all_by_filters_sort\n return _instances_fill_metadata(context, query_prefix.all(), manual_joins)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, in all\n return list(self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3081, in __iter__\n return self._execute_and_instances(context)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3103, in _execute_and_instances\n querycontext, self._connection_from_session, close_with_result=True\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3111, in _get_bind_args\n mapper=self._bind_mapper(), clause=querycontext.statement, **kw\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3096, in _connection_from_session\n conn = self.session.connection(**kw)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1120, in connection\n execution_options=execution_options,\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1126, in _connection_for_bind\n engine, execution_options\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 424, in _connection_for_bind\n conn = bind.contextual_connect()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2192, in contextual_connect\n self._wrap_pool_connect(self.pool.connect, None),\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2232, in _wrap_pool_connect\n e, dialect, self\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1526, in _handle_dbapi_exception_noconnection\n util.raise_from_cause(newraise, exc_info)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause\n reraise(type(exception), exception, tb=exc_tb, cause=cause)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect\n return fn()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 425, in connect\n return _ConnectionFairy._checkout(self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 822, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 554, in checkout\n rec = pool._do_get()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1250, in _do_get\n self._dec_overflow()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1247, in _do_get\n return self._create_connection()\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 370, in _create_connection\n return _ConnectionRecord(self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 499, in __init__\n self.__connect(first_connect_check=True)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 701, in __connect\n connection = pool._invoke_creator(self)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect\n return dialect.connect(*cargs, **cparams)\n', u' File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 437, in connect\n return self.dbapi.connect(*cargs, **cparams)\n', u' File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 94, in Connect\n return Connection(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 327, in __init__\n self.connect()\n', u' File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 598, in connect\n self._request_authentication()\n', u' File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 862, in _request_authentication\n auth_packet = self._process_auth(plugin_name, auth_packet)\n', u' File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 933, in _process_auth\n pkt = self._read_packet()\n', u' File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, in _read_packet\n packet.check_error()\n', u' File "/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in check_error\n err.raise_mysql_exception(self._data)\n', u' File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception\n raise errorclass(errno, errval)\n', u"OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections') (Background on this error at: http://sqlalche.me/e/e3q8)\n"].
2019-12-26 13:14:55.422 6141 ERROR nova.compute.manager
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-6587bfad-921e-4795-934c-d51b8f3e6ae1']
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-6587bfad-921e-4795-934c-d51b8f3e6ae1']
2019-12-26 13:15:47.477 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-b1a63211-f56e-4145-a9db-22433175fc47']
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-b1a63211-f56e-4145-a9db-22433175fc47']
2019-12-26 13:16:49.380 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-5257f295-1d7a-4646-826b-d3dd81b758eb']
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-5257f295-1d7a-4646-826b-d3dd81b758eb']
2019-12-26 13:17:50.381 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-e1727888-2715-437c-bd0c-b8a9877e2702']
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-e1727888-2715-437c-bd0c-b8a9877e2702']
2019-12-26 13:18:50.567 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [req-92d1e32e-31c1-4f60-9218-693a118434da - - - - -] [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] An error occurred while refreshing the network cache.: InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-539c7b58-3802-4cda-8bde-5fa5051e417f']
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Traceback (most recent call last):
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7475, in _heal_instance_info_cache
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._require_nw_info_update(context, instance):
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7382, in _require_nw_info_update
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ports = self.network_api.list_ports(context, **search_opts)
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1788, in list_ports
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] return get_client(context).list_ports(**search_opts)
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] **_params)
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] for r in self._pagination(collection, path, **params):
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] res = self.get(path, params=params)
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] headers=headers, params=params)
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] self._handle_fault_response(status_code, replybody, resp)
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 129, in wrapper
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] ret = obj(*args, **kwargs)
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] exception_handler_v20(status_code, error_body)
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] request_ids=request_ids)
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] InternalServerError: Request Failed: internal server error while processing your request.
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000] Neutron server returns request_ids: ['req-539c7b58-3802-4cda-8bde-5fa5051e417f']
2019-12-26 13:19:52.389 6141 ERROR nova.compute.manager [instance: 150ac0fd-cd81-42da-ac31-a0fb3d7ef000]

Anyone can help please reply..

Launchpad Janitor (janitor) said : #3

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Dear Janitor,
I still facing the same issue and the problem is not resolved yet.
Question is open not expired, If you have knowledge kindly answer this question, please.

Matthias Runge (mrunge) said : #5

The janitor is a bot. You can not prevent it from cleaning/closing bugs.

What you can or should do is to verify that horizon can connect to the endpoints returned by openstack endpoint list (or so, you need to look that command up). Use telnet from the node running horizon to test that. All of the endpoints need to answer. In your case, it hints that either nova or neutron (or both) are misbehaving.

There is not much horizon can do here, as it relies on answers from underlying services. If any of those fails, you'll see an issue in horizon and blame it for that.

Dear Matthias Runge,
Thanks for replying.
In the dashboard after click on any object its giving such error and after that error unable to access the dashboard.
Kindly suggest that what should i do. I already attached logs herewith.

Can you help with this problem?

Provide an answer of your own, or ask Chattar Singh Rathore for more information if necessary.

To post a message you must log in.