LBaaSv2 with HAproxy Agent RPC timeout

Asked by Kollar Laszlo

I would like to Neutron LBaaSv2 to a working, multinode OpenStack Mitaka deployment. lbaas agent does not listed by neutron agent-list.
the lbaas-agent sends a message on the queue and times out waiting for a reply. On the other hand all the other components of openstack deployment work as expected, I can launch instances with network, can assign floating ip, ssh into it, etc.

(Notice: it is running on Centos7, not Ubuntu, Centos does not have lbaasv2-agent topic)

-- listed in ext-list --
neutron ext-list |grep lbaas
| lbaasv2 | LoadBalancing service v2 |
| lbaas_agent_schedulerv2 | Loadbalancer Agent Scheduler V2 |

------

-- lbaas message queues in rabbit --

n-lbaasv2-plugin_fanout_984d77d4f115458684682020884ca024 0
n-lbaasv2-plugin.amygdala.cloud.ceu.edu 0
n-lbaas_agent.thalamus.cloud.ceu.edu 0
n-lbaasv2-plugin 0
n-lbaasv2-plugin_fanout_d956a5ace5ca48ce999d3ddafd435b4d 0
n-lbaas_agent.amygdala.cloud.ceu.edu 0
n-lbaas_agent 0
n-lbaas_agent_fanout_96c7440b0ea04a4ebc17f8a0674469ad 0

--------------------------

--- lbaas-agent.log ----
2016-09-19 15:49:58.682 30267 INFO neutron.common.config [-] Logging enabled!
2016-09-19 15:49:58.682 30267 INFO neutron.common.config [-] /usr/bin/neutron-lbaas-agent version 8.1.2
2016-09-19 15:50:58.769 30267 ERROR neutron.common.rpc [-] Timeout in RPC method get_ready_devices. Waiting for 26 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.
2016-09-19 15:50:58.770 30267 WARNING neutron.common.rpc [-] Increasing timeout for get_ready_devices calls to 120 seconds. Restart the agent to restore it to the default value.
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager [-] Unable to retrieve ready devices
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager Traceback (most recent call last):
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/agent/agent_manager.py", line 152, in sync_state
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager ready_instances = set(self.plugin_rpc.get_ready_devices())
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/agent/agent_api.py", line 36, in get_ready_devices
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager return cctxt.call(self.context, 'get_ready_devices', host=self.host)
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 155, in call
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager time.sleep(wait)
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager self.force_reraise()
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager six.reraise(self.type_, self.value, self.tb)
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 136, in call
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager return self._original_context.call(ctxt, method, **kwargs)
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager retry=self.retry)
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager timeout=timeout, retry=retry)
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager retry=retry)
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager result = self._waiter.wait(msg_id, timeout)
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager message = self.waiters.get(msg_id, timeout=timeout)
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 'to message ID %s' % msg_id)
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager MessagingTimeout: Timed out waiting for a reply to message ID 886835ddb37c4906957cdb6045ca1baa
2016-09-19 15:51:24.563 30267 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager
2016-09-19 15:53:24.600 30267 ERROR neutron.common.rpc [-] Timeout in RPC method get_ready_devices. Waiting for 58 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.
2016-09-19 15:53:24.601 30267 WARNING neutron.common.rpc [-] Increasing timeout for get_ready_devices calls to 240 seconds. Restart the agent to restore it to the default value.
.
.
-------------------

Question information

Language:
English Edit question
Status:
Expired
For:
Ubuntu neutron-lbaas Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Launchpad Janitor (janitor) said :
#1

This question was expired because it remained in the 'Open' state without activity for the last 15 days.