lbaas Unable to retrieve ready devices

Asked by lee jian

on havana release,I install lbaas agent as it recommended step by step:
http://docs.openstack.org/network-admin/admin/content/install_neutron-lbaas-agent.html

but when I try to visit the service from horizon,an error occured,there is the log information in
/var/log/neutron/lbaas-agent.log
....................................................................................................................................................................................................

 ERROR neutron.services.loadbalancer.drivers.haproxy.agent_manager [-] Unable to retrieve ready devices
2013-11-06 14:31:09.457 7434 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Tble to retrieve ready devices
raceback (most recent call last):
2013-11-06 14:31:09.457 7434 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/usr/lib/python2.7/dist-packages/neutron/services/loadbalancer/drivers/haproxy/agent_manager.py", line 212, in sync_state
2013-11-06 14:31:09.457 7434 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager ready_logical_devices = set(self.plugin_rpc.get_ready_devices())
2013-11-06 14:31:09.457 7434 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/usr/lib/python2.7/dist-packages/neutron/services/loadbalancer/drivers/haproxy/agent_api.py", line 36, in get_ready_devices
2013-11-06 14:31:09.457 7434 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager topic=self.topic
2013-11-06 14:31:09.457 7434 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/proxy.py", line 130, in call
2013-11-06 14:31:09.457 7434 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager exc.info, real_topic, msg.get('method'))
2013-11-06 14:31:09.457 7434 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Timeout: Timeout while waiting on RPC response - topic: "q-lbaas-process-on-host", RPC method: "get_ready_devices" info: "<unknown>"
..................................................................................................................................................................................................................
can someone tell me what cause the error,and how to solve it?

Question information

Language:
English Edit question
Status:
Answered
For:
neutron Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
yong sheng gong (gongysh) said :
#1

can u try to increase the rpc_response_timeout in neutron.conf and have a try then?

how many compute nodes in your system? how many neutron agents in all? This should not happen in light load system.

to reduce the rpc message, try to increase report_interval and agent_down_time.

Revision history for this message
lee jian (leejian0612) said :
#2

Thanks very much!
after I update the system, I can't reproduce this error now,but I meet another question which may be ubuntu's issue
https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1240712

Revision history for this message
Ryan O'Hara (rohara) said :
#3

I get this error and related trace messages each time I deploy Havana with LBaaS enabled.

2013-12-05 16:57:10.243 16238 ERROR neutron.services.loadbalancer.drivers.haproxy.agent_manager [-] Unable to retrieve ready devices
2013-12-05 16:57:10.243 16238 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Traceback (most recent call last):
2013-12-05 16:57:10.243 16238 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/usr/lib/python2.6/site-packages/neutron/services/loadbalancer/drivers/haproxy/agent_manager.py", line 212, in sync_state
2013-12-05 16:57:10.243 16238 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager ready_logical_devices = set(self.plugin_rpc.get_ready_devices())
2013-12-05 16:57:10.243 16238 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/usr/lib/python2.6/site-packages/neutron/services/loadbalancer/drivers/haproxy/agent_api.py", line 36, in get_ready_devices
2013-12-05 16:57:10.243 16238 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager topic=self.topic
2013-12-05 16:57:10.243 16238 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/proxy.py", line 130, in call
2013-12-05 16:57:10.243 16238 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager exc.info, real_topic, msg.get('method'))
2013-12-05 16:57:10.243 16238 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Timeout: Timeout while waiting on RPC response - topic: "q-lbaas-process-on-host", RPC method: "get_ready_devices" info: "<unknown>"
2013-12-05 16:57:10.243 16238 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager

This appears to be the exact same issue as the reporter experience. If I restart the neutron-lbaas-agent I no longer see this error.

Revision history for this message
Eugene Nikanorov (enikanorov) said :
#4

Could you provide neutron-server logs during this error is seen on the agent?

Revision history for this message
Ryan O'Hara (rohara) said :
#5

http://paste.openstack.org/show/54595/

Notice that the lbaas log reports the error right around the time the neutron server connects to the RPC server.

Revision history for this message
ephem (tpiperatgod) said :
#6

It mostly caused by configuration problems.

Pls check the conifg items below:

service_provider in /etc/neutron/neutron_lbaas.conf
device_driver in /etc/neutron/lbaas_agent.ini
service_plugins in /etc/neutron/neutron.conf

The config would look like below if use v1 plugin:
service_provider = LOADBALANCER:Haproxy:neutron_lbaas.services.loadbalancer.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default
device_driver = neutron.services.loadbalancer.drivers.haproxy.namespace_driver.HaproxyNSDriver
service_plugins = lbaas

and use v2 plugin:
service_provider = LOADBALANCERV2:Haproxy:neutron_lbaas.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default
device_driver = neutron_lbaas.drivers.haproxy.namespace_driver.HaproxyNSDriver
service_plugins = lbaasv2(you may need to config the lbaasv2 path in neutron's entry_points.txt)

Can you help with this problem?

Provide an answer of your own, or ask lee jian for more information if necessary.

To post a message you must log in.