Timeout: Timeout while waiting on RPC response.

Asked by Maharajan Nachiappan

I am running a heat list command, it fails intermittently and works other time. It seems like rabbit mq having some issue on receiving the message from the heat-api. But anyone facing issues like these and suggestion would be appreciated.

time heat list
Traceback (most recent call last):
  File "/opt/openstack-heat/lib64/python2.6/site-packages/eventlet/wsgi.py", line 384, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/heat-2012.2.3-py2.6.egg/heat/common/wsgi.py", line 305, in __call__
    response = req.get_response(self.application)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/heat-2012.2.3-py2.6.egg/heat/common/auth_token.py", line 176, in __call__
    return self.app(env, start_response)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/heat-2012.2.3-py2.6.egg/heat/common/wsgi.py", line 305, in __call__
    response = req.get_response(self.application)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/heat-2012.2.3-py2.6.egg/heat/common/wsgi.py", line 563, in __call__
    request, **action_args)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/heat-2012.2.3-py2.6.egg/heat/common/wsgi.py", line 605, in dispatch
    return method(*args, **kwargs)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/heat-2012.2.3-py2.6.egg/heat/api/openstack/v1/util.py", line 30, in handle_stack_method
    return handler(controller, req, **kwargs)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/heat-2012.2.3-py2.6.egg/heat/api/openstack/v1/stacks.py", line 155, in index
    stacks = self.engine.list_stacks(req.context)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/heat-2012.2.3-py2.6.egg/heat/rpc/client.py", line 79, in list_stacks
    topic=_engine_topic(self.topic, ctxt, None))
  File "/opt/openstack-heat/lib64/python2.6/site-packages/heat-2012.2.3-py2.6.egg/heat/openstack/common/rpc/proxy.py", line 80, in call
    return rpc.call(context, self._get_topic(topic), msg, timeout)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/heat-2012.2.3-py2.6.egg/heat/openstack/common/rpc/__init__.py", line 140, in call
    return _get_impl().call(CONF, context, topic, msg, timeout)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/heat-2012.2.3-py2.6.egg/heat/openstack/common/rpc/impl_kombu.py", line 798, in call
    rpc_amqp.get_connection_pool(conf, Connection))
  File "/opt/openstack-heat/lib64/python2.6/site-packages/heat-2012.2.3-py2.6.egg/heat/openstack/common/rpc/amqp.py", line 612, in call
    rv = list(rv)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/heat-2012.2.3-py2.6.egg/heat/openstack/common/rpc/amqp.py", line 554, in __iter__
    self.done()
  File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
    self.gen.next()
  File "/opt/openstack-heat/lib64/python2.6/site-packages/heat-2012.2.3-py2.6.egg/heat/openstack/common/rpc/amqp.py", line 551, in __iter__
    self._iterator.next()
  File "/opt/openstack-heat/lib64/python2.6/site-packages/heat-2012.2.3-py2.6.egg/heat/openstack/common/rpc/impl_kombu.py", line 648, in iterconsume
    yield self.ensure(_error_callback, _consume)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/heat-2012.2.3-py2.6.egg/heat/openstack/common/rpc/impl_kombu.py", line 566, in ensure
    error_callback(e)
  File "/opt/openstack-heat/lib64/python2.6/site-packages/heat-2012.2.3-py2.6.egg/heat/openstack/common/rpc/impl_kombu.py", line 629, in _error_callback
    raise rpc_common.Timeout()
Timeout: Timeout while waiting on RPC response.

real 1m0.650s
user 0m0.282s
sys 0m0.077s

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Heat Edit question
Assignee:
No assignee Edit question
Solved by:
Steve Baker
Solved:
Last query:
Last reply:
Revision history for this message
Best Steve Baker (steve-stevebaker) said :
#1

This is usually because there is too much load for heat-engine to respond before the timeout. The load could be actually caused by heat-engine, or anything else on that host, so it is hard to say specifically what your problem is.

Revision history for this message
Maharajan Nachiappan (mnachiappan) said :
#2

HEAT-LIst hangs, when I created 70+ stacks with. I hope it has scalable issue. Even nova says no quota exists, still heat create the stack, this creates issue. When I cleaned HEAT DB down to 20 stack, becomes normal. Any thoughts on this?