Metadata service sometimes fails to answer

Bug #757293 reported by Thierry Carrez
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Soren Hansen

Bug Description

Running with current trunk, starting an Ubuntu image, when cloud-init queries the metadata service requests sometime fail (about 2/3 of the time), with the following error:

Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.6/eventlet/wsgi.py", line 336, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/pymodules/python2.6/paste/urlmap.py", line 203, in __call__
    return app(environ, start_response)
  File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/pymodules/python2.6/nova/api/ec2/__init__.py", line 59, in __call__
    rv = req.get_response(self.application)
  File "/usr/lib/pymodules/python2.6/webob/request.py", line 919, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/pymodules/python2.6/webob/request.py", line 887, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/pymodules/python2.6/nova/api/ec2/metadatarequesthandler.py", line 74, in __call__
    meta_data = cc.get_metadata(remote_address)
  File "/usr/lib/pymodules/python2.6/nova/api/ec2/cloud.py", line 173, in get_metadata
    'instance-type': instance_ref['instance_type'],
  File "/usr/lib/pymodules/python2.6/nova/db/sqlalchemy/models.py", line 74, in __getitem__
    return getattr(self, key)
  File "/usr/lib/python2.6/dist-packages/sqlalchemy/orm/attributes.py", line 163, in __get__
    instance_dict(instance))
  File "/usr/lib/python2.6/dist-packages/sqlalchemy/orm/attributes.py", line 382, in get
    value = callable_(passive=passive)
  File "/usr/lib/python2.6/dist-packages/sqlalchemy/orm/strategies.py", line 578, in __call__
    (mapperutil.state_str(state), self.key)
DetachedInstanceError: Parent instance <Instance at 0x41039d0> is not bound to a Session; lazy load operation of attribute 'instance_type' cannot proceed

Related branches

Revision history for this message
Thierry Carrez (ttx) wrote :

Trace with sane linewrapping at http://pastebin.com/R0LRQSxe

Revision history for this message
Thierry Carrez (ttx) wrote :

On the guest side, that shows up as a delayed boot (every retry costs 2 seconds) with multiple lines like:

Caught exception reading instance data: http://169.254.169.254/2009-04-04/meta-data/instance-action
Caught exception reading instance data: http://169.254.169.254/2009-04-04/meta-data/instance-action
Caught exception reading instance data: http://169.254.169.254/2009-04-04/meta-data/public-ipv4

Thierry Carrez (ttx)
Changed in nova:
importance: Undecided → Medium
status: New → Confirmed
tags: added: regression
Revision history for this message
Thierry Carrez (ttx) wrote :

Attached branch fixes it for me.

Changed in nova:
assignee: nobody → Soren Hansen (soren)
milestone: none → cactus-rc
status: Confirmed → In Progress
Thierry Carrez (ttx)
tags: removed: regression
Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: cactus-rc → 2011.2
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.