OpenStack Compute (nova)

nova-compute 2015.1.1 InstanceList Fail

Asked by Alan Civita

Hi,
upgraded yesterday nova-* components from version 1:2015.1.0-0ubuntu1 to version 1:2015.1.1-0ubuntu1.
On controller first and compute nodes then...
a complete disaster,
nova-compute process on computer node don't want to start anymore!

i always get IncompatibleObjectVersion: Version 1.16 of InstanceList is not supported

Checked nova version on compute and controller node..they are the same:

Controller:
nova-manage --version
2015.1.1
nova-manage db version
280
nova --version
2.22.0

Compute Node:
nova-manage --version
2015.1.1
nova-manage db version
280
nova --version
2.22.0

All was working fine before this "minor" update.

I also have strange behaviour on neutron agent-list command...agents seems to go up and down.

Could it be rabbit related issue?
Rabbit is at the latest version available.

Any tips?

from /var/log/nova/nova-compute

 File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 224, in obj_class_from_name
    supported=latest_ver)

IncompatibleObjectVersion: Version 1.16 of InstanceList is not supported

2015-08-30 13:31:37.771 19405 ERROR nova.openstack.common.threadgroup [req-09be3325-1e25-47f0-830b-f537a14a37d1 - - - - -] Version 1.16 of InstanceList is not supported
Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
    incoming.message))

  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)

  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 1043, in object_class_action
    objver, args, kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 605, in object_class_action
    objver)

  File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 224, in obj_class_from_name
    supported=latest_ver)

IncompatibleObjectVersion: Version 1.16 of InstanceList is not supported
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 145, in wait
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup x.wait()
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 47, in wait
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 502, in run_service
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup service.start()
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/service.py", line 164, in start
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup self.manager.init_host()
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1273, in init_host
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup context, self.host, expected_attrs=['info_cache'])
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 161, in wrapper
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup args, kwargs)
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 335, in object_class_action
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup objver=objver, args=args, kwargs=kwargs)
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup retry=self.retry)
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup timeout=timeout, retry=retry)
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup retry=retry)
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup raise result
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup IncompatibleObjectVersion_Remote: Version 1.16 of InstanceList is not supported
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup incoming.message))
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup return self._do_dispatch(endpoint, method, ctxt, args)
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup result = getattr(endpoint, method)(ctxt, **new_args)
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 1043, in object_class_action
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup objver, args, kwargs)
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 605, in object_class_action
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup objver)
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 224, in obj_class_from_name
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup supported=latest_ver)
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup IncompatibleObjectVersion: Version 1.16 of InstanceList is not supported
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup
2015-08-30 13:31:37.771 19405 TRACE nova.openstack.common.threadgroup

Question information

Language:
English Edit question
Status:
Expired
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Launchpad Janitor (janitor) said : 		#1

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Revision history for this message
Xabier (xelkano) said : 		#2

Hi Alan,

I got the same message on compute nodes after upgrading from JUNO to KILO in Ubuntu 14.04. Did you manage to solve this problem?

Revision history for this message
Alan Civita (alan-civita2) said : 		#3

Hi Xavier,
well at the end i've solved it by upgrading rabbitmq to the latest version.
also i suggest to check the permissions of the nova-compute.log file - i've found the file owned by root.root which prevented nova-compute to start correctly.
Let me know if you need any further informations

Alan