Nova completely stuck while giving some commands

Asked by ColOfAbRiX on 2013-01-26

Hello! I'm setting up a system with openstack for the frist time and i'm becoming really crazy with this problem.
After the installation everything was fine, I did my test and when i wanted to delete an instance suddenly the problem appeared. I gave the following command with no result, meaning it was waiting to complete. Ctrl+C stopped nova with this log:

fabrizio@uranus:/etc/nova$ nova delete ef61492b-14b6-4b43-9f49-51e6f152e26d

^CTraceback (most recent call last):
  File "/usr/bin/nova", line 9, in <module>
    load_entry_point('python-novaclient==2.9.0', 'console_scripts', 'nova')()
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 531, in main
    OpenStackComputeShell().main(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 467, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py", line 1008, in do_delete
    _find_server(cs, args.server).delete()
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 41, in delete
    self.manager.delete(self)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 528, in delete
    self._delete("/servers/%s" % base.getid(server))
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 157, in _delete
    _resp, _body = self.api.client.delete(url)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 248, in delete
    return self._cs_request(url, 'DELETE', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 226, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 208, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 182, in request
    resp, body = super(HTTPClient, self).request(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1543, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1293, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1263, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1034, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline()
  File "/usr/lib/python2.7/socket.py", line 430, in readline
    data = recv(1)
KeyboardInterrupt

I obtained the same result with other commands like nova boot or nova stop. I analyzed log files and looked inside socket.py with no result os useful informations. Playing with services I noticed that nova-api is involved in this problem but I don't know what exactly is.
I think there could be something misconfigured or some locked file, but I really don't know. I also tried do uninstall all the nova components and their dependencies and the first time it worked, then the problem appeared again and again I uninstalled everything but this time it didn't fix.

I forgot! All the services seems to be running fine:
fabrizio@uranus:~$ sudo nova-manage service list
Binary Host Zone Status State Updated_At
nova-compute hemera nova enabled :-) 2013-01-26 17:26:25
nova-cert uranus nova enabled :-) 2013-01-26 17:26:15
nova-consoleauth uranus nova enabled :-) 2013-01-26 17:26:19
nova-scheduler uranus nova enabled :-) 2013-01-26 17:26:21
nova-network uranus nova enabled :-) 2013-01-26 17:26:21
nova-compute uranus nova disabled XXX 2013-01-26 16:04:37

Does somebody know how to solve?

Tanks.
Fabrizio.

Question information

Language:
English Edit question
Status:
Answered
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Last query:
2013-01-26
Last reply:
2013-01-29
ColOfAbRiX (colofabrix) said : #1

This is the log of nova-api after the call

fabrizio@uranus:~/nova-log$ sudo nova delete ef61492b-14b6-4b43-9f49-51e6f152e26d

18:30:07 INFO nova.api.openstack.wsgi - GET http://localhost:8774/v1.1/<project_id>/servers/detail
18:30:07 DEBUG nova.api.openstack.wsgi - No Content-Type provided in request
18:30:07 DEBUG nova.compute.api - Searching by: {'deleted': False, 'project_id': u'<project_id>'}
18:30:08 INFO nova.api.openstack.wsgi - http://localhost:8774/v1.1/<project_id>/servers/detail returned with HTTP 200
18:30:08 INFO nova.osapi_compute.wsgi.server - 127.0.0.1 - - - "GET /v1.1/<project_id>/servers/detail HTTP/1.1" 200 1631 1.194018

18:30:08 INFO nova.api.openstack.wsgi - GET http://localhost:8774/v1.1/<project_id>/servers/detail
18:30:08 DEBUG nova.api.openstack.wsgi - No Content-Type provided in request
18:30:08 DEBUG nova.compute.api - Searching by: {'deleted': False, 'project_id': u'<project_id>'}
18:30:09 INFO nova.api.openstack.wsgi - http://localhost:8774/v1.1/<project_id>/servers/detail returned with HTTP 200
18:30:09 INFO nova.osapi_compute.wsgi.server - 127.0.0.1 - - - "GET /v1.1/<project_id>/servers/detail HTTP/1.1" 200 1631 1.179668

18:30:10 INFO nova.api.openstack.wsgi - DELETE http://localhost:8774/v1.1/<project_id>/servers/<instance-id>
18:30:10 DEBUG nova.api.openstack.wsgi - No Content-Type provided in request
18:30:10 DEBUG nova.compute.api - - Going to try to terminate instance
18:30:11 DEBUG nova.openstack.common.rpc.amqp - Making asynchronous cast on compute.hemera...
18:30:11 DEBUG nova.openstack.common.rpc.amqp - Pool creating new connection
18:30:11 INFO nova.openstack.common.rpc.common - Connected to AMQP server on 192.168.3.1:5672

Keith Tobin (keith-tobin) said : #2

Is their an more of the log you posted? If so can you post here.
What sort of a install have you?

Keith Tobin (keith-tobin) said : #3

It may be a good idea to just post the log if that is possible, use a paste bin like pastie.org

ColOfAbRiX (colofabrix) said : #4

All my machines are old, 32bit pentium 4. The cluster controller is running with Ubuntu 12.04, no virtualization, with keystone, glance and all the nova services except nova-compute. nova-compute is running on the other 2 machines over XCP 1.6, but the problem appear in the cluster controller.
In this file there are all the logs and che configuration files for nova: http://wikisend.com/download/966144/nova-all.tar.gz

ColOfAbRiX (colofabrix) said : #5

I uninstalled all the nova components and dependencies and then I installed again them. Everything seemed fine but when I nova boot the problem appeared again. After that command also nova network-list got stuck
Here there is all the configuration and logs of the cloud controller: http://wikisend.com/download/610934/node-controller.tar.gz and here everything aboud the compute node: http://wikisend.com/download/120592/node-compute.tar.gz

Keith Tobin (keith-tobin) said : #6

Hold tight I am looking at files thanks.

Keith Tobin (keith-tobin) said : #7

Sorry for delay in getting back to you, in you log their is a a error/exception that occurs when you start nova-compute. This error occurs because the database is missing a table called 'instances'. If this table is missing from the database, something has gone wrong with your install of the database. I would log in in to MySql and verify if this table is missing and then figure why you did not add it, or did something go wrong with your install in some way. Here is a snip of the error. Feel free to ask any other questions and I will try to help. If you get thing working please close this bug.

Keith

2013-01-28 17:19:29 AUDIT nova.service [-] Starting compute node (version 2012.2-LOCALBRANCH:LOCALREVISION)
2013-01-28 17:19:30 DEBUG nova.utils [req-5a439208-87d2-4545-a35f-08607fdb3f49 None None] backend <module 'nova.db.sqlalchemy.api' from '/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.pyc'> from (pid=945) __get_backend /usr/lib/python2.7/dist-packages/nova/utils.py:494
2013-01-28 17:19:31 CRITICAL nova [-] (ProgrammingError) (1146, "Table 'nova.instances' doesn't exist") 'SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.server_name AS instances_server_name, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.hostname AS instances_hostname, instances.host AS instances_host, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.scheduled_at AS instances_scheduled_at, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instance_metadata_1.created_at AS instance_metadata_1_created_at, instance_metadata_1.updated_at AS instance_metadata_1_updated_at, instance_metadata_1.deleted_at AS instance_metadata_1_deleted_at, instance_metadata_1.deleted AS instance_metadata_1_deleted, instance_metadata_1.id AS instance_metadata_1_id, instance_metadata_1.`key` AS instance_metadata_1_key, instance_metadata_1.value AS instance_metadata_1_value, instance_metadata_1.instance_uuid AS instance_metadata_1_instance_uuid, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, instance_types_1.created_at AS instance_types_1_created_at, instance_types_1.updated_at AS instance_types_1_updated_at, instance_types_1.deleted_at AS instance_types_1_deleted_at, instance_types_1.deleted AS instance_types_1_deleted, instance_types_1.id AS instance_types_1_id, instance_types_1.name AS instance_types_1_name, instance_types_1.memory_mb AS instance_types_1_memory_mb, instance_types_1.vcpus AS instance_types_1_vcpus, instance_types_1.root_gb AS instance_types_1_root_gb, instance_types_1.ephemeral_gb AS instance_types_1_ephemeral_gb, instance_types_1.flavorid AS instance_types_1_flavorid, instance_types_1.swap AS instance_types_1_swap, instance_types_1.rxtx_factor AS instance_types_1_rxtx_factor, instance_types_1.vcpu_weight AS instance_types_1_vcpu_weight, instance_types_1.disabled AS instance_types_1_disabled, instance_types_1.is_public AS instance_types_1_is_public, instance_info_caches_1.created_at AS instance_info_caches_1_created_at, instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at, instance_info_caches_1.deleted_at AS instance_info_caches_1_deleted_at, instance_info_caches_1.deleted AS instance_info_caches_1_deleted, instance_info_caches_1.id AS instance_info_caches_1_id, instance_info_caches_1.network_info AS instance_info_caches_1_network_info, instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid \nFROM instances LEFT OUTER JOIN instance_metadata AS instance_metadata_1 ON instance_metadata_1.instance_uuid = instances.uuid AND instance_metadata_1.deleted = %s LEFT OUTER JOIN security_group_instance_association AS security_group_instance_association_1 ON security_group_instance_association_1.instance_uuid = instances.uuid AND instances.deleted = %s LEFT OUTER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s LEFT OUTER JOIN instance_types AS instance_types_1 ON instances.instance_type_id = instance_types_1.id LEFT OUTER JOIN instance_info_caches AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid = instances.uuid \nWHERE instances.deleted = %s AND instances.host = %s' (0, 0, 0, 0, 0, 'hemera')
2013-01-28 17:19:31 TRACE nova Traceback (most recent call last):
2013-01-28 17:19:31 TRACE nova File "/usr/bin/nova-compute", line 48, in <module>
2013-01-28 17:19:31 TRACE nova service.wait()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 659, in wait
2013-01-28 17:19:31 TRACE nova _launcher.wait()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 192, in wait
2013-01-28 17:19:31 TRACE nova super(ServiceLauncher, self).wait()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 162, in wait
2013-01-28 17:19:31 TRACE nova service.wait()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
2013-01-28 17:19:31 TRACE nova return self._exit_event.wait()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2013-01-28 17:19:31 TRACE nova return hubs.get_hub().switch()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
2013-01-28 17:19:31 TRACE nova return self.greenlet.switch()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
2013-01-28 17:19:31 TRACE nova result = function(*args, **kwargs)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 132, in run_server
2013-01-28 17:19:31 TRACE nova server.start()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 398, in start
2013-01-28 17:19:31 TRACE nova self.manager.init_host()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 274, in init_host
2013-01-28 17:19:31 TRACE nova instances = self.db.instance_get_all_by_host(context, self.host)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 626, in instance_get_all_by_host
2013-01-28 17:19:31 TRACE nova return IMPL.instance_get_all_by_host(context, host)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 111, in wrapper
2013-01-28 17:19:31 TRACE nova return f(*args, **kwargs)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1694, in instance_get_all_by_host
2013-01-28 17:19:31 TRACE nova return _instance_get_all_query(context).filter_by(host=host).all()
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2115, in all
2013-01-28 17:19:31 TRACE nova return list(self)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
2013-01-28 17:19:31 TRACE nova return self._execute_and_instances(context)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances
2013-01-28 17:19:31 TRACE nova result = conn.execute(querycontext.statement, self._params)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute
2013-01-28 17:19:31 TRACE nova params)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
2013-01-28 17:19:31 TRACE nova compiled_sql, distilled_params
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
2013-01-28 17:19:31 TRACE nova context)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
2013-01-28 17:19:31 TRACE nova context)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute
2013-01-28 17:19:31 TRACE nova cursor.execute(statement, parameters)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
2013-01-28 17:19:31 TRACE nova self.errorhandler(self, exc, value)
2013-01-28 17:19:31 TRACE nova File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2013-01-28 17:19:31 TRACE nova raise errorclass, errorvalue
2013-01-28 17:19:31 TRACE nova ProgrammingError: (ProgrammingError) (1146, "Table 'nova.instances' doesn't exist") 'SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.server_name AS instances_server_name, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.hostname AS instances_hostname, instances.host AS instances_host, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.scheduled_at AS instances_scheduled_at, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instance_metadata_1.created_at AS instance_metadata_1_created_at, instance_metadata_1.updated_at AS instance_metadata_1_updated_at, instance_metadata_1.deleted_at AS instance_metadata_1_deleted_at, instance_metadata_1.deleted AS instance_metadata_1_deleted, instance_metadata_1.id AS instance_metadata_1_id, instance_metadata_1.`key` AS instance_metadata_1_key, instance_metadata_1.value AS instance_metadata_1_value, instance_metadata_1.instance_uuid AS instance_metadata_1_instance_uuid, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, instance_types_1.created_at AS instance_types_1_created_at, instance_types_1.updated_at AS instance_types_1_updated_at, instance_types_1.deleted_at AS instance_types_1_deleted_at, instance_types_1.deleted AS instance_types_1_deleted, instance_types_1.id AS instance_types_1_id, instance_types_1.name AS instance_types_1_name, instance_types_1.memory_mb AS instance_types_1_memory_mb, instance_types_1.vcpus AS instance_types_1_vcpus, instance_types_1.root_gb AS instance_types_1_root_gb, instance_types_1.ephemeral_gb AS instance_types_1_ephemeral_gb, instance_types_1.flavorid AS instance_types_1_flavorid, instance_types_1.swap AS instance_types_1_swap, instance_types_1.rxtx_factor AS instance_types_1_rxtx_factor, instance_types_1.vcpu_weight AS instance_types_1_vcpu_weight, instance_types_1.disabled AS instance_types_1_disabled, instance_types_1.is_public AS instance_types_1_is_public, instance_info_caches_1.created_at AS instance_info_caches_1_created_at, instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at, instance_info_caches_1.deleted_at AS instance_info_caches_1_deleted_at, instance_info_caches_1.deleted AS instance_info_caches_1_deleted, instance_info_caches_1.id AS instance_info_caches_1_id, instance_info_caches_1.network_info AS instance_info_caches_1_network_info, instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid \nFROM instances LEFT OUTER JOIN instance_metadata AS instance_metadata_1 ON instance_metadata_1.instance_uuid = instances.uuid AND instance_metadata_1.deleted = %s LEFT OUTER JOIN security_group_instance_association AS security_group_instance_association_1 ON security_group_instance_association_1.instance_uuid = instances.uuid AND instances.deleted = %s LEFT OUTER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s LEFT OUTER JOIN instance_types AS instance_types_1 ON instances.instance_type_id = instance_types_1.id LEFT OUTER JOIN instance_info_caches AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid = instances.uuid \nWHERE instances.deleted = %s AND instances.host = %s' (0, 0, 0, 0, 0, 'hemera')
2013-01-28 17:19:31 TRACE nova

Keith Tobin (keith-tobin) said : #8

Adding to my above/last post, you have a second error in you config file, I would need to have a further look at what may cause it but, I would try fix the database problem I pointed out in last post and then restart all the services and see if you have problems still, if so post the new logs and I will help.

2013-01-28 17:15:48 CRITICAL nova [-] [Errno 13] Permission denied
2013-01-28 17:15:48 TRACE nova Traceback (most recent call last):
2013-01-28 17:15:48 TRACE nova File "/usr/bin/nova-compute", line 48, in <module>
2013-01-28 17:15:48 TRACE nova service.wait()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 659, in wait
2013-01-28 17:15:48 TRACE nova _launcher.wait()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 192, in wait
2013-01-28 17:15:48 TRACE nova super(ServiceLauncher, self).wait()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 162, in wait
2013-01-28 17:15:48 TRACE nova service.wait()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
2013-01-28 17:15:48 TRACE nova return self._exit_event.wait()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2013-01-28 17:15:48 TRACE nova return hubs.get_hub().switch()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
2013-01-28 17:15:48 TRACE nova return self.greenlet.switch()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
2013-01-28 17:15:48 TRACE nova result = function(*args, **kwargs)
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 132, in run_server
2013-01-28 17:15:48 TRACE nova server.start()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 398, in start
2013-01-28 17:15:48 TRACE nova self.manager.init_host()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 272, in init_host
2013-01-28 17:15:48 TRACE nova self.driver.init_host(host=self.host)
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/driver.py", line 164, in init_host
2013-01-28 17:15:48 TRACE nova vm_utils.ensure_correct_host(self._session)
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 2155, in ensure_correct_host
2013-01-28 17:15:48 TRACE nova this_vm_uuid = get_this_vm_uuid()
2013-01-28 17:15:48 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 1803, in get_this_vm_uuid
2013-01-28 17:15:48 TRACE nova return f.readline().strip()
2013-01-28 17:15:48 TRACE nova IOError: [Errno 13] Permission denied
2013-01-28 17:15:48 TRACE nova

ColOfAbRiX (colofabrix) said : #9

The table 'instances' is present in the database and the hypervisor is fully accessible. Those where installation issues that I already solved before testing. For example after uninstalling and then installing again the command "nova network-list" worked without problem. Only after I used "nova boot" the first time then "nova network-list" stopped to work. Something locked after nova boot.

While waiting for your kind response I set up a clone of the cluster controller where I managed to get everything working. This means that the compute node is ok as the config files on the controller.
I think I will erase the controller to get things done faster with the hope of never see this problem again.

Thanks for your help Keith ;)

Can you help with this problem?

Provide an answer of your own, or ask ColOfAbRiX for more information if necessary.

To post a message you must log in.