After reboot instance shuts down and nova-compute throws a libvrit error

Asked by Tomasz Kłosiński

After stress reboot test, nova-compute can't start because of this error:

2012-08-10 11:35:45 CRITICAL nova [-] Unable to pre-create chardev file '/var/lib/nova/instances/instance-00000049/console.log': No such file or directory
2012-08-10 11:35:45 TRACE nova Traceback (most recent call last):
2012-08-10 11:35:45 TRACE nova File "/usr/bin/nova-compute", line 49, in <module>
2012-08-10 11:35:45 TRACE nova service.wait()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 413, in wait
2012-08-10 11:35:45 TRACE nova _launcher.wait()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 131, in wait
2012-08-10 11:35:45 TRACE nova service.wait()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
2012-08-10 11:35:45 TRACE nova return self._exit_event.wait()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2012-08-10 11:35:45 TRACE nova return hubs.get_hub().switch()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
2012-08-10 11:35:45 TRACE nova return self.greenlet.switch()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
2012-08-10 11:35:45 TRACE nova result = function(*args, **kwargs)
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 101, in run_server
2012-08-10 11:35:45 TRACE nova server.start()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 162, in start
2012-08-10 11:35:45 TRACE nova self.manager.init_host()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 253, in init_host
2012-08-10 11:35:45 TRACE nova self.reboot_instance(context, instance['uuid'])
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-08-10 11:35:45 TRACE nova return f(*args, **kw)
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 159, in decorated_function
2012-08-10 11:35:45 TRACE nova function(self, context, instance_uuid, *args, **kwargs)
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 183, in decorated_function
2012-08-10 11:35:45 TRACE nova sys.exc_info())
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-08-10 11:35:45 TRACE nova self.gen.next()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 177, in decorated_function
2012-08-10 11:35:45 TRACE nova return function(self, context, instance_uuid, *args, **kwargs)
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 904, in reboot_instance
2012-08-10 11:35:45 TRACE nova reboot_type)
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-08-10 11:35:45 TRACE nova return f(*args, **kw)
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 721, in reboot
2012-08-10 11:35:45 TRACE nova if self._soft_reboot(instance):
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 757, in _soft_reboot
2012-08-10 11:35:45 TRACE nova dom.create()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/libvirt.py", line 551, in create
2012-08-10 11:35:45 TRACE nova if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
2012-08-10 11:35:45 TRACE nova libvirtError: Unable to pre-create chardev file '/var/lib/nova/instances/instance-00000049/console.log': No such file or directory
2012-08-10 11:35:45 TRACE nova

It seems that in nova db the instance is exists, but on the host it doesn't. Libvirt sees this instance as shut off.

root@KVM7:~# virsh list --all
 Id Name State
----------------------------------
  - instance-00000047 shut off
  - instance-00000048 shut off
  - instance-00000049 shut off
  - instance-00000057 shut off
  - instance-000000f9 shut off

root@KVM7:~# virsh start instance-00000049
error: Failed to start domain instance-00000049
error: Unable to pre-create chardev file '/var/lib/nova/instances/instance-00000049/console.log': No such file or directory

Here is mentioned instance record (exported to json format - for readability) form nova.instance table:

[
 {
  'created_at' : '2012-07-24 15:15:12',
  'updated_at' : '2012-07-24 15:26:42',
  'deleted_at' : '2012-07-24 15:26:42',
  'deleted' : 1,
  'id' : 2,
  'internal_id' : NULL,
  'user_id' : '1f5039d64f624c9c936177d04f1bc480',
  'project_id' : 'f4ac28a96c0747008142aa0a6ba87169',
  'image_ref' : '9e4e2f95-ccaa-40e7-9079-c7d7c22b8de2',
  'kernel_id' : '',
  'ramdisk_id' : '',
  'server_name' : NULL,
  'launch_index' : 1,
  'key_name' : 'mykey',
  'key_data' : 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtg+MlE5i8BJCcjrgOAE+/NGwBi7k+nREWQnqJ9tOmZ4KJP8UeqIGm52i9fV49d1zvUnK2Ssw8RkVpei/D/9oCtyMdx+mZiFkvOzyhawPypxCmzWBgOTnW5kSnUaMzawIPa7rvBsXpBgjRecCX2X0VNPBkArzidvdGMD1Bz2Lh4Jea1rH+ieAPCG6JU9lRgnmyOBKS9HaQOoTu/YcuZy5Ic2cbQOWk9Vv8z9I8IOJEgOAa+0bOxryrJE1ndftOcqgTx2hGTzOPHKUhuNq4iE8BesQ66Wwc8Vqaf7BNONh+ksFf0qhqKCssCOxUjOD6+yWdoShTqGFZKMC42T86o9vb root@KVM7',
  'power_state' : 1,
  'vm_state' : 'deleted',
  'memory_mb' : 2048,
  'vcpus' : 1,
  'hostname' : '10',
  'host' : 'KVM7',
  'user_data' : '',
  'reservation_id' : 'r-bcv08dl8',
  'scheduled_at' : '2012-07-24 15:15:13',
  'launched_at' : '2012-07-24 15:15:30',
  'terminated_at' : '2012-07-24 15:26:42',
  'display_name' : '10',
  'display_description' : '10',
  'availability_zone' : NULL,
  'locked' : 0,
  'os_type' : NULL,
  'launched_on' : 'KVM7',
  'instance_type_id' : 5,
  'vm_mode' : NULL,
  'uuid' : '3ccd3b88-3ad3-41bf-a074-de94b191bbcb',
  'architecture' : NULL,
  'root_device_name' : '/dev/vda',
  'access_ip_v4' : NULL,
  'access_ip_v6' : NULL,
  'config_drive' : '',
  'task_state' : NULL,
  'default_ephemeral_device' : '/dev/vdb',
  'default_swap_device' : NULL,
  'progress' : 0,
  'auto_disk_config' : NULL,
  'shutdown_terminate' : 1,
  'disable_terminate' : 0,
  'root_gb' : 10,
  'ephemeral_gb' : 20,
  'cell_name' : NULL
 }
]

Is it possible to recreate the /var/lib/nova/instances/instance-00000049 somehow? I need to either get rid of that problem or at least have a possibility to resurrect this intance.

Question information

Language:
English Edit question
Status:
Expired
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Tomasz Kłosiński (tomasz-michal-klosinski) said :
#1

One more thing:

root@KVM7:~# ls /var/lib/nova/instances/instance-00000049/
ls: cannot access /var/lib/nova/instances/instance-00000049/: No such file or directory

It seems that nova or libvirt have deleted this directory for some reason.

Revision history for this message
Tiantian Gao (gtt116) said :
#2

Maybe the instance is in deleteing process. nova-compute just delete the /var/lib/nova/instance/instance-00000049, and not delete from database. And at this time, you reboot it, so the vm is in database, but the dictionary instance-00000049 can not be found.

Revision history for this message
Tomasz Kłosiński (tomasz-michal-klosinski) said :
#3

Thank you TianTian for you answer. However, I still don't know how to deal with this problem. It seems that nova is (or at least was at time of Essex release) very immature project.

Revision history for this message
Tom Fifield (fifieldt) said :
#4

Hi Tomasz,

Do you still have this issue? Have you tried with Folsom?

Revision history for this message
Tomasz Kłosiński (tomasz-michal-klosinski) said :
#5

Hello Tom,

I haven't tried it yet. Unfortunetly time for R&D in my company passed and (maybe) I will get back to OpenStack in next year.

Best wishes,
Tom

Revision history for this message
Launchpad Janitor (janitor) said :
#6

This question was expired because it remained in the 'Needs information' state without activity for the last 15 days.