How to clean nova DB

Asked by Warik

Hi stackers,

I have a question regarding the nova DB. I have currently a cluster of 3 Openstack Essex servers (2012.1-0ubuntu2.1) on Precise. Earlier today, one of the nova node crashed (the service nova-compute from the host too), without noticed it, i tried to terminated an instance on that host. It tried to terminated it.. usually it takes 5 seconds to delete an instances but not this time...because the host was down. I connected the node, restart nova-api, nova-network and finally nova-compute. But nova-compute doesn't start because the instance is not on kvm anymore:

//////////
root@nova-compute:~# virsh list --all
 Id Name State
----------------------------------

root@nova-compute:~#
//////////

When i am trying to launch nova-compute here is what I get from the logs:

root@nova-compute:~# service nova-compute start

LOG file:

May 10 14:07:20 nova-compute 2012-05-10 14:07:20 WARNING nova.compute.manager [req-738f927d-ab0a-4da1-b795-6723df600067 None None] trying to reboot a non-running instance: 5ed0222e-3220-4be8-849a-d92916728ef9 (state: 8 expected: 1)

May 10 14:07:20 nova-compute 2012-05-10 14:07:20 CRITICAL nova [-] Instance instance-00000004 could not be found.#0122012-05-10 14:07:20 TRACE nova Traceback (most recent call last):#0122012-05-10 14:07:20 TRACE nova File "/usr/bin/nova-compute", line 49, in <module>#0122012-05-10 14:07:20 TRACE nova service.wait()#0122012-05-10 14:07:20 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 413, in wait#0122012-05-10 14:07:20 TRACE nova _launcher.wait()#0122012-05-10 14:07:20 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 131, in wait#0122012-05-10 14:07:20 TRACE nova service.wait()#0122012-05-10 14:07:20 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait#0122012-05-10 14:07:20 TRACE nova return self._exit_event.wait()#0122012-05-10 14:07:20 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait#0122012-05-10 14:07:20 TRACE nova return hubs.get_hub().switch()#0122012-05-10 14:07:20 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch#0122012-05-10 14:07:20 TRACE nova return self.greenlet.switch()#0122012-05-10 14:07:20 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main#0122012-05-10 14:07:20 TRACE nova result = function(*args, **kwargs)#0122012-05-10 14:07:20 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 101, in run_server#0122012-05-10 14:07:20 TRACE nova server.start()#0122012-05-10 14:07:20 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 162, in start#0122012-05-10 14:07:20 TRACE nova self.manager.init_host()#0122012-05-10 14:07:20 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 247, in init_host#0122012-05-10 14:07:20 TRACE nova self.reboot_instance(context, instance['uuid'])#0122012-05-10 14:07:20 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrappe

May 10 14:07:20 nova-compute kernel: [ 2745.651930] init: nova-compute main process (4788) terminated with status 1

So, here I am, stuck in this limbo state.. I tried to remove the row from the nova DB but there is a foreign key that block me. And I don't want to destroy my DB (even though I dump nova DB).

I was wondering if you could let me know how can I "clean" nova DB so I can restart nova-compute on that host.

Thanks a lot.
Warik

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
Max Schilling
Solved:
Last query:
Last reply:
Revision history for this message
Best Max Schilling (mx-chilly) said :
#1

I remember having a similar problem not long ago. You don't need to delete the entry from the "instance" table, just set "deleted=1" in this entry:

update instance set deleted=1 where id=XXX;

With XXX being the id of the entry you want to change.

That is just my "quick&dirty" solution, I'm not sure whether that's all nova changes in the db when it deletes an instance. For example I noticed that if you're instance had a floating ip you'll also need to modify that entry, so that the floating ip is available once more.

Revision history for this message
Warik (warik) said :
#2

Thanks Max Schilling, that solved my question.

Revision history for this message
Fawad Khaliq (fawadkhaliq) said :
#3

Max, your method was really helpful. Thank you.