OpenStack Compute (nova)

Quota exceeded: code=%(code)s (HTTP 413)

Asked by Shirley Woo

I'm running Xenserver and Openstack.
I'm following the XenServer Development doc. Everything is installed by I get the Quota error when I try to run an instance.

nova@OSserver2b:/home/localadmin/openstack/nova$ nova image-list
+--------------------------------------+---------+--------+
| ID | Name | Status |
+--------------------------------------+---------+--------+
| 465442ad-da0a-401a-9efc-4a38a165361e | kernal | ACTIVE |
| 6c5f6148-34db-424b-bd8f-82c4688c7d78 | ramdisk | ACTIVE |
| 7b17d37e-8a76-4791-bf51-d887146ced52 | kernal | ACTIVE |
+--------------------------------------+---------+--------+
nova@OSserver2b:/home/localadmin/openstack/nova$ nova flavor-list
+----+-----------+-----------+------+----------+-------+------------+----------+
| ID | Name | Memory_MB | Swap | Local_GB | VCPUs | RXTX_Quota | RXTX_Cap |
+----+-----------+-----------+------+----------+-------+------------+----------+
| 1 | m1.tiny | 512 | | 0 | 1 | | |
| 2 | m1.small | 2048 | | 20 | 1 | | |
| 3 | m1.medium | 4096 | | 40 | 2 | | |
| 4 | m1.large | 8192 | | 80 | 4 | | |
| 5 | m1.xlarge | 16384 | | 160 | 8 | | |
+----+-----------+-----------+------+----------+-------+------------+----------+
nova@OSserver2b:/home/localadmin/openstack/nova$ nova boot test --flavor 1 --image 465442ad-da0a-401a-9efc-4a38a165361e
Quota exceeded: code=%(code)s (HTTP 413)
nova@OSserver2b:/home/localadmin/openstack/nova$ nova boot test --flavor 1 --image 6c5f6148-34db-424b-bd8f-82c4688c7d78
Quota exceeded: code=%(code)s (HTTP 413)
nova@OSserver2b:/home/localadmin/openstack/nova$ nova boot test --flavor 1 --image 7b17d37e-8a76-4791-bf51-d887146ced52
Quota exceeded: code=%(code)s (HTTP 413)
nova@OSserver2b:/home/localadmin/openstack/nova$ ./bin/nova-manage project quota openstack
/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py:8: UserWarning: Module nova was already imported from /home/localadmin/openstack/nova/nova/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path
  import pkg_resources

2012-03-16 17:11:47 DEBUG nova.utils [req-f8ec300f-ed76-4b48-b93f-7e0b1623dc8e None None] backend <module 'nova.db.sqlalchemy.api' from '/home/localadmin/openstack/nova/nova/db/sqlalchemy/api.pyc'> from (pid=843) __get_backend /home/localadmin/openstack/nova/nova/utils.py:607
metadata_items: 128
instances: 10
injected_file_content_bytes: 10240
injected_files: 5
volumes: 10
gigabytes: 1000
cores: 20
ram: 51200
floating_ips: 10
nova@OSserver2b:/home/localadmin/openstack/nova$

As far as I can tell, I should be able to create unto 10 instances, but it won't even let me create 1.

Am I missing another setting? Any help would be appreciated. Thanks.

Question information

Language:
English Edit question
Status:
Answered
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
John Garbutt (johngarbutt) said : 		#1

I would recommend you try using DevStack (http://wiki.openstack.org/XenServer/DevStack).

I have not seen that error myself, but it looks a lot like a configuration issue.

If you could check the api-server and the scheduler logs they may be able to give you some hints as to why it thinks you have reached your quota. Maybe the way you integrated keystone is not quite working?

It might be worth checking the clocks to ensure it does not simply think there are no resources avaiable (but I thought that would give you a different error).

I hope that is of some help.

Revision history for this message
Shirley Woo (swoo) said : 		#2

Yes, I am currently using a pull from DevStack.

The nova-api.log file shows:
2012-03-19 12:28:02 INFO nova.wsgi [-] Started metadata on 0.0.0.0:8775
2012-03-19 12:28:11 INFO nova.api.openstack.wsgi [req-e7e8696d-11ac-40a7-9013-11fdb90d025e localadmin openstack] POST http://10.228.24.234:8774/v1.1/openstack/servers
2012-03-19 12:28:12 DEBUG nova.utils [req-e7e8696d-11ac-40a7-9013-11fdb90d025e localadmin openstack] backend <module 'nova.db.sqlalchemy.api' from '/home/localadmin/openstack/nova/nova/db/sqlalchemy/api.pyc'> from (pid=2534) __get_backend /home/localadmin/openstack/nova/nova/utils.py:607
2012-03-19 12:28:14 WARNING nova.compute.api [req-e7e8696d-11ac-40a7-9013-11fdb90d025e localadmin openstack] Quota exceeded for openstack, tried to run 1 instances
2012-03-19 12:28:14 INFO nova.api.openstack.wsgi [req-e7e8696d-11ac-40a7-9013-11fdb90d025e localadmin openstack] HTTP exception thrown: Quota exceeded: code=%(code)s
2012-03-19 12:28:14 INFO nova.api.openstack.wsgi [req-e7e8696d-11ac-40a7-9013-11fdb90d025e localadmin openstack] http://10.228.24.234:8774/v1.1/openstack/servers returned with HTTP 413

I didn't notice any activity on the other logs.

Revision history for this message
Shirley Woo (swoo) said : 		#3

Also checked the clock and they are all in sync.

Revision history for this message
Shirley Woo (swoo) said : 		#4

Interesting. Just ran a "nova list" and found:

nova@OSserver2b:/home/localadmin/openstack/nova$ nova list
+--------------------------------------+------+--------+----------+
| ID | Name | Status | Networks |
 +--------------------------------------+------+--------+----------+
| 20854cdd-b158-449c-a44a-c676ebdc0f8f | test | ERROR | |
| 5a0d5c03-4869-48bd-a15b-60afb6e526c7 | test | ERROR | |
| 62c3f5cb-62e0-46de-be46-4aaf3e5d676e | test | ERROR | |
| 7b3045bd-2af5-4074-8352-42bc1ef4095b | test | ERROR | |
| b1967369-d887-47a2-a53e-d077f5349563 | test | ERROR | |
| b481cd67-ef31-4072-9723-48240f4b9ee1 | test | ERROR | |
| b4894f33-243f-4f4e-97e4-1f44f9c773b4 | test | ERROR | |
| c194ec35-b798-44f8-b6a4-10c43cc66236 | test | ERROR | |
| c6049034-8f33-4f3f-8c49-dba4f238406c | test | ERROR | |
| c813bc01-c590-4ede-bcf4-fdcbe5414b00 | test | ERROR | |
 +--------------------------------------+------+--------+----------+

So it seems that instances were being created and I did indeed have 10 of them already.
Will delete and they to create again and see what happens.

Revision history for this message
Shirley Woo (swoo) said : 		#5

Ran "nova delete 20854cdd-b158-449c-a44a-c676ebdc0f8f"

Command didn't return an error and didn't delete the instance.

 nova-api output:
2012-03-19 16:31:49 INFO nova.api.openstack.wsgi [req-bae31505-1f34-4415-8994-7b07a8497ff9 localadmin openstack] GET http://10.228.24.234:8774/v1.1/openstack/servers/detail?fresh=1332199909.35
2012-03-19 16:31:49 DEBUG nova.api.openstack.wsgi [req-bae31505-1f34-4415-8994-7b07a8497ff9 localadmin openstack] Unrecognized Content-Type provided in request from (pid=3143) get_body /home/localadmin/openstack/nova/nova/api/openstack/wsgi.py:697
2012-03-19 16:31:49 DEBUG nova.compute.api [req-bae31505-1f34-4415-8994-7b07a8497ff9 localadmin openstack] Searching by: {'deleted': False, 'fresh': '1332199909.35', 'project_id': 'openstack'} from (pid=3143) get_all /home/localadmin/openstack/nova/nova/compute/api.py:1010
2012-03-19 16:31:49 DEBUG nova.api.openstack.common [req-bae31505-1f34-4415-8994-7b07a8497ff9 localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3143) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:31:49 DEBUG nova.api.openstack.common [req-bae31505-1f34-4415-8994-7b07a8497ff9 localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3143) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:31:49 DEBUG nova.api.openstack.common [req-bae31505-1f34-4415-8994-7b07a8497ff9 localadmin openstack] Generated ERROR from vm_state=error task_state=spawning. from (pid=3143) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:31:49 DEBUG nova.api.openstack.common [req-bae31505-1f34-4415-8994-7b07a8497ff9 localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3143) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:31:49 DEBUG nova.api.openstack.common [req-bae31505-1f34-4415-8994-7b07a8497ff9 localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3143) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:31:49 DEBUG nova.api.openstack.common [req-bae31505-1f34-4415-8994-7b07a8497ff9 localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3143) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:31:49 DEBUG nova.api.openstack.common [req-bae31505-1f34-4415-8994-7b07a8497ff9 localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3143) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:31:49 DEBUG nova.api.openstack.common [req-bae31505-1f34-4415-8994-7b07a8497ff9 localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3143) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:31:49 DEBUG nova.api.openstack.common [req-bae31505-1f34-4415-8994-7b07a8497ff9 localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3143) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:31:49 DEBUG nova.api.openstack.common [req-bae31505-1f34-4415-8994-7b07a8497ff9 localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3143) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:31:49 DEBUG nova.compute.api [req-bae31505-1f34-4415-8994-7b07a8497ff9 localadmin openstack] Searching by: {'uuid': ['7b3045bd-2af5-4074-8352-42bc1ef4095b', '62c3f5cb-62e0-46de-be46-4aaf3e5d676e', 'c6049034-8f33-4f3f-8c49-dba4f238406c', 'c194ec35-b798-44f8-b6a4-10c43cc66236', 'b4894f33-243f-4f4e-97e4-1f44f9c773b4', 'c813bc01-c590-4ede-bcf4-fdcbe5414b00', '20854cdd-b158-449c-a44a-c676ebdc0f8f', '5a0d5c03-4869-48bd-a15b-60afb6e526c7', 'b481cd67-ef31-4072-9723-48240f4b9ee1', 'b1967369-d887-47a2-a53e-d077f5349563']} from (pid=3143) get_all /home/localadmin/openstack/nova/nova/compute/api.py:1010
2012-03-19 16:31:49 DEBUG nova.compute.api [req-bae31505-1f34-4415-8994-7b07a8497ff9 localadmin openstack] Searching by: {'uuid': ['7b3045bd-2af5-4074-8352-42bc1ef4095b', '62c3f5cb-62e0-46de-be46-4aaf3e5d676e', 'c6049034-8f33-4f3f-8c49-dba4f238406c', 'c194ec35-b798-44f8-b6a4-10c43cc66236', 'b4894f33-243f-4f4e-97e4-1f44f9c773b4', 'c813bc01-c590-4ede-bcf4-fdcbe5414b00', '20854cdd-b158-449c-a44a-c676ebdc0f8f', '5a0d5c03-4869-48bd-a15b-60afb6e526c7', 'b481cd67-ef31-4072-9723-48240f4b9ee1', 'b1967369-d887-47a2-a53e-d077f5349563']} from (pid=3143) get_all /home/localadmin/openstack/nova/nova/compute/api.py:1010
2012-03-19 16:31:49 INFO nova.api.openstack.wsgi [req-bae31505-1f34-4415-8994-7b07a8497ff9 localadmin openstack] http://10.228.24.234:8774/v1.1/openstack/servers/detail?fresh=1332199909.35 returned with HTTP 200

Revision history for this message
Shirley Woo (swoo) said : 		#6

Tried again to delete and now getting a similar, but different error:

2012-03-19 16:36:11 INFO nova.api.openstack.wsgi [req-30892558-52b1-4d42-89b3-e8675b5710ca localadmin openstack] GET http://10.228.24.234:8774/v1.1/openstack/servers/detail?fresh=1332200171.12
2012-03-19 16:36:11 DEBUG nova.api.openstack.wsgi [req-30892558-52b1-4d42-89b3-e8675b5710ca localadmin openstack] Unrecognized Content-Type provided in request from (pid=3850) get_body /home/localadmin/openstack/nova/nova/api/openstack/wsgi.py:697
2012-03-19 16:36:11 DEBUG nova.compute.api [req-30892558-52b1-4d42-89b3-e8675b5710ca localadmin openstack] Searching by: {'deleted': False, 'fresh': '1332200171.12', 'project_id': 'openstack'} from (pid=3850) get_all /home/localadmin/openstack/nova/nova/compute/api.py:1010
2012-03-19 16:36:11 DEBUG nova.api.openstack.common [req-30892558-52b1-4d42-89b3-e8675b5710ca localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3850) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:36:11 DEBUG nova.api.openstack.common [req-30892558-52b1-4d42-89b3-e8675b5710ca localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3850) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:36:11 DEBUG nova.api.openstack.common [req-30892558-52b1-4d42-89b3-e8675b5710ca localadmin openstack] Generated ERROR from vm_state=error task_state=spawning. from (pid=3850) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:36:11 DEBUG nova.api.openstack.common [req-30892558-52b1-4d42-89b3-e8675b5710ca localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3850) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:36:11 DEBUG nova.api.openstack.common [req-30892558-52b1-4d42-89b3-e8675b5710ca localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3850) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:36:11 DEBUG nova.api.openstack.common [req-30892558-52b1-4d42-89b3-e8675b5710ca localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3850) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:36:11 DEBUG nova.api.openstack.common [req-30892558-52b1-4d42-89b3-e8675b5710ca localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3850) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:36:11 DEBUG nova.api.openstack.common [req-30892558-52b1-4d42-89b3-e8675b5710ca localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3850) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:36:11 DEBUG nova.api.openstack.common [req-30892558-52b1-4d42-89b3-e8675b5710ca localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3850) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:36:11 DEBUG nova.api.openstack.common [req-30892558-52b1-4d42-89b3-e8675b5710ca localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3850) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:36:11 DEBUG nova.compute.api [req-30892558-52b1-4d42-89b3-e8675b5710ca localadmin openstack] Searching by: {'uuid': ['7b3045bd-2af5-4074-8352-42bc1ef4095b', '62c3f5cb-62e0-46de-be46-4aaf3e5d676e', 'c6049034-8f33-4f3f-8c49-dba4f238406c', 'c194ec35-b798-44f8-b6a4-10c43cc66236', 'b4894f33-243f-4f4e-97e4-1f44f9c773b4', 'c813bc01-c590-4ede-bcf4-fdcbe5414b00', '20854cdd-b158-449c-a44a-c676ebdc0f8f', '5a0d5c03-4869-48bd-a15b-60afb6e526c7', 'b481cd67-ef31-4072-9723-48240f4b9ee1', 'b1967369-d887-47a2-a53e-d077f5349563']} from (pid=3850) get_all /home/localadmin/openstack/nova/nova/compute/api.py:1010
2012-03-19 16:36:11 DEBUG nova.compute.api [req-30892558-52b1-4d42-89b3-e8675b5710ca localadmin openstack] Searching by: {'uuid': ['7b3045bd-2af5-4074-8352-42bc1ef4095b', '62c3f5cb-62e0-46de-be46-4aaf3e5d676e', 'c6049034-8f33-4f3f-8c49-dba4f238406c', 'c194ec35-b798-44f8-b6a4-10c43cc66236', 'b4894f33-243f-4f4e-97e4-1f44f9c773b4', 'c813bc01-c590-4ede-bcf4-fdcbe5414b00', '20854cdd-b158-449c-a44a-c676ebdc0f8f', '5a0d5c03-4869-48bd-a15b-60afb6e526c7', 'b481cd67-ef31-4072-9723-48240f4b9ee1', 'b1967369-d887-47a2-a53e-d077f5349563']} from (pid=3850) get_all /home/localadmin/openstack/nova/nova/compute/api.py:1010
2012-03-19 16:36:11 INFO nova.api.openstack.wsgi [req-30892558-52b1-4d42-89b3-e8675b5710ca localadmin openstack] http://10.228.24.234:8774/v1.1/openstack/servers/detail?fresh=1332200171.12 returned with HTTP 200
2012-03-19 16:36:16 INFO nova.api.openstack.wsgi [req-bc3cc0e5-b508-40a0-87bd-b41d55182a4d localadmin openstack] GET http://10.228.24.234:8774/v1.1/openstack/servers/20854cdd-b158-449c-a44a-c676ebdc0f8f?fresh=1332200176.3
2012-03-19 16:36:16 DEBUG nova.api.openstack.wsgi [req-bc3cc0e5-b508-40a0-87bd-b41d55182a4d localadmin openstack] Unrecognized Content-Type provided in request from (pid=3850) get_body /home/localadmin/openstack/nova/nova/api/openstack/wsgi.py:697
2012-03-19 16:36:16 DEBUG nova.api.openstack.common [req-bc3cc0e5-b508-40a0-87bd-b41d55182a4d localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3850) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:36:16 INFO nova.api.openstack.wsgi [req-bc3cc0e5-b508-40a0-87bd-b41d55182a4d localadmin openstack] http://10.228.24.234:8774/v1.1/openstack/servers/20854cdd-b158-449c-a44a-c676ebdc0f8f?fresh=1332200176.3 returned with HTTP 200
2012-03-19 16:36:16 INFO nova.api.openstack.wsgi [req-b3bd8074-1f59-4be7-b408-e99dd6f2b5f7 localadmin openstack] GET http://10.228.24.234:8774/v1.1/openstack/servers/20854cdd-b158-449c-a44a-c676ebdc0f8f?fresh=1332200176.55
2012-03-19 16:36:16 DEBUG nova.api.openstack.wsgi [req-b3bd8074-1f59-4be7-b408-e99dd6f2b5f7 localadmin openstack] Unrecognized Content-Type provided in request from (pid=3850) get_body /home/localadmin/openstack/nova/nova/api/openstack/wsgi.py:697
2012-03-19 16:36:16 DEBUG nova.api.openstack.common [req-b3bd8074-1f59-4be7-b408-e99dd6f2b5f7 localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3850) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:36:16 INFO nova.api.openstack.wsgi [req-b3bd8074-1f59-4be7-b408-e99dd6f2b5f7 localadmin openstack] http://10.228.24.234:8774/v1.1/openstack/servers/20854cdd-b158-449c-a44a-c676ebdc0f8f?fresh=1332200176.55 returned with HTTP 200
2012-03-19 16:36:16 INFO nova.api.openstack.wsgi [req-211c4deb-76cc-4603-8f2a-9f3ac5e3dac8 localadmin openstack] DELETE http://10.228.24.234:8774/v1.1/openstack/servers/20854cdd-b158-449c-a44a-c676ebdc0f8f
2012-03-19 16:36:16 DEBUG nova.api.openstack.wsgi [req-211c4deb-76cc-4603-8f2a-9f3ac5e3dac8 localadmin openstack] Unrecognized Content-Type provided in request from (pid=3850) get_body /home/localadmin/openstack/nova/nova/api/openstack/wsgi.py:697
2012-03-19 16:36:16 DEBUG nova.compute.api [req-211c4deb-76cc-4603-8f2a-9f3ac5e3dac8 localadmin openstack] [instance: 20854cdd-b158-449c-a44a-c676ebdc0f8f] Going to try to terminate instance from (pid=3850) delete /home/localadmin/openstack/nova/nova/compute/api.py:883
2012-03-19 16:36:17 DEBUG nova.rpc.common [req-211c4deb-76cc-4603-8f2a-9f3ac5e3dac8 localadmin openstack] Making asynchronous cast on compute.OSserver2b... from (pid=3850) cast /home/localadmin/openstack/nova/nova/rpc/amqp.py:344
2012-03-19 16:36:17 INFO nova.api.openstack.wsgi [req-211c4deb-76cc-4603-8f2a-9f3ac5e3dac8 localadmin openstack] http://10.228.24.234:8774/v1.1/openstack/servers/20854cdd-b158-449c-a44a-c676ebdc0f8f returned with HTTP 204
2012-03-19 16:38:16 INFO nova.api.openstack.wsgi [req-2f1d1a90-e8b1-4939-ba28-78140f482d4c localadmin openstack] GET http://10.228.24.234:8774/v1.1/openstack/servers/20854cdd-b158-449c-a44a-c676ebdc0f8f?fresh=1332200296.48
2012-03-19 16:38:16 DEBUG nova.api.openstack.wsgi [req-2f1d1a90-e8b1-4939-ba28-78140f482d4c localadmin openstack] Unrecognized Content-Type provided in request from (pid=3850) get_body /home/localadmin/openstack/nova/nova/api/openstack/wsgi.py:697
2012-03-19 16:38:16 DEBUG nova.api.openstack.common [req-2f1d1a90-e8b1-4939-ba28-78140f482d4c localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3850) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:38:16 INFO nova.api.openstack.wsgi [req-2f1d1a90-e8b1-4939-ba28-78140f482d4c localadmin openstack] http://10.228.24.234:8774/v1.1/openstack/servers/20854cdd-b158-449c-a44a-c676ebdc0f8f?fresh=1332200296.48 returned with HTTP 200
2012-03-19 16:38:16 INFO nova.api.openstack.wsgi [req-32117cbc-6d1d-45e5-9e20-efdd7e4ee42e localadmin openstack] GET http://10.228.24.234:8774/v1.1/openstack/servers/20854cdd-b158-449c-a44a-c676ebdc0f8f?fresh=1332200296.73
2012-03-19 16:38:16 DEBUG nova.api.openstack.wsgi [req-32117cbc-6d1d-45e5-9e20-efdd7e4ee42e localadmin openstack] Unrecognized Content-Type provided in request from (pid=3850) get_body /home/localadmin/openstack/nova/nova/api/openstack/wsgi.py:697
2012-03-19 16:38:16 DEBUG nova.api.openstack.common [req-32117cbc-6d1d-45e5-9e20-efdd7e4ee42e localadmin openstack] Generated ERROR from vm_state=error task_state=deleting. from (pid=3850) status_from_state /home/localadmin/openstack/nova/nova/api/openstack/common.py:96
2012-03-19 16:38:16 INFO nova.api.openstack.wsgi [req-32117cbc-6d1d-45e5-9e20-efdd7e4ee42e localadmin openstack] http://10.228.24.234:8774/v1.1/openstack/servers/20854cdd-b158-449c-a44a-c676ebdc0f8f?fresh=1332200296.73 returned with HTTP 200
2012-03-19 16:38:16 INFO nova.api.openstack.wsgi [req-7e9b8583-c296-4baf-bf4a-a16fd6535e9a localadmin openstack] DELETE http://10.228.24.234:8774/v1.1/openstack/servers/20854cdd-b158-449c-a44a-c676ebdc0f8f
2012-03-19 16:38:16 DEBUG nova.api.openstack.wsgi [req-7e9b8583-c296-4baf-bf4a-a16fd6535e9a localadmin openstack] Unrecognized Content-Type provided in request from (pid=3850) get_body /home/localadmin/openstack/nova/nova/api/openstack/wsgi.py:697
2012-03-19 16:38:17 DEBUG nova.compute.api [req-7e9b8583-c296-4baf-bf4a-a16fd6535e9a localadmin openstack] [instance: 20854cdd-b158-449c-a44a-c676ebdc0f8f] Going to try to terminate instance from (pid=3850) delete /home/localadmin/openstack/nova/nova/compute/api.py:883
2012-03-19 16:38:17 DEBUG nova.rpc.common [req-7e9b8583-c296-4baf-bf4a-a16fd6535e9a localadmin openstack] Making asynchronous cast on compute.OSserver2b... from (pid=3850) cast /home/localadmin/openstack/nova/nova/rpc/amqp.py:344
2012-03-19 16:38:17 INFO nova.api.openstack.wsgi [req-7e9b8583-c296-4baf-bf4a-a16fd6535e9a localadmin openstack] http://10.228.24.234:8774/v1.1/openstack/servers/20854cdd-b158-449c-a44a-c676ebdc0f8f returned with HTTP 204

Revision history for this message
Shirley Woo (swoo) said : 		#7

The nova-compute output shows:

2012-03-19 17:23:09 DEBUG nova.rpc.common [-] received {u'_context_roles': [u'admin'], u'_context_request_id': u'req-d7d78fd8-093d-4ea9-8cc4-31714b917e37', u'_context_read_deleted': u'no', u'args': {u'instance_uuid': u'c813bc01-c590-4ede-bcf4-fdcbe5414b00'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'openstack', u'_context_timestamp': u'2012-03-20T00:23:09.408167', u'_context_user_id': u'localadmin', u'method': u'terminate_instance', u'_context_remote_address': u'127.0.0.1'} from (pid=3939) _safe_log /home/localadmin/openstack/nova/nova/rpc/common.py:144
2012-03-19 17:23:09 DEBUG nova.rpc.common [req-d7d78fd8-093d-4ea9-8cc4-31714b917e37 localadmin openstack] unpacked context: {'request_id': u'req-d7d78fd8-093d-4ea9-8cc4-31714b917e37', 'user_id': u'localadmin', 'roles': [u'admin'], 'timestamp': '2012-03-20T00:23:09.408167', 'is_admin': True, 'auth_token': None, 'project_id': u'openstack', 'remote_address': u'127.0.0.1', 'read_deleted': u'no', 'strategy': u'noauth'} from (pid=3939) unpack_context /home/localadmin/openstack/nova/nova/rpc/amqp.py:186
2012-03-19 17:23:09 INFO nova.compute.manager [req-d7d78fd8-093d-4ea9-8cc4-31714b917e37 localadmin openstack] check_instance_lock: decorating: |<function terminate_instance at 0x1c51398>|
2012-03-19 17:23:09 INFO nova.compute.manager [req-d7d78fd8-093d-4ea9-8cc4-31714b917e37 localadmin openstack] check_instance_lock: arguments: |<nova.compute.manager.ComputeManager object at 0xef1910>| |<nova.rpc.amqp.RpcContext object at 0x35f77d0>| |c813bc01-c590-4ede-bcf4-fdcbe5414b00|
2012-03-19 17:23:09 DEBUG nova.compute.manager [req-d7d78fd8-093d-4ea9-8cc4-31714b917e37 localadmin openstack] instance c813bc01-c590-4ede-bcf4-fdcbe5414b00: getting locked state from (pid=3939) get_lock /home/localadmin/openstack/nova/nova/compute/manager.py:1577
2012-03-19 17:23:09 INFO nova.compute.manager [req-d7d78fd8-093d-4ea9-8cc4-31714b917e37 localadmin openstack] check_instance_lock: locked: |False|
2012-03-19 17:23:09 INFO nova.compute.manager [req-d7d78fd8-093d-4ea9-8cc4-31714b917e37 localadmin openstack] check_instance_lock: admin: |True|
2012-03-19 17:23:09 INFO nova.compute.manager [req-d7d78fd8-093d-4ea9-8cc4-31714b917e37 localadmin openstack] check_instance_lock: executing: |<function terminate_instance at 0x1c51398>|
2012-03-19 17:23:09 AUDIT nova.compute.manager [req-d7d78fd8-093d-4ea9-8cc4-31714b917e37 localadmin openstack] [instance: c813bc01-c590-4ede-bcf4-fdcbe5414b00] Terminating instance
2012-03-19 17:23:09 DEBUG nova.rpc.common [-] Making asynchronous call on network ... from (pid=3939) multicall /home/localadmin/openstack/nova/nova/rpc/amqp.py:319
2012-03-19 17:23:09 DEBUG nova.rpc.common [-] MSG_ID is 738bf27394b34afc996dcdf766ac2f8c from (pid=3939) multicall /home/localadmin/openstack/nova/nova/rpc/amqp.py:322
2012-03-19 17:23:09 DEBUG nova.rpc.common [-] Pool creating new connection from (pid=3939) create /home/localadmin/openstack/nova/nova/rpc/amqp.py:57
2012-03-19 17:23:09 INFO nova.rpc.common [-] Connected to AMQP server on 10.228.24.60:5672

Revision history for this message
Shirley Woo (swoo) said : 		#8

The instance is not getting deleted.
"nova list" output shows there are still 10 instances.

Revision history for this message
John Garbutt (johngarbutt) said : 		#9

Ouch, that looks annoying. Afraid I have not see that set of errors before.

Since you are using DevStack, it might be easier to start again (although it would be nice to know what has cause all your pain):
killall screen
./stack.sh

When you try to create an instance (easier using horizon, you can see what is going on better), you can then trace what is happening through the different logs, and you should be able to find the route cause of your issues.

Just want to double check, but I assume there is just a single XenSever with a single DomU running the nova services? And you are using EXT storage for your local storage SR?

Revision history for this message
Shirley Woo (swoo) said : 		#10

yes, I'm about at the point of starting again.

Yes, one XenServer with a single DomU running the nova services. Local storage.

Revision history for this message
Shirley Woo (swoo) said : 		#11

It seems that the table for the instances is not in mysql. Is that true?
I thought I'd force delete the instances in the db, but since it doesn't seem to be in mysql, not sure where it is. Any ideas?

Revision history for this message
Shirley Woo (swoo) said : 		#12

Found that the table was actually in the mysql db on server1 not on server2.
This, of course, has be a bit confused. Are there suppose to be separate mysql dbs on sever1 and sever2?

Revision history for this message
Shirley Woo (swoo) said : 		#13

I was able to workaround the failed nova delete by directly going into mysql on server1 and deleted the entries in the instance table.

But still not clear why the instances creation is failing.

Revision history for this message
Vish Ishaya (vishvananda) said : 		#14

no there are not. you need everything talking to the same mysql and rabbit servers.

You should set:
rabbit_host
and:
sql_connection

in your conf file

Can you help with this problem?

Provide an answer of your own, or ask Shirley Woo for more information if necessary.

To post a message you must log in.