OpenStack Compute (Nova)

Instance metadata not found (http error [404]) for instances started on a compute node

Asked by Ahmad Al-Shishtawy on 2012-02-27

I installed OpenStack Diablo release on two nodes: a controller node with (api, compute, network, scheduler, vncproxy) and a compute node with (compute, network). I use FlatDHCPManager.

The problem is that instances started on the compute node can not find meta data (they can reach the api server)
I can ping those instances but not ssh. The console output contains the following error:
...
cloud-init start-local running: Mon, 27 Feb 2012 19:48:47 +0000. up 1.95 seconds
no instance data found in start-local
cloud-init-nonet waiting 120 seconds for a network device.
ci-info: lo : 1 127.0.0.1 255.0.0.0
ci-info: eth0 : 1 192.168.22.40 255.255.255.0 02:16:3e:70:79:48
ci-info: route-0: 0.0.0.0 192.168.22.20 0.0.0.0 eth0 UG
ci-info: route-1: 192.168.22.0 0.0.0.0 255.255.255.0 eth0 U
cloud-init start running: Mon, 27 Feb 2012 19:48:59 +0000. up 14.12 seconds
2012-02-27 19:49:09,381 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: http error [404]
2012-02-27 19:49:10,487 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: http error [404]
2012-02-27 19:49:11,591 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: http error [404]
...

instances on the controller node run correctly. I also use the flag --ec2_dmz_host on the compute node

Question information

Language:
English Edit question
Status:
Answered
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Last query:
2012-02-27
Last reply:
2012-07-18

When I remove the flag --ec2_dmz_host from the compute node the console output becomes:

cloud-init start-local running: Mon, 27 Feb 2012 20:54:39 +0000. up 1.91 seconds
no instance data found in start-local
ci-info: lo : 1 127.0.0.1 255.0.0.0
ci-info: eth0 : 1 192.168.22.41 255.255.255.0 02:16:3e:52:1f:dd
ci-info: route-0: 0.0.0.0 192.168.22.20 0.0.0.0 eth0 UG
ci-info: route-1: 192.168.22.0 0.0.0.0 255.255.255.0 eth0 U
cloud-init start running: Mon, 27 Feb 2012 20:54:53 +0000. up 15.14 seconds
2012-02-27 20:55:03,278 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: url error [[Errno 111] Connection refused]
2012-02-27 20:55:04,282 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: url error [[Errno 111] Connection refused]

So it seems in the original setup, with the --ec2_dmz_host flag, instances can reach the api server. Why can't find metadata?

Vish Ishaya (vishvananda) said : #2

Your traffic is probably being snatted, so the metadata server gets the wrong source ip and can't look up info. Try setting
--dmz_cidr to the ip address of your api host
e.g. --dmz_cidr=192.168.22.1/32 (or wherever your api host is running)

On Feb 27, 2012, at 1:00 PM, Ahmad Al-Shishtawy wrote:

> Question #189026 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/189026
>
> Ahmad Al-Shishtawy gave more information on the question:
> When I remove the flag --ec2_dmz_host from the compute node the console
> output becomes:
>
> cloud-init start-local running: Mon, 27 Feb 2012 20:54:39 +0000. up 1.91 seconds
> no instance data found in start-local
> ci-info: lo : 1 127.0.0.1 255.0.0.0
> ci-info: eth0 : 1 192.168.22.41 255.255.255.0 02:16:3e:52:1f:dd
> ci-info: route-0: 0.0.0.0 192.168.22.20 0.0.0.0 eth0 UG
> ci-info: route-1: 192.168.22.0 0.0.0.0 255.255.255.0 eth0 U
> cloud-init start running: Mon, 27 Feb 2012 20:54:53 +0000. up 15.14 seconds
> 2012-02-27 20:55:03,278 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: url error [[Errno 111] Connection refused]
> 2012-02-27 20:55:04,282 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: url error [[Errno 111] Connection refused]
>
> So it seems in the original setup, with the --ec2_dmz_host flag,
> instances can reach the api server. Why can't find metadata?
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

When I set --dmz_cidr=<api host ip>/32 I get:

2012-02-27 21:35:50,279 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: url error [timed out]

When I set --dmz_cidr=192.168.22.1/32 I get same error:
2012-02-27 21:41:28,384 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: http error [404]

I forgot to mention that I have one nic (eth0)

Xiaolin Zhang (zhangxiaolins) said : #4

Same issue here, after I added iptables nova-network-PREROUTING

cloud-init start-local running: Thu, 01 Mar 2012 07:59:09 +0000. up 1.70 seconds
no instance data found in start-local
ci-info: lo : 1 127.0.0.1 255.0.0.0
ci-info: eth0 : 1 10.100.80.3 255.255.255.0 02:16:3e:6f:92:f3
ci-info: route-0: 0.0.0.0 10.100.80.1 0.0.0.0 eth0 UG
ci-info: route-1: 10.100.80.0 0.0.0.0 255.255.255.0 eth0 U
cloud-init start running: Thu, 01 Mar 2012 07:59:11 +0000. up 3.03 seconds
2012-03-01 07:59:11,365 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: http error [404]
2012-03-01 07:59:12,433 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: http error [404]
2012-03-01 07:59:13,509 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: http error [404]
2012-03-01 07:59:14,577 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: http error [404]

Xiaolin Zhang (zhangxiaolins) said : #5

For multi-node installation, the root cause for both http error[404] or url error are, I think, request bad url upon vm initialization when starting up on compute mode, which is resulted from bad nova.conf configuration, esp. the network config.

When I changed nova.conf at each of the compute node as follows, all vms can be set up correctly on each of node, and ssh to the vm would be fine as well.

--network_manager=nova.network.manager.FlatDHCPManager
--fixed_range=10.100.80.0/24
--network_size=256
--flat_network_dhcp_start=10.100.80.2
--public_interface=eth1
--flat_interface=eth0
--flat_network_bridge=br100
--flat_injected=false

P.S., the scenario for my deployment is as fallows,
control: 10.100.79.136
compute node: 10.100.79.136, 10.100.79.137.

and I created the private network as:
nova-manage network create private --fixed_range_v4=10.100.80.0/24 --num_networks=1 --network_size=256 --multi_host=T

Dražen Lučanin (kermit666) said : #6

See a possible solution here:

http://askubuntu.com/questions/165115/ubuntu-cloud-instances-in-openstack-cant-import-public-ssh-key

Basically, you should try restarting nova-api and rebooting your instance:

    $ sudo restart nova-api

Can you help with this problem?

Provide an answer of your own, or ask Ahmad Al-Shishtawy for more information if necessary.

To post a message you must log in.