Nova Netowrk Multihost Issue

Asked by Stephen on 2013-01-25

I am setting up a FlatDHCPManager network using multi host. I am having an issue where my cloud vms don't boot correctly because they can't communicate with the metadata service. Nova network is installed on both the controller and the compute node. The vm receives an IP address, but boots with the following errors:

cloud-init start-local running: Fri, 25 Jan 2013 21:19:24 +0000. up 2.58 seconds
no instance data found in start-local
ci-info: lo : 1 127.0.0.1 255.0.0.0 .
ci-info: eth0 : 1 10.0.1.130 255.255.255.128 fa:16:3e:41:08:3e
ci-info: route-0: 0.0.0.0 10.0.1.129 0.0.0.0 eth0 UG
ci-info: route-1: 10.0.1.128 0.0.0.0 255.255.255.128 eth0 U
cloud-init start running: Fri, 25 Jan 2013 21:19:25 +0000. up 2.98 seconds
2013-01-25 21:19:47,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:19:51,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [5/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:19:54,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [8/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:19:57,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [11/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:20:00,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [14/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:20:03,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [17/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:20:06,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [20/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:20:09,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [23/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:20:12,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [26/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:20:15,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [29/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:20:18,146 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [32/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:20:24,150 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [38/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:20:30,154 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [44/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:20:36,158 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:20:42,162 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [57/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:20:48,166 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [63/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:20:55,170 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [70/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:21:02,174 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [77/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:21:09,178 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [84/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:21:16,182 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [91/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:21:23,186 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [98/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:21:31,190 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [106/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:21:39,194 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [114/120s]: url error [[Errno 113] No route to host]
2013-01-25 21:21:44,200 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: url error [[Errno 115] Operation now in progress]
2013-01-25 21:21:49,206 - DataSourceEc2.py[CRITICAL]: giving up on md after 124 seconds

no instance data found in start

Here are the interface configurations for my controller and compute nodes:

irad-controller@irad-controller:~$ ifconfig
lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:8221 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8221 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3875243 (3.8 MB) TX bytes:3875243 (3.8 MB)

p4p1 Link encap:Ethernet HWaddr 00:25:90:7d:a0:f2
          inet addr:10.0.1.4 Bcast:10.0.1.255 Mask:255.255.255.0
          inet6 addr: fe80::225:90ff:fe7d:a0f2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:2697 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2590 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1064317 (1.0 MB) TX bytes:1325297 (1.3 MB)
          Interrupt:16 Memory:f7a00000-f7a20000

p5p1 Link encap:Ethernet HWaddr 00:25:90:7d:a0:f3
          inet addr:7.7.7.101 Bcast:7.7.7.255 Mask:255.255.255.0
          inet6 addr: fe80::225:90ff:fe7d:a0f3/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:468 errors:0 dropped:0 overruns:0 frame:0
          TX packets:425 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:49016 (49.0 KB) TX bytes:72701 (72.7 KB)
          Interrupt:17 Memory:f7900000-f7920000

irad-compute@irad-compute:~$ ifconfig
br100 Link encap:Ethernet HWaddr 00:25:90:91:2f:c6
          inet addr:10.0.1.131 Bcast:10.0.1.255 Mask:255.255.255.128
          inet6 addr: fe80::a8a3:3bff:fe51:76d1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:2203 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2492 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1314972 (1.3 MB) TX bytes:1049064 (1.0 MB)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:84 errors:0 dropped:0 overruns:0 frame:0
          TX packets:84 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:6832 (6.8 KB) TX bytes:6832 (6.8 KB)

p255p1 Link encap:Ethernet HWaddr 00:25:90:91:2f:c6
          inet6 addr: fe80::225:90ff:fe91:2fc6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:2785 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2792 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1421687 (1.4 MB) TX bytes:1075777 (1.0 MB)
          Memory:df920000-df940000

p255p2 Link encap:Ethernet HWaddr 00:25:90:91:2f:c7
          inet addr:7.7.7.102 Bcast:7.7.7.255 Mask:255.255.255.0
          inet6 addr: fe80::225:90ff:fe91:2fc7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:248 errors:0 dropped:0 overruns:0 frame:0
          TX packets:262 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:24649 (24.6 KB) TX bytes:27432 (27.4 KB)
          Memory:df900000-df920000

vnet0 Link encap:Ethernet HWaddr fe:16:3e:41:08:3e
          inet6 addr: fe80::fc16:3eff:fe41:83e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:138 errors:0 dropped:0 overruns:0 frame:0
          TX packets:200 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:5928 (5.9 KB) TX bytes:38396 (38.3 KB)

Is it a problem that my compute node does not have a bridge interface configured?

Thank you!

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
Stephen
Solved:
2013-01-28
Last query:
2013-01-28
Last reply:
2013-01-27
Keith Tobin (keith-tobin) said : #1

I need to see config files for controller and compute, I also want to see you 'interfaces' file.

Stephen (stephen-kahmann) said : #2

Sure, here are the requested files:

Controller Config:
=============================
[DEFAULT]

iscsi_helper=tgtadm
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
ec2_private_dns_show_ip=True
volumes_path=/var/lib/nova/volumes

# LOGS/STATE
verbose=True
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/run/lock/nova

# AUTHENTICATION
auth_strategy=keystone

# SCHEDULER
scheduler_driver=nova.scheduler.multi.MultiScheduler
compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler

# CINDER
volume_api_class=nova.volume.cinder.API

# DATABASE
sql_connection=mysql://novaUser:novaPass@10.0.1.4/nova

# COMPUTE
libvirt_type=kvm
libvirt_use_virtio_for_bridges=True
start_guests_on_host_boot=True
resume_guests_state_on_host_boot=True
api_paste_config=/etc/nova/api-paste.ini
allow_admin_api=True
use_deprecated_auth=False
nova_url=http://10.0.1.4:8774/v1.1/
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf

# APIS
ec2_host=10.0.1.4
ec2_url=http://10.0.1.4:8773/services/Cloud
keystone_ec2_url=http://10.0.1.4:5000/v2.0/ec2tokens
s3_host=10.0.1.4
cc_host=10.0.1.4
metadata_host=10.0.1.4
#metadata_listen=0.0.0.0
enabled_apis=ec2,osapi_compute,metadata

# RABBITMQ
rabbit_host=10.0.1.4

# GLANCE
image_service=nova.image.glance.GlanceImageService
glance_api_servers=10.0.1.4:9292

# NETWORK
network_manager=nova.network.manager.FlatDHCPManager
force_dhcp_release=True
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver
public_interface=p5p1
flat_interface=p4p1
flat_network_bridge=br100
fixed_range=10.0.1.129/25
network_size=128
flat_network_dhcp_start=10.0.1.129
flat_injected=False
connection_type=libvirt
multi_host=True

# NOVNC CONSOLE
novnc_enabled=True
novncproxy_base_url=http://7.7.7.101:6080/vnc_auto.html
vncserver_proxyclient_address=10.0.1.4
vncserver_listen=10.0.1.4
=============================

Compute Config:
=============================
[DEFAULT]
iscsi_helper=tgtadm
ec2_private_dns_show_ip=True
volumes_path=/var/lib/nova/volumes

[DEFAULT]

# LOGS/STATE
verbose=True
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/run/lock/nova

# AUTHENTICATION
auth_strategy=keystone

# SCHEDULER
scheduler_driver=nova.scheduler.multi.MultiScheduler
compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler

# CINDER
volume_api_class=nova.volume.cinder.API

# DATABASE
sql_connection=mysql://novaUser:novaPass@10.0.1.4/nova

# COMPUTE
libvirt_type=kvm
libvirt_use_virtio_for_bridges=True
start_guests_on_host_boot=True
resume_guests_state_on_host_boot=True
api_paste_config=/etc/nova/api-paste.ini
allow_admin_api=True
use_deprecated_auth=False
nova_url=http://10.0.1.4:8774/v1.1/
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf

# APIS
ec2_host=10.0.1.4
ec2_url=http://10.0.1.4:8773/services/Cloud
keystone_ec2_url=http://10.0.1.4:5000/v2.0/ec2tokens
s3_host=10.0.1.4
cc_host=10.0.1.4
metadata_host=10.0.1.4
enabled_apis=metadata

# RABBITMQ
rabbit_host=10.0.1.4

# GLANCE
image_service=nova.image.glance.GlanceImageService
glance_api_servers=10.0.1.4:9292

# NETWORK
network_manager=nova.network.manager.FlatDHCPManager
force_dhcp_release=True
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver
public_interface=p255p2
flat_interface=p255p1
flat_network_bridge=br100
fixed_range=10.0.1.129/25
network_size=128
flat_network_dhcp_start=10.0.1.129
flat_injected=False
connection_type=libvirt
multi_host=True

# NOVNC CONSOLE
novnc_enabled=True
novncproxy_base_url=http://7.7.7.101:6080/vnc_auto.html
vncserver_proxyclient_address=10.0.1.2
vncserver_listen=10.0.1.2
=============================

Controller Interfaces:
=============================
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# Management interface
auto p4p1
iface p4p1 inet static
 address 10.0.1.4
 netmask 255.255.255.0
 network 10.0.1.0
 broadcast 10.0.1.255

# The primary network interface
auto p5p1
iface p5p1 inet static
 address 7.7.7.101
 netmask 255.255.255.0
 network 7.7.7.0
 broadcast 7.7.7.255
 gateway 7.7.7.1
 # dns-* options are implemented by the resolvconf package, if installed
 dns-nameservers 7.7.7.1
=============================

Compute Interfaces:
=============================
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# Management Interface
auto p255p1
iface p255p1 inet static
 address 10.0.1.2
 netmask 255.255.255.0

# The primary network interface
auto p255p2
iface p255p2 inet static
 address 7.7.7.102
 netmask 255.255.255.0
 network 7.7.7.0
 broadcast 7.7.7.255
 gateway 7.7.7.1
 # dns-* options are implemented by the resolvconf package, if installed
 dns-nameservers 7.7.7.1
=============================

Thank you Keith!

Stephen (stephen-kahmann) said : #3

Looking through the nova network config I see that the gateway is 10.0.1.129, but that ip is never assigned to an interface:

+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| bridge | br100 |
| bridge_interface | p5p1 |
| broadcast | 10.0.1.255 |
| cidr | 10.0.1.128/25 |
| cidr_v6 | None |
| created_at | 2013-01-23T23:01:49.000000 |
| deleted | False |
| deleted_at | None |
| dhcp_start | 10.0.1.130 |
| dns1 | 8.8.4.4 |
| dns2 | None |
| gateway | 10.0.1.129 |
| gateway_v6 | None |
| host | None |
| id | 74289662-65ed-4f36-90c4-67624e722063 |
| injected | True |
| label | private |
| multi_host | True |
| netmask | 255.255.255.128 |
| netmask_v6 | None |
| priority | None |
| project_id | None |
| rxtx_base | None |
| updated_at | 2013-01-25T16:36:20.000000 |
| vlan | None |
| vpn_private_address | None |
| vpn_public_address | None |
| vpn_public_port | None |
+---------------------+--------------------------------------+

Stephen (stephen-kahmann) said : #4

Also here's my iptables config:

irad-compute@irad-compute:~$ sudo iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N nova-api-metadat-OUTPUT
-N nova-api-metadat-POSTROUTING
-N nova-api-metadat-PREROUTING
-N nova-api-metadat-float-snat
-N nova-api-metadat-snat
-N nova-compute-OUTPUT
-N nova-compute-POSTROUTING
-N nova-compute-PREROUTING
-N nova-compute-float-snat
-N nova-compute-snat
-N nova-network-OUTPUT
-N nova-network-POSTROUTING
-N nova-network-PREROUTING
-N nova-network-float-snat
-N nova-network-snat
-N nova-postrouting-bottom
-A PREROUTING -j nova-network-PREROUTING
-A PREROUTING -j nova-compute-PREROUTING
-A PREROUTING -j nova-api-metadat-PREROUTING
-A OUTPUT -j nova-network-OUTPUT
-A OUTPUT -j nova-compute-OUTPUT
-A OUTPUT -j nova-api-metadat-OUTPUT
-A POSTROUTING -j nova-network-POSTROUTING
-A POSTROUTING -j nova-compute-POSTROUTING
-A POSTROUTING -j nova-api-metadat-POSTROUTING
-A POSTROUTING -j nova-postrouting-bottom
-A nova-api-metadat-snat -j nova-api-metadat-float-snat
-A nova-compute-snat -j nova-compute-float-snat
-A nova-network-POSTROUTING -s 10.0.1.128/25 -d 10.0.1.4/32 -j ACCEPT
-A nova-network-POSTROUTING -s 10.0.1.128/25 -d 10.0.1.128/25 -m conntrack ! --ctstate DNAT -j ACCEPT
-A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.0.1.4:8775
-A nova-network-snat -j nova-network-float-snat
-A nova-network-snat -s 10.0.1.128/25 -o p255p2 -j SNAT --to-source 7.7.7.102
-A nova-postrouting-bottom -j nova-network-snat
-A nova-postrouting-bottom -j nova-compute-snat
-A nova-postrouting-bottom -j nova-api-metadat-snat

Stephen (stephen-kahmann) said : #5

I solved the problem. As it turns out, the default gateway for the nova network was 10.0.1.129, but nova had assigned 10.0.1.131 to br100. All I had to do was 'ip addr add 10.0.1.129 dev br100' and the machine was capable of booting and I can ssh into it. Not sure if this is a bug or known behavior.

Thank you!

Keith Tobin (keith-tobin) said : #6

Would it be possible that at some stage you hard 10.0.1.131 in config and started nova-networking.
Nova-networking would have then populated the network/interfaces with br100 and the incorrect IP.
later when you change it in config, it would not changed in network/interface file, even after restarting nova-networking as it would see their was a br100 already configured.

Thanks for the feed back, it helped me understand you problem.

Stephen (stephen-kahmann) said : #7

The only IPs I configured were the ones in the interface files as this was a fresh install. I'm not sure what caused the issue.
Thanks again for your help. I'm just glad the issue is resolved.