OpenStack Compute (nova)

multi-node compute: Determining IP information for eth0... failed

Asked by josh

Hi All,

I have configured a two-node Ubuntu 11.04 based openstack diablo compute environment. Both nodes are running 11.04 server. The controller node has enabled all required nova services including nova-compute and nova-network, and the compute node is only running nova-compute.

For my experiment, I also created a customized CentOS 5.7 image at the controller's glance server.

So far from the controller node, running euca-run-instances command, I have no problem spawning instances onto the controller node. I am able to ssh into the spawned instance from the controller.

Now I disabled nova-compute on the controller node, and enabled nova-compute on the pure compute node. But after I started euca-run-instances from the controller, from the compute node's nova-compute.log, I found the instance was spawned successfully. For instance,

2012-01-14 14:35:08,588 INFO nova.virt.libvirt_conn [-] Instance instance-00000095 destroyed successfully.

However, I cannot ping the instance. After I checked the instance's console.log, I found its network was not started successfully at the booting time, and saw the error message,

Bringing up interface eth0:
Determining IP information for eth0... failed.
[FAILED]^M

I have tried hard on modifying both nova.conf and /etc/network/interfaces, but had no luck to fix the problem. So I desparately need your help based on the following configuration:

On the controller node, ospc001, I have nova.conf like this:
root@ospc001:~/euca# cat /etc/nova/nova.conf
--network_manager=nova.network.manager.FlatDHCPManager
--flat_network_dhcp_start=192.168.0.2
--flat_network_bridge=br100
--public_interface=eth0
--flat_injected=False
--use_ipv6=false
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--force_dhcp_release=True
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--flagfile=/etc/nova/nova-compute.conf
--use_deprecated_auth
--verbose
--my_ip=192.55.50.172
--network_host=ospc001
--s3_host=192.55.50.172
--rabbit_host=192.55.50.172
--cc_host=192.55.50.172
--ec2_url=http://192.55.50.172:8773/services/Cloud
--s3_url=http://192.55.50.172:3333
--vncproxy_url=http://192.55.50.172:6080
--sql_connection=mysql://nova:notnova@192.55.50.172/nova
--routing_source_ip=192.55.50.172
--glance_host=192.55.50.172
--glance_port=9292
--glance_api_servers=192.55.50.172:9292
--image_service=nova.image.glance.GlanceImageService
--fixed_range=192.168.0.0/16
--network_size=8
--FAKE_subdomain=ec2
--iscsi_ip_prefix=192.168.0
--libvirt_type=kvm

root@ospc001:~/euca# ps -ef | grep dnsmasq
nobody 6928 1 0 Jan12 ? 00:00:05 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.0.1 --except-interface=lo --dhcp-range=192.168.0.2,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro
root 6929 6928 0 Jan12 ? 00:00:04 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.0.1 --except-interface=lo --dhcp-range=192.168.0.2,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro

On the controller node, I have network interface like this

root@ospc001:~/euca# cat /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#auto br100
auto eth0
iface eth0 inet static
        address 192.55.50.172
        netmask 255.255.255.0
        network 192.55.50.0
        broadcast 192.55.50.255
        gateway 192.55.50.1
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 8.8.8.8

root@ospc001:~/euca# ifconfig
br100 Link encap:Ethernet HWaddr fe:16:3e:3e:8d:de
          inet addr:192.168.0.1 Bcast:192.168.0.255 Mask:255.255.255.0
          inet6 addr: fe80::48b2:55ff:feb2:e310/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:31893646 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20007570 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:43891773305 (43.8 GB) TX bytes:6601209049 (6.6 GB)

eth0 Link encap:Ethernet HWaddr 00:24:81:88:c0:76
          inet addr:192.55.50.172 Bcast:192.55.50.255 Mask:255.255.255.0
          inet6 addr: fe80::224:81ff:fe88:c076/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:60173158 errors:0 dropped:0 overruns:0 frame:0
          TX packets:106526500 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:9126572449 (9.1 GB) TX bytes:145891796019 (145.8 GB)
          Interrupt:16 Memory:f8000000-f8012800

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:77599939 errors:0 dropped:0 overruns:0 frame:0
          TX packets:77599939 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:804933672914 (804.9 GB) TX bytes:804933672914 (804.9 GB)

virbr0 Link encap:Ethernet HWaddr 02:eb:e4:dc:ca:02
          inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

vnet0 Link encap:Ethernet HWaddr fe:16:3e:3e:8d:de
          inet6 addr: fe80::fc16:3eff:fe3e:8dde/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:36392297 errors:0 dropped:0 overruns:0 frame:0
          TX packets:21283892 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:51027008608 (51.0 GB) TX bytes:6694949201 (6.6 GB)

vnet1 Link encap:Ethernet HWaddr fe:16:3e:4a:ee:5d
          inet6 addr: fe80::fc16:3eff:fe4a:ee5d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:11770 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10285 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:1291390 (1.2 MB) TX bytes:1519580 (1.5 MB)

vnet2 Link encap:Ethernet HWaddr fe:16:3e:4f:ad:cf
          inet6 addr: fe80::fc16:3eff:fe4f:adcf/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:18439 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16081 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:2059159 (2.0 MB) TX bytes:2390970 (2.3 MB)

vnet3 Link encap:Ethernet HWaddr fe:16:3e:74:92:90
          inet6 addr: fe80::fc16:3eff:fe74:9290/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:10825 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9374 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:1200309 (1.2 MB) TX bytes:1431800 (1.4 MB)

On the compute node, ospc005, I have nova.conf like this

root@ospc005:~# cat /etc/nova/nova.conf
--network_manager=nova.network.manager.FlatDHCPManager
--flat_network_dhcp_start=192.168.0.2
--public_interface=eth0
--flat_injected=False
--use_ipv6=false
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--force_dhcp_release=True
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--flagfile=/etc/nova/nova-compute.conf
--use_deprecated_auth
--verbose
--my_ip=192.55.50.172
--network_host=ospc001
--s3_host=192.55.50.172
--rabbit_host=192.55.50.172
--cc_host=192.55.50.172
--ec2_url=http://192.55.50.172:8773/services/Cloud
--s3_url=http://192.55.50.172:3333
--vncproxy_url=http://192.55.50.172:6080
--sql_connection=mysql://nova:notnova@192.55.50.172/nova
--routing_source_ip=192.55.50.172
--glance_host=192.55.50.172
--glance_port=9292
--glance_api_servers=192.55.50.172:9292
--image_service=nova.image.glance.GlanceImageService
--fixed_range=192.168.0.0/16
--network_size=8
--FAKE_subdomain=ec2
--iscsi_ip_prefix=192.168.0
--libvirt_type=kvm

On the compute node, I have network interface like this

root@ospc005:~# cat /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
        address 192.55.50.171
        netmask 255.255.255.0
        network 192.55.50.0
        broadcast 192.55.50.255
        gateway 192.55.50.1
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 8.8.8.8

auto br100
iface br100 inet static
        bridge_ports eth1
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0

root@ospc005:~# ifconfig
br100 Link encap:Ethernet HWaddr 00:00:00:00:00:00
          inet6 addr: fe80::bc67:aeff:fe8c:bf65/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:24 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2624 (2.6 KB) TX bytes:2016 (2.0 KB)

eth0 Link encap:Ethernet HWaddr 00:26:9e:02:65:44
          inet addr:192.55.50.171 Bcast:192.55.50.255 Mask:255.255.255.0
          inet6 addr: fe80::226:9eff:fe02:6544/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:4794 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5267 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2394723 (2.3 MB) TX bytes:1993543 (1.9 MB)
          Memory:9bb20000-9bb40000

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

virbr0 Link encap:Ethernet HWaddr da:22:90:e5:b0:05
          inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

root@ospc005:~# ps -ef | grep dnsmasq
nobody 2166 1 0 14:26 ? 00:00:00 dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/libvirt/network/default.pid --conf-file= --except-interface lo --listen-address 192.168.122.1 --dhcp-range 192.168.122.2,192.168.122.254 --dhcp-lease-max=253 --dhcp-no-override

What else information do you need if you can help?

Thanks lot!

Josh

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
josh
Solved:
Last query:
Last reply:
Revision history for this message
josh (shh-han) said : 		#1

On the controller node, I saw this bridge interface

root@ospc001:~/euca# brctl show
bridge name bridge id STP enabled interfaces
br100 8000.fe163e3e8dde no vnet0
                                                        vnet1
                                                        vnet2
                                                        vnet3
virbr0 8000.000000000000 yes

So br100 is sitting on vnet0, not sitting on eth1.
Is it possible a root cause?
In my case, eth1 does not have a valid IP as described below, so neither br100 and vnet0.

On the controller
root@ospc001:~/euca# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 169.254.169.254/32 scope link lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:24:81:88:c0:76 brd ff:ff:ff:ff:ff:ff
    inet 192.55.50.172/24 brd 192.55.50.255 scope global eth0
    inet 192.55.50.173/32 scope global eth0
    inet6 fe80::224:81ff:fe88:c076/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:24:81:88:c0:74 brd ff:ff:ff:ff:ff:ff
4: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 02:eb:e4:dc:ca:02 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
5: br100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fe:16:3e:3e:8d:de brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/24 brd 192.168.0.255 scope global br100
    inet6 fe80::48b2:55ff:feb2:e310/64 scope link
       valid_lft forever preferred_lft forever
8: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether fe:16:3e:3e:8d:de brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fe3e:8dde/64 scope link
       valid_lft forever preferred_lft forever
29: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether fe:16:3e:4f:ad:cf brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fe4f:adcf/64 scope link
       valid_lft forever preferred_lft forever
31: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether fe:16:3e:4a:ee:5d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fe4a:ee5d/64 scope link
       valid_lft forever preferred_lft forever
32: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether fe:16:3e:74:92:90 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fe74:9290/64 scope link
       valid_lft forever preferred_lft forever

But on the compute node, I saw this bridge interface
root@ospc005:~# brctl show
bridge name bridge id STP enabled interfaces
br100 8000.00269e026545 no eth1
virbr0 8000.000000000000 yes

So br100 is sitting sitting on eth1. But again, eth1 does not have a valid IP as shown below, so neither br100.

root@ospc005:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:26:9e:02:65:44 brd ff:ff:ff:ff:ff:ff
    inet 192.55.50.171/24 brd 192.55.50.255 scope global eth0
    inet6 fe80::226:9eff:fe02:6544/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 00:26:9e:02:65:45 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:26:9e:02:65:46 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:26:9e:02:65:47 brd ff:ff:ff:ff:ff:ff
6: br100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 00:26:9e:02:65:45 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::226:9eff:fe02:6545/64 scope link
       valid_lft forever preferred_lft forever
7: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 86:af:07:08:aa:a4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

Revision history for this message
Zhongyue Luo (zyluo) said : 		#2

You need the nova-compute, nova-network, and metadata-api to run together on multi-node mode. Tthe metadata ip has to be its ip in each nova.conf

Revision history for this message
josh (shh-han) said : 		#3

Thanks for reply. I solved my problem by reconfiguring br100 on the controller node. It was set on vnet0 (that was automatically set by nova), now I set it on eth1 which has no IP address but is connected to a local private switch. On the compute, I did the same thing. So VMs on the compute node can communicate with the controller nova-network via a matching pair of br100, and obtained valid private IP address.

What if I have only a single NIC on the controller and compute node? I know if eth0 is a dhcp interface, setting br100 on eth0 will work. But if eth0 is static interface, then setting br100 on eth0 won't work because their interfaces are conflicting to each other, one dhcp and one static.

Revision history for this message
Vish Ishaya (vishvananda) said : 		#4

nova will move the ip from eth0 to br100 when it creates it. So it actually works better without dhcp on eth0

On Jan 26, 2012, at 6:11 PM, josh wrote:

> Question #184736 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/184736
>
> Status: Answered => Solved
>
> josh confirmed that the question is solved:
> Thanks for reply. I solved my problem by reconfiguring br100 on the
> controller node. It was set on vnet0 (that was automatically set by
> nova), now I set it on eth1 which has no IP address but is connected to
> a local private switch. On the compute, I did the same thing. So VMs on
> the compute node can communicate with the controller nova-network via a
> matching pair of br100, and obtained valid private IP address.
>
> What if I have only a single NIC on the controller and compute node? I
> know if eth0 is a dhcp interface, setting br100 on eth0 will work. But
> if eth0 is static interface, then setting br100 on eth0 won't work
> because their interfaces are conflicting to each other, one dhcp and one
> static.
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).