IP address not applied to VM interface

Asked by Stephen

Hello,

I tried a new guide and decided to use nova-network instead of quantum because I was having so many problems with it. The guide I followed can be found here <https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst>.

I am running with a controller and compute node. The services are distributed as follows:

irad-controller@irad-controller:~$ sudo nova-manage service list
Binary Host Zone Status State Updated_At
nova-consoleauth irad-controller nova enabled :-) 2013-01-24 15:53:44
nova-cert irad-controller nova enabled :-) 2013-01-24 15:53:44
nova-scheduler irad-controller nova enabled :-) 2013-01-24 15:53:44
nova-network irad-controller nova enabled :-) 2013-01-24 15:53:44
nova-compute irad-compute nova enabled :-) 2013-01-24 15:53:51
nova-network irad-compute nova enabled :-) 2013-01-24 15:53:49

Currently my only issue is applying an IP (from dhcp) to the vm instances. I have one instance running:

irad-controller@irad-controller:~$ nova list
Please enter password for encrypted keyring:
+--------------------------------------+------+--------+--------------------+
| ID | Name | Status | Networks |
+--------------------------------------+------+--------+--------------------+
| 3049174d-151f-4978-bede-8ff056181788 | Test | ACTIVE | private=10.0.1.130 |
+--------------------------------------+------+--------+--------------------+

You can see that this vm was assigned an IP correctly, however, the IP was never actually applied to the interface as seen from the snippets below:

irad-compute@irad-compute:~$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 169.254.169.254/32 scope link lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: p255p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br100 state UP qlen 1000
    link/ether 00:25:90:91:2f:c6 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::225:90ff:fe91:2fc6/64 scope link
       valid_lft forever preferred_lft forever
3: p255p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:25:90:91:2f:c7 brd ff:ff:ff:ff:ff:ff
    inet 7.7.7.102/24 brd 7.7.7.255 scope global p255p2
    inet6 fe80::225:90ff:fe91:2fc7/64 scope link
       valid_lft forever preferred_lft forever
5: br100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 00:25:90:91:2f:c6 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.131/25 brd 10.0.1.255 scope global br100
    inet 10.0.1.2/24 brd 10.0.1.255 scope global br100
    inet6 fe80::dc2f:d6ff:fee4:39b7/64 scope link
       valid_lft forever preferred_lft forever
#vnet0 is the vm's interface
6: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br100 state UNKNOWN qlen 500
    link/ether fe:16:3e:29:1f:29 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fe29:1f29/64 scope link
       valid_lft forever preferred_lft forever

This is a snippet from the vm's console

cloud-init start-local running: Thu, 24 Jan 2013 15:49:15 +0000. up 2.56 seconds
no instance data found in start-local
cloud-init-nonet waiting 120 seconds for a network device.
cloud-init-nonet gave up waiting for a network device.
ci-info: lo : 1 127.0.0.1 255.0.0.0 .
ci-info: eth0 : 1 . . fa:16:3e:29:1f:29
route_info failed
 * Stopping Handle applying cloud-config[74G[ OK ]
Waiting for network configuration...
Waiting up to 60 more seconds for network configuration...
Booting system without full network configuration...

The bridge interface is set up by nova and seems to understand that it should apply this IP to the interface. Here's the bridge config file:

irad-compute@irad-compute:~$ sudo cat /var/lib/nova/networks/nova-br100.conf
fa:16:3e:29:1f:29,test.novalocal,10.0.1.130

Here's my network interface setup (10.0.1.x = private, 7.7.7.x = public)

Control Node: p4p1 (10.0.1.4), p5p1 (7.7.7.101)
Compute Node: p255p1 (10.0.1.2), p255p2 (7.7.7.102)

Any help would be greatly appreciated. Thank you!!!

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
Stephen
Solved:
Last query:
Last reply:
Revision history for this message
Stephen (stephen-kahmann) said :
#1

I found the following in the /var/log/libvirt/libvirtd.log which seems relevant:

2013-01-24 15:15:00.468+0000: 1683: info : libvirt version: 0.9.13
2013-01-24 15:15:00.468+0000: 1683: error : virNWFilterSnoopLeaseFileRefresh:1882 : open("/var/run/libvirt/network/nwfilter.ltmp"): No such file or directory
2013-01-24 15:47:54.896+0000: 1672: error : virNetSocketReadWire:1006 : End of file while reading data: Input/output error
2013-01-24 15:47:54.897+0000: 1672: error : virNetSocketReadWire:1006 : End of file while reading data: Input/output error
2013-01-24 15:47:55.799+0000: 1672: error : qemuMonitorIO:616 : internal error End of file from monitor
2013-01-24 15:47:55.800+0000: 1672: error : virNWFilterDHCPSnoopEnd:2113 : internal error ifname "vnet0" not in key map
2013-01-24 15:47:55.803+0000: 1672: error : virNetDevGetIndex:657 : Unable to get index for interface vnet0: No such device

Revision history for this message
Stephen (stephen-kahmann) said :
#2

Another relevant piece of information is the dhcp traffic from /var/log/syslog. I only see DHCPOFFERs in this log. No requests or acks:

irad-compute@irad-compute:~$ grep 10.0.1.130 /var/log/syslog
.
.
.
Jan 24 17:14:47 irad-compute dnsmasq-dhcp[2206]: DHCPOFFER(br100) 10.0.1.130 fa:16:3e:29:1f:29
Jan 24 17:14:57 irad-compute dnsmasq-dhcp[2206]: DHCPOFFER(br100) 10.0.1.130 fa:16:3e:29:1f:29
Jan 24 17:15:08 irad-compute dnsmasq-dhcp[2206]: DHCPOFFER(br100) 10.0.1.130 fa:16:3e:29:1f:29
Jan 24 17:15:28 irad-compute dnsmasq-dhcp[2206]: DHCPOFFER(br100) 10.0.1.130 fa:16:3e:29:1f:29
.
.
.

Revision history for this message
Stephen (stephen-kahmann) said :
#3

Performing a tcpdump on br100 on the compute node during a vm reboot I get a lot of ARP, Requests, but no ARP replies:

17:25:19.148508 ARP, Request who-has 10.0.1.130 tell 10.0.1.131, length 28
17:25:20.148478 ARP, Request who-has 10.0.1.130 tell 10.0.1.131, length 28
17:25:21.148485 ARP, Request who-has 10.0.1.130 tell 10.0.1.131, length 28
17:25:24.337513 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:25:90:9d:39:0d, length 548
17:25:27.340290 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:25:90:9d:39:0d, length 548

Revision history for this message
Stephen (stephen-kahmann) said :
#4

After doing a tcpdump on vent0 I am seeing the following information:

10:44:06.344261 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:29:1f:29, length 300
10:44:06.344522 IP 10.0.1.131.67 > 10.0.1.130.68: BOOTP/DHCP, Reply, length 303
10:44:13.372533 ARP, Request who-has 10.0.1.130 tell 10.0.1.131, length 28
10:44:17.725273 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:25:90:9d:39:0d, length 548

I see the br100 interface returning a DHCP reply, but I think there are two problems. One, it looks like there is a DHCP request from both the vent0 interface (fa:16:3e:29:1f:29) and some other interface not known to my machine (00:25:90:9d:39:0d). Two, the DHCP Reply is going to 10.0.1.130 (10.0.1.131.67 > 10.0.1.130.68). Isn't this reply supposed to be broadcast since it hasn't been accepted yet? Perhaps this IP is being accepted by the 00:25:90:9d:39:0d interface.

Regardless, I am going to try the flatNetwork config without DHCP and see how that works. I would really like a solution to this problem as DHCP is what we would like to use, but I need something working today.

Any help would be appreciated. Thank you!

Revision history for this message
Stephen (stephen-kahmann) said :
#5

I am getting the following in my syslog:

Jan 25 11:41:17 irad-compute dnsmasq-dhcp[2254]: DHCPDISCOVER(br100) fa:16:3e:29:1f:29
Jan 25 11:41:17 irad-compute dnsmasq-dhcp[2254]: DHCPOFFER(br100) 10.0.1.130 fa:16:3e:29:1f:29

looks like there are no DHCPREPLY or DHCPACK

Revision history for this message
Stephen (stephen-kahmann) said :
#6

It is now working although honestly I'm not sure what the issue was. I performed the following:

killall dnsmasq; service nova-network restart

as outlined in this thread <https://answers.launchpad.net/nova/+question/181288> then restarted a few times.