neutron

Devstack Folsom: should I be able to ping VM's private IP from host?

Asked by Will Dennis

Hi all,

I have installed Devstack (Folsom, using Quantum) via the "stack.sh" installer. Here's the relevant bits of my localrc:

HOST_IP=111.22.100.100
FLOATING_RANGE=111.22.100.128/28
FIXED_RANGE=192.168.0.0/24
FIXED_NETWORK_SIZE=256
FLAT_INTERFACE=
NETWORK_GATEWAY=192.168.0.1
EXT_GW_IP=111.22.100.254

disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service quantum
enable_service g-api
enable_service rabbit

So after the installer runs, I see that I have two OVS bridges "br-ex" and "br-int", and from digging around, I see that they are connected by a Quantum router named "router1". The lowest avail IP address on the FLOATING_RANGE is in use by br-ex, and the second lowest by the router's gateway to br-ex. On the "private" network's subnet, the lowest address (192.168.0.1) is in use by the router, and the second lowest looks like it's in use by the DHCP service. When I spawn a VM, it predictably gets the third-lowest IP address of 192.168.0.3 assigned to it. All this makes sense to me. The VM is in the "default" secgroup, and I have authored a rule in that group to allow all ICMP from anywhere inbound. The routing table of the host looks correct as well:

$ netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 111.22.100.254 0.0.0.0 UG 0 0 0 eth0
111.22.100.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
111.22.100.128 0.0.0.0 255.255.255.240 U 0 0 0 br-ex
192.168.0.0 111.22.100.130 255.255.255.0 UG 0 0 0 br-ex

I am able to ping the two router IP's (in my case, 111.22.100.130 and 192.168.0.1) from the host, but when I try to ping the VM, I'm getting "destination unreachable" errors. I even tried allocating a floating IP and assigning it to the VM's port, but I get the same thing (destination unreachable) when I ping the floating IP that was assigned. Why am I not able to ping my VM?

Question information

Language:
English Edit question
Status:
Solved
For:
neutron Edit question
Assignee:
No assignee Edit question
Solved by:
Will Dennis
Solved:
Last query:
Last reply:
Revision history for this message
ZhiQiang Fan (aji-zqfan) said : 		#1

use "ip netns" to get you network namespace
it shall be print something like qrouter-xxxxx
use "ip netns exec qrouter-xxxxx ping $your_private_ip"
you cannot access an instance by private ip in another namespace unless via a router

Revision history for this message
Will Dennis (willard-dennis) said : 		#2

Hi, @zqfan - thanks for responding.
The use of "network namespaces" to implement the Quantum network architecture is a new technology for me. (Heck, the whole Openstack thing is a new technology to me!) In any case, I don't know how exactly that impacts things.

Here is a diagram I made up of my perception of the Quantum network topology on the host I'm running Devstack on:
https://www.dropbox.com/s/ikevzkl2quju9by/devstack-quantum-arch.png

From the host system itself, I can ping both the external and internal "router1" interfaces:

stacker@devstack01:~/devstack$ ping 111.22.100.130
PING 111.22.100.130 (111.22.100.130) 56(84) bytes of data.
64 bytes from 111.22.100.130: icmp_req=1 ttl=64 time=0.318 ms
64 bytes from 111.22.100.130: icmp_req=2 ttl=64 time=0.055 ms
64 bytes from 111.22.100.130: icmp_req=3 ttl=64 time=0.031 ms
^C
--- 111.22.100.130 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.031/0.134/0.318/0.130 ms

stacker@devstack01:~/devstack$ ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1: icmp_req=1 ttl=64 time=0.365 ms
64 bytes from 192.168.0.1: icmp_req=2 ttl=64 time=0.045 ms
64 bytes from 192.168.0.1: icmp_req=3 ttl=64 time=0.028 ms
^C
--- 192.168.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.028/0.146/0.365/0.155 ms

But if I try to ping my VM (which reports an IP address of '192.168.0.3' thru nova list) it does not work:

stacker@devstack01:~/devstack$ ping 192.168.0.3
PING 192.168.0.3 (192.168.0.3) 56(84) bytes of data.
^C
--- 192.168.0.3 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6047ms

(I also cannot ping the "DHCP server" at 192.168.0.2, but that's less of a worry right now.)

If I list the network namespaces, I see:

stacker@devstack01:~/devstack$ ip netns list
qrouter-893cc9d9-c06a-4dcf-aa79-bee6dceac3c5
qdhcp-4aadea46-cb7e-44cb-a037-af0e6dee11c4

Then I look at the routes for both namespaces:

stacker@devstack01:~/devstack$ sudo ip netns exec qrouter-893cc9d9-c06a-4dcf-aa79-bee6dceac3c5 ip route
default via 111.22.100.129 dev qg-48bcd8f6-c6
111.22.100.128/28 dev qg-48bcd8f6-c6 proto kernel scope link src 111.22.100.130
192.168.0.0/24 dev qr-04eb6aba-d7 proto kernel scope link src 192.168.0.1

stacker@devstack01:~/devstack$ sudo ip netns exec qdhcp-4aadea46-cb7e-44cb-a037-af0e6dee11c4 ip route
192.168.0.0/24 dev tap3d5f026d-b8 proto kernel scope link src 192.168.0.2

...which looks as expected. So then I tried some pings from these namespaces:

stacker@devstack01:~/devstack$ sudo ip netns exec qdhcp-4aadea46-cb7e-44cb-a037-af0e6dee11c4 ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1: icmp_req=1 ttl=64 time=0.392 ms
64 bytes from 192.168.0.1: icmp_req=2 ttl=64 time=0.057 ms
64 bytes from 192.168.0.1: icmp_req=3 ttl=64 time=0.038 ms
^C
--- 192.168.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.038/0.162/0.392/0.162 ms

stacker@devstack01:~/devstack$ sudo ip netns exec qrouter-893cc9d9-c06a-4dcf-aa79-bee6dceac3c5 ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
64 bytes from 192.168.0.2: icmp_req=1 ttl=64 time=0.304 ms
64 bytes from 192.168.0.2: icmp_req=2 ttl=64 time=0.036 ms
64 bytes from 192.168.0.2: icmp_req=3 ttl=64 time=0.053 ms
^C
--- 192.168.0.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.036/0.131/0.304/0.122 ms

...looks good; but when I tried pinging VM from the either the router or DHCP namespaces:

stacker@devstack01:~/devstack$ sudo ip netns exec qrouter-893cc9d9-c06a-4dcf-aa79-bee6dceac3c5 ping 192.168.0.3
PING 192.168.0.3 (192.168.0.3) 56(84) bytes of data.
From 192.168.0.1 icmp_seq=1 Destination Host Unreachable
From 192.168.0.1 icmp_seq=2 Destination Host Unreachable
From 192.168.0.1 icmp_seq=3 Destination Host Unreachable
^C
--- 192.168.0.3 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2999ms

stacker@devstack01:~/devstack$ sudo ip netns exec qdhcp-4aadea46-cb7e-44cb-a037-af0e6dee11c4 ping 192.168.0.3
PING 192.168.0.3 (192.168.0.3) 56(84) bytes of data.
From 192.168.0.2 icmp_seq=1 Destination Host Unreachable
From 192.168.0.2 icmp_seq=2 Destination Host Unreachable
From 192.168.0.2 icmp_seq=3 Destination Host Unreachable
^C
--- 192.168.0.3 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4009ms

...it does not work. It should, right?

Revision history for this message
Will Dennis (willard-dennis) said : 		#3

UPDATE:

Looked at the logs from the VM (using included 'cirros' image.) I see the following in the logs:

Starting network...
udhcpc (v1.18.5) started
Sending discover...
Sending discover...
Sending discover...
No lease, failing
WARN: /etc/rc3.d/S40-network failed

Thankfully, cirros has a way to log in without SSH keys :) so upon logging in as user cirros and looking around, I see that the 'eth0' NIC doesn't have an IPv4 address. Upon setting one with 'ifconfig eth0 192.168.0.3 netmask 255.255.255.0 up' and then setting the default gateway via 'route add default gw 192.168.0.1 eth0' I then can ping out from the VM all the way to my host's IP, and, I can ping from the host to the VM now:

stacker@devstack01:~/devstack$ ping 192.168.0.3
PING 192.168.0.3 (192.168.0.3) 56(84) bytes of data.
64 bytes from 192.168.0.3: icmp_req=1 ttl=63 time=18.5 ms
64 bytes from 192.168.0.3: icmp_req=2 ttl=63 time=0.439 ms
64 bytes from 192.168.0.3: icmp_req=3 ttl=63 time=0.450 ms
^C
--- 192.168.0.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.439/6.477/18.542/8.531 ms

So, looks like this is not a Quantum networking stack issue, but instead a DHCP issue...

I tied a tcpdump on the q-dhcp netns device, here's what I saw:

stacker@devstack01:~/devstack$ sudo ip netns exec qdhcp-4aadea46-cb7e-44cb-a037-af0e6dee11c4 tcpdump -i tap3d5f026d-b8 | grep DHCP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap3d5f026d-b8, link-type EN10MB (Ethernet), capture size 65535 bytes
17:22:53.252116 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:ad:e7:c0 (oui Unknown), length 280
17:22:56.246977 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:ad:e7:c0 (oui Unknown), length 280
^C167 packets captured
167 packets received by filter
0 packets dropped by kernel

So, DHCP discover messages are coming in, but no offers going out... Is there a problem with dnsmasq? How can I debug?

Revision history for this message
Will Dennis (willard-dennis) said : 		#4

So, I think I will close out this question, since I have narrowed down the problem to DHCP, and will start a new question to try to resolve that problem.

The answer to the original question is "YES" based on my testing. I believe to be able to ping (or otherwise access) the VM from outside the host system, one would "publish" the VM out to the public network by assigning it a floating IP, which is in the publicly-accessible IP space. But you can ping/access the VM from the host system (running OpenStack), as it has the necesarry routing set up (and given that the correct secgroup access rules are in place.)

Revision history for this message
ZhiQiang Fan (aji-zqfan) said : 		#5

thanks for so much work and sorry for this delay reply because i take some time to test in my environment to reappear your problem

instance will get an ip from dhcp if your route is set correctly
in my case, if i set a wrong route with destination is instance's private network and via a gateway( no matter qg-xx or br-xx or ehtxx), instance will discover dhcp then fail when boot up, so i'm not quite sure if your route 192.168.0.0 via 111.22.100.130 is necessary but i think it may effect the dhcp reply

good luck, any clarification will be appreciated. please email to <email address hidden>

Revision history for this message
ZhiQiang Fan (aji-zqfan) said : 		#6

and here is a link may be helpful for you
https://answers.launchpad.net/nova/+question/162919
although it occurs on essex