VM's can't get IP + 169.254.169.254 unreachable

Asked by Tatenda Mupaso on 2013-01-24

Hi all,

I'm currently running a 3 node setup with controller node, network node with OVS switch plugin and compute node installed using this guide.
https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/GRE/2NICs/OpenStack_Folsom_Install_Guide_WebVersion.rst

When I load up an instance in dashboard, It loads up fine but according to the dashboard, they don't get the actual IP they are supposed to get. I get the error below in the instance log:

Initializing random number generator... done.
Starting network...
udhcpc (v1.18.5) started
Sending discover...
Sending discover...
Sending discover...
No lease, failing
WARN: /etc/rc3.d/S40-network failed
cloud-setup: checking http://169.254.169.254/2009-04-04/meta-data/instance-id
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-setup: failed 1/30: up 10.11. request failed

cloud-setup: failed to read iid from metadata. tried 30
WARN: /etc/rc3.d/S45-cloud-setup failed
Starting dropbear sshd: OK
===== cloud-final: system completely up in 40.37 seconds ====
wget: can't connect to remote host (169.254.169.254): Network is unreachable
wget: can't connect to remote host (169.254.169.254): Network is unreachable
wget: can't connect to remote host (169.254.169.254): Network is unreachable
  instance-id:
  public-ipv4:
  local-ipv4 :
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-userdata: failed to read instance id
WARN: /etc/rc3.d/S99-cloud-userdata failed

I don't know if the problem is dnsmasq, because the guide never fully configures it, or it's nova, or my config setups. My config files are pretty much identical to the config files listed in the guide, except I have different IP's of course. I will paste network config files below.
**************************************************************************
L3_Agent.ini

[DEFAULT]
# Show debugging output in log (sets DEBUG log level output)
# debug = True

# L3 requires that an interface driver be set. Choose the one that best
# matches your plugin.

# OVS
interface_driver = quantum.agent.linux.interface.OVSInterfaceDriver
# LinuxBridge
#interface_driver = quantum.agent.linux.interface.BridgeInterfaceDriver

# The Quantum user information for accessing the Quantum API.
auth_url = http://10.1.1.3:35357/v2.0
auth_region = RegionOne
admin_tenant_name = service
admin_user = quantum
admin_password = password

# Use "sudo quantum-rootwrap /etc/quantum/rootwrap.conf" to use the real
# root filter facility.
# Change to "sudo" to skip the filtering and just run the comand directly
root_helper = sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf

# Allow overlapping IP (Must have kernel build with CONFIG_NET_NS=y and
# iproute2 package that supports namespaces).
use_namespaces = True

# If use_namespaces is set as False then the agent can only configure one router.
# This is done by setting the specific router_id.
router_id = d3724dc3-014a-4014-95ca-5010e12c3d2c

# Each L3 agent can be associated with at most one external network. This
# value should be set to the UUID of that external network. If empty,
# the agent will enforce that only a single external networks exists and
# use that external network id
gateway_external_net_id = a31b9dd9-ce0e-484a-9d12-88eb15bd66b2

# Indicates that this L3 agent should also handle routers that do not have
# an external network gateway configured. This option should be True only
# for a single agent in a Quantum deployment, and may be False for all agents
# if all routers must have an external network gateway
#handle_internal_only_routers = True

# Name of bridge used for external network traffic. This should be set to
# empty value for the linux bridge
external_network_bridge = br-ex

# IP address used by Nova metadata server
metadata_ip = 10.1.1.3

# TCP Port used by Nova metadata server
metadata_port = 8775

# The time in seconds between state poll requests
# polling_interval = 3
*************************************************************************************

DHCP_AGENT.INI

[DEFAULT]
# Show debugging output in log (sets DEBUG log level output)
# debug = true

# Where to store dnsmasq state files. This directory must be writable by the
# user executing the agent. The value below is compatible with a default
# devstack installation.
state_path = /var/lib/quantum

# The DHCP agent will resync its state with Quantum to recover from any
# transient notification or rpc errors. The interval is number of
# seconds between attempts.
# resync_interval = 30

# The DHCP requires that an inteface driver be set. Choose the one that best
# matches you plugin.

# OVS
interface_driver = quantum.agent.linux.interface.OVSInterfaceDriver
# LinuxBridge
#interface_driver = quantum.agent.linux.interface.BridgeInterfaceDriver
# Ryu
#interface_driver = quantum.agent.linux.interface.RyuInterfaceDriver

# The agent can use other DHCP drivers. Dnsmasq is the simplest and requires
# no additional setup of the DHCP server.
dhcp_driver = quantum.agent.linux.dhcp.Dnsmasq

# Allow overlapping IP (Must have kernel build with CONFIG_NET_NS=y and
# iproute2 package that supports namespaces).
use_namespaces = True

# Use "sudo quantum-rootwrap /etc/quantum/rootwrap.conf" to use the real
# root filter facility.
# Change to "sudo" to skip the filtering and just run the comand directly
root_helper = sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf

*****************************************************************************************

Other troubleshooting

root@quantum:~# sudo ovs-vsctl show
3991bc6c-14ac-4db5-b55a-2c98afd8a4da
    Bridge br-int
        Port "qr-95da0388-d3"
            tag: 1
            Interface "qr-95da0388-d3"
                type: internal
        Port "tap66ba434e-f6"
            tag: 1
            Interface "tap66ba434e-f6"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
    Bridge "br-eth1"
        Port "br-eth1"
            Interface "br-eth1"
                type: internal
        Port "eth1"
            Interface "eth1"
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-1"
            Interface "gre-1"
                type: gre
                options: {in_key=flow, out_key=flow, remote_ip="10.1.1.5"}
        Port br-tun
            Interface br-tun
                type: internal
    Bridge br-ex
        Port "qg-939d57b6-d1"
            Interface "qg-939d57b6-d1"
                type: internal
        Port "eth0"
            Interface "eth0"
        Port br-ex
            Interface br-ex
                type: internal
    ovs_version: "1.4.3"

***********************************************************************************

root@quantum:~# ip netns list
qdhcp-eec84bc1-756e-4c9d-a32a-b03d7a799f80
qrouter-d3724dc3-014a-4014-95ca-5010e12c3d2c

***********************************************************************************

root@quantum:~# ip netns exec qrouter-d3724dc3-014a-4014-95ca-5010e12c3d2c ping 50.50.1.1
PING 50.50.1.1 (50.50.1.1) 56(84) bytes of data.
64 bytes from 50.50.1.1: icmp_req=1 ttl=64 time=0.061 ms
64 bytes from 50.50.1.1: icmp_req=2 ttl=64 time=0.056 ms
64 bytes from 50.50.1.1: icmp_req=3 ttl=64 time=0.055 ms
64 bytes from 50.50.1.1: icmp_req=4 ttl=64 time=0.055 ms
^C
--- 50.50.1.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.055/0.056/0.061/0.009 ms
root@quantum:~# ip netns exec qrouter-d3724dc3-014a-4014-95ca-5010e12c3d2c ping 50.50.1.2
PING 50.50.1.2 (50.50.1.2) 56(84) bytes of data.
From 50.50.1.1 icmp_seq=1 Destination Host Unreachable
From 50.50.1.1 icmp_seq=2 Destination Host Unreachable
From 50.50.1.1 icmp_seq=3 Destination Host Unreachable

**********************************************************************************

Anything I need to post for help will happily do

Question information

Language:
English Edit question
Status:
Answered
For:
neutron Edit question
Assignee:
No assignee Edit question
Last query:
2013-02-04
Last reply:
2013-04-17

MY Quantum DHCP LOG shows me the following...

2013-01-24 07:01:01 INFO [quantum.agent.dhcp_agent] Synchronizing state
2013-01-24 07:01:01 DEBUG [quantum.openstack.common.rpc.amqp] Making asynchronous call on q-plugin ...
2013-01-24 07:01:01 DEBUG [quantum.openstack.common.rpc.amqp] MSG_ID is 8d2d328fda454c37a213d1117ddf985d
2013-01-24 07:01:01 DEBUG [amqplib] Closed channel #1
2013-01-24 07:01:01 DEBUG [amqplib] using channel_id: 1
2013-01-24 07:01:01 DEBUG [amqplib] Channel open
2013-01-24 07:01:01 DEBUG [quantum.openstack.common.rpc.amqp] Making asynchronous call on q-plugin ...
2013-01-24 07:01:01 DEBUG [quantum.openstack.common.rpc.amqp] MSG_ID is 3767610be6014be3a691fe59803ccbd9
2013-01-24 07:01:01 DEBUG [amqplib] Closed channel #1
2013-01-24 07:01:01 DEBUG [amqplib] using channel_id: 1
2013-01-24 07:01:01 DEBUG [amqplib] Channel open
2013-01-24 07:01:01 DEBUG [quantum.openstack.common.rpc.amqp] Making asynchronous call on q-plugin ...
2013-01-24 07:01:01 DEBUG [quantum.openstack.common.rpc.amqp] MSG_ID is bb0f8b24bb714b2bb4ce1665503a1860
2013-01-24 07:01:01 DEBUG [amqplib] Closed channel #1
2013-01-24 07:01:01 DEBUG [amqplib] using channel_id: 1
2013-01-24 07:01:01 DEBUG [amqplib] Channel open
2013-01-24 07:01:01 DEBUG [quantum.openstack.common.rpc.amqp] Making asynchronous call on q-plugin ...
2013-01-24 07:01:01 DEBUG [quantum.openstack.common.rpc.amqp] MSG_ID is 8f4b7c583fc64c90bc04dfdfed3ddabd
2013-01-24 07:01:01 DEBUG [amqplib] Closed channel #1
2013-01-24 07:01:01 DEBUG [amqplib] using channel_id: 1
2013-01-24 07:01:01 DEBUG [amqplib] Channel open
2013-01-24 07:01:01 DEBUG [quantum.agent.linux.utils] Running command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ip netns exec qdhcp-eec84bc1-756e-4c9d-a32a-b03d7a799f80 ip -o link show tap66ba434e-f6
2013-01-24 07:01:01 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-eec84bc1-756e-4c9d-a32a-b03d7a799f80', 'ip', '-o', 'link', 'show', 'tap66ba434e-f6']
Exit code: 0
Stdout: '17: tap66ba434e-f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT \\ link/ether fa:16:3e:55:13:3a brd ff:ff:ff:ff:ff:ff\n'
Stderr: ''
2013-01-24 07:01:01 DEBUG [quantum.agent.dhcp_agent] Reusing existing device: tap66ba434e-f6.
2013-01-24 07:01:01 DEBUG [quantum.agent.linux.utils] Running command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ip netns exec qdhcp-eec84bc1-756e-4c9d-a32a-b03d7a799f80 ip addr show tap66ba434e-f6 permanent scope glo$
2013-01-24 07:01:01 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-eec84bc1-756e-4c9d-a32a-b03d7a799f80', 'ip', 'addr', 'show', 'tap66ba434e-f6', 'permanent', 'scope', 'global']
Exit code: 0
Stdout: '17: tap66ba434e-f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \n link/ether fa:16:3e:55:13:3a brd ff:ff:ff:ff:ff:ff\n inet 50.50.1.3/24 brd 50.50.1.255 scope global tap66ba434e-f6\n'
Stderr: ''
2013-01-24 07:01:01 DEBUG [quantum.agent.linux.dhcp] Unable to access /var/lib/quantum/dhcp/eec84bc1-756e-4c9d-a32a-b03d7a799f80/pid
2013-01-24 07:01:01 DEBUG [quantum.agent.linux.utils] Running command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf cat /proc/None/cmdline
2013-01-24 07:01:01 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'cat', '/proc/None/cmdline']
Exit code: 99
Stdout: 'Unauthorized command: cat /proc/None/cmdline\n'
Stderr: ''
2013-01-24 07:01:01 DEBUG [quantum.agent.linux.utils] Running command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf QUANTUM_RELAY_SOCKET_PATH=/var/lib/quantum/dhcp/lease_relay QUANTUM_NETWORK_ID=eec84bc1-756e-4c9d-a32a-b$
2013-01-24 07:01:01 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'QUANTUM_RELAY_SOCKET_PATH=/var/lib/quantum/dhcp/lease_relay', 'QUANTUM_NETWORK_ID=eec84bc1-756e-4c9d-a32a-b03d7a799f80', 'ip', 'netns', 'exec', 'qdhcp-eec84bc$
Exit code: 1

Thanks Gary, Currently working on the metadata proxy, I will get back to you and let you know how it goes. As of right now I hope this stays open until I have solved it.

Spatialist (fsluiter) said : #4

I noticed in the log: Stdout: 'Unauthorized command: cat /proc/None/cmdline\n'

that indicates a permissions problem. What user is quantum running with?
We had a NFS share, and we ran into a problem that the quantum user had a different uid on the master then on the nodes. That does not work with an NFS.

I'm not sure what you mean by that but here is what I did from the cloud controller,

I created a new tenant, then added a user to that tenant with member role assigned to it.
Created a new network for that tenant.
Created a subnet in that tenant network.
Created a router for that tenant.
Added the router to the subnet.

Then I created a new network for external network with the tenant_ID of the service tenant (according to the guide I followed)
Then I added router_id and gateway_external_net_id to the /etc/quantum/l3_agent.ini file and restart l3-agent.
Then created a subnet within the service tenant network with the range of the floating IP's I want.
Then i set the router-gateway with router_ID and ID of ext_net.

I believe the commands that should make VM's reach the metadata server are,
quantum router-gateway-set $Router_ID $ext_net_ID
quantum port-list -- --device_id $Router_ID --device_owner network:router_gateway
route add -net x.x.x.x/24 gw $Router_IP

In config files, I have the keystone, quantum user configured as well, with proper connection to MySQL database.

Let me know if you meant something else specifically.

Spatialist (fsluiter) said : #6

There are several accounts for quantum itself:

- a sytem account, quantum on linux
- a database account quantum (mysql/postgresql)
- a keystone account quantum

For nova etc this is similar.

The linux account has a uid and a groupid, that you can find inside the linux file /etc/passwd
If you run openstack on a cluster, the linux accounts might have different ids on each separate system. This is sometimes a bad thing. Now to change it is quite hard, if you are not an experienced linux administrator you might want to ask for help from someone who is. The problem is that all files that are written by this user will have the wrong uid and groupid after you changed it and those need to be corrected as well. First check if the ids in the /etc/passwds match up.

Now another problem that you might have:
Another silly thing is that RabbitMQ has a default user guest with password guest. Some applications have this hardcoded as default (nova), but for some crazy reason, quantum has another default in the documentation.
Now when you follow the documentation, quantum cannot acces RabbitMQ.
This is easily changed, use the default rabitmq user guest/guest for quantum, or change it every where in something more secure.

Danny (surezk) said : #7

Hi Tatenda Mupaso,

Have you resolved this issue? because I ran into this issue also, and it has cost me 2 days..but no any solution. if you have and hint ,please tell me. thanks.

novelkumar (novel-kumar) said : #8

I am also facing the same issue.. Have you resolved it...

Can you help with this problem?

Provide an answer of your own, or ask Tatenda Mupaso for more information if necessary.

To post a message you must log in.