instances can't connect whit remote host (169.254.169.254) network is unreachable

Asked by Daniel Nunez E. on 2012-11-24

Hi everyone

I have 2 nodes
ControllerNode: 172.31.15.167. It has all nova services include nova-compute
ComputeNode: 172.31.15.165. It has installed only nova-api and nova-compute

I have images:

# nova image-list
+--------------------------------------+-----------------------------------------------------+--------+--------+
| ID | Name | Status | Server |
+--------------------------------------+-----------------------------------------------------+--------+--------+
| bfc16412-5fc1-4423-9868-92314f4066fd | Cirrus test | ACTIVE | |

I can run an instance
# nova boot cirros1 --image bfc16412-5fc1-4423-9868-92314f4066fd --flavor 1 --key_name key --security_group default
# nova list
+--------------------------------------+---------+--------+----------------------------------+
| ID | Name | Status | Networks |
+--------------------------------------+---------+--------+----------------------------------+
| 809b9b15-c84d-4a13-8fc5-69a459a85316 | cirros1 | ACTIVE | cookbook=10.0.3.2, 172.31.15.209 |
+--------------------------------------+---------+--------+----------------------------------+

The relevant output of nova console-log is

.......
.......
Starting network...
udhcpc (v1.18.5) started
Sending discover...
Sending discover...
Sending discover...
No lease, failing
WARN: /etc/rc3.d/S40-network failed
cloud-setup: checking http://169.254.169.254/2009-04-04/meta-data/instance-id
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-setup: failed 1/30: up 11.03. request failed
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-setup: failed 2/30: up 12.05. request failed
......
......
############ debug start ##############
### /etc/rc.d/init.d/sshd start
/etc/rc3.d/S45-cloud-setup: line 66: /etc/rc.d/init.d/sshd: not found
route: fscanf
### ifconfig -a
eth0 Link encap:Ethernet HWaddr FA:16:3E:29:EF:7A
          inet6 addr: fe80::f816:3eff:fe29:ef7a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:13 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1706 (1.6 KiB) TX bytes:1224 (1.1 KiB)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

### route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
route: fscanf
### cat /etc/resolv.conf
cat: can't open '/etc/resolv.conf': No such file or directory
### gateway not found
/etc/rc3.d/S45-cloud-setup: line 66: can't open /etc/resolv.conf: no such file
.......
.......
===== cloud-final: system completely up in 42.36 seconds ====
wget: can't connect to remote host (169.254.169.254): Network is unreachable
wget: can't connect to remote host (169.254.169.254): Network is unreachable
wget: can't connect to remote host (169.254.169.254): Network is unreachable
  instance-id:
  public-ipv4:
  local-ipv4 :
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-userdata: failed to read instance id
WARN: /etc/rc3.d/S99-cloud-userdata failed

I can't ping the instance

# ping 10.0.3.2
PING 10.0.3.2 (10.0.3.2) 56(84) bytes of data.
From 10.0.3.1 icmp_seq=1 Destination Host Unreachable
# ping 172.31.15.209
PING 172.31.15.209 (172.31.15.209) 56(84) bytes of data.
From 172.31.15.209 icmp_seq=1 Destination Host Unreachable

This is my nova.conf file present in the cloud controller and compute node

##Network Config
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--network_manager=nova.network.manager.FlatDHCPManager
--flat_network_dhcp_start=10.0.3.3
--flat_network_bridge=br100
--flat-interface=eth0
--flat_injected=False
--public_interface=eth1
--fixed_range= 10.0.3.0/24
--network_size=256
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--sql_connection=mysql://nova:openstack@localhost/nova
--force_dhcp_release
--iscsi_helper=tgtadm
--libvirt_use_virtio_for_bridges
--connection_type=libvirt
--root_helper=sudo nova-rootwrap
--verbose
--ec2_private_dns_show_ip
--s3_host=172.31.15.167
--rabbit_host=172.31.15.167
--ec2_host=172.31.15.167
--ec2_dmz_host=172.31.15.167
--image_service=nova.image.glance.GlanceImageService
--glance_api_servers=172.31.15.167:9292
--auto_assign_floating_ip=true
--scheduler_default_filters=AllHostsFilter
--metadata_port=8775
--api-paste_config=/etc/nova/api-paste.ini
--keystone_ec2_url=http://172.31.15.167:5000/v2.0/ec2tokens
--auth_strategy=keystone

The firewall nat rules are

# iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N nova-api-OUTPUT
-N nova-api-POSTROUTING
-N nova-api-PREROUTING
-N nova-api-float-snat
-N nova-api-snat
-N nova-compute-OUTPUT
-N nova-compute-POSTROUTING
-N nova-compute-PREROUTING
-N nova-compute-float-snat
-N nova-compute-snat
-N nova-network-OUTPUT
-N nova-network-POSTROUTING
-N nova-network-PREROUTING
-N nova-network-float-snat
-N nova-network-snat
-N nova-postrouting-bottom
-A PREROUTING -j nova-compute-PREROUTING
-A PREROUTING -j nova-network-PREROUTING
-A PREROUTING -j nova-api-PREROUTING
-A PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.31.15.167:8773
-A OUTPUT -j nova-compute-OUTPUT
-A OUTPUT -j nova-network-OUTPUT
-A OUTPUT -j nova-api-OUTPUT
-A POSTROUTING -j nova-compute-POSTROUTING
-A POSTROUTING -j nova-network-POSTROUTING
-A POSTROUTING -j nova-api-POSTROUTING
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
-A POSTROUTING -j nova-postrouting-bottom
-A nova-api-snat -j nova-api-float-snat
-A nova-compute-snat -j nova-compute-float-snat
-A nova-network-OUTPUT -d 172.31.15.211/32 -j DNAT --to-destination 10.0.0.5
-A nova-network-OUTPUT -d 172.31.15.209/32 -j DNAT --to-destination 10.0.3.2
-A nova-network-POSTROUTING -s 10.0.3.0/24 -d 172.31.15.167/32 -j ACCEPT
-A nova-network-POSTROUTING -s 10.0.3.0/24 -d 10.128.0.0/24 -j ACCEPT
-A nova-network-POSTROUTING -s 10.0.3.0/24 -d 10.0.3.0/24 -m conntrack ! --ctstate DNAT -j ACCEPT
-A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.31.15.167:8775
-A nova-network-PREROUTING -d 172.31.15.211/32 -j DNAT --to-destination 10.0.0.5
-A nova-network-PREROUTING -d 172.31.15.209/32 -j DNAT --to-destination 10.0.3.2
-A nova-network-float-snat -s 10.0.0.5/32 -j SNAT --to-source 172.31.15.211
-A nova-network-float-snat -s 10.0.3.2/32 -j SNAT --to-source 172.31.15.209
-A nova-network-snat -j nova-network-float-snat
-A nova-network-snat -s 10.0.3.0/24 -j SNAT --to-source 172.31.15.167
-A nova-postrouting-bottom -j nova-compute-snat
-A nova-postrouting-bottom -j nova-network-snat
-A nova-postrouting-bottom -j nova-api-snat

The network config in CloudController is:

# cat /etc/network/interfaces
auto lo
iface lo inet loopback
auto eth1
iface eth1 inet static
 address 172.31.15.167
 netmask 255.255.255.0
 network 172.31.15.0
 broadcast 172.31.15.255
 gateway 172.31.15.251
 # dns-* options are implemented by the resolvconf package, if installed
 dns-nameservers 200.31.6.34
auto br100
iface br100 inet static
 address 10.0.3.1
 netmask 255.255.255.0
 bridge_stp off
 bridge_fd 0

The network config in computeNode is

# cat /etc/network/interfaces
auto lo
iface lo inet loopback
auto eth1
iface eth1 inet static
 address 172.31.15.165
 netmask 255.255.255.0
 network 172.31.15.0
 broadcast 172.31.15.255
 gateway 172.31.15.251
 # dns-* options are implemented by the resolvconf package, if installed
 dns-nameservers 172.31.15.106

In the nova log files there aren't error messages. and I don't know why the instances doesn't have IP.

Please Help me

Question information

Language:
English Edit question
Status:
Answered
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Last query:
2012-11-24
Last reply:
2012-11-26

Check if you have an iptables rule corresponding to 169.254.169.254
http://www.mirantis.com/blog/openstack-networking-single-host-flatdhcpmanager/

Snippet from above url - " but currently the only important rule is this one: -A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.56.200:8775. It makes the nova metadata service “listen” on the link-local address 169.254.169.254 by doing DNAT from that address to its actual bind address on the controller, 192.168.56.200:8775."

Daniel Nunez E. (danielnuneze) said : #2

Yes, in the first post is that line. Nova understand the instance have IPs.

# nova list
+--------------------------------------+---------+--------+----------------------------------+
| ID | Name | Status | Networks |
+--------------------------------------+---------+--------+----------------------------------+
| 809b9b15-c84d-4a13-8fc5-69a459a85316 | cirros1 | ACTIVE | cookbook=10.0.3.2, 172.31.15.209 |
+--------------------------------------+---------+--------+----------------------------------+

but the instances can't get metadata.

===== cloud-final: system completely up in 42.36 seconds ====
wget: can't connect to remote host (169.254.169.254): Network is unreachable
wget: can't connect to remote host (169.254.169.254): Network is unreachable
wget: can't connect to remote host (169.254.169.254): Network is unreachable
  instance-id:
  public-ipv4:
  local-ipv4 :
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-userdata: failed to read instance id
WARN: /etc/rc3.d/S99-cloud-userdata failed

nova.conf file needs a config line for activate some metadata service???

Can you help with this problem?

Provide an answer of your own, or ask Daniel Nunez E. for more information if necessary.

To post a message you must log in.