Instance can not get metadata..........

Asked by Hugo Kou on 2011-04-08

There's my topology, It's a multi nodes archi that separate nova-network into a single box
http://1.bp.blogspot.com/-KsV9TEoTjk4/TZ7cAlQMFEI/AAAAAAAACko/Jrjb3cczPcI/s1600/current.png

============euca-get-console-out-put==================
bringing up loopback interface lo [ OK ]
stty: \/dev\/console
udhcpc (v1.17.2) started
Sending discover...
Sending discover...
Sending discover...
No lease, forking to background
starting DHCP forEthernet interface eth0 [ OK ]
cloud-setup: checking http:\/\/169.254.169.254\/2009-04-04\/meta-data\/instance-id
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-setup: failed 1\/30: up 10.46. request failed
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-setup: failed 2\/30: up 11.47. request failed
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-setup: failed 3\/30: up 12.49. request failed
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-setup: failed 4\/30: up 13.50. request failed
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-setup: failed 5\/30: up 14.52. request failed
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-setup: failed 6\/30: up 15.53. request failed
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-setup: failed 7\/30: up 16.55. request failed
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-setup: failed 8\/30: up 17.56. request failed
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-setup: failed 9\/30: up 18.58. request failed

==========iptables-save==on nova-network host==========
*nat
:PREROUTING ACCEPT [2:636]
:OUTPUT ACCEPT [8:504]
:POSTROUTING ACCEPT [8:504]
:nova-network-OUTPUT - [0:0]
:nova-network-POSTROUTING - [0:0]
:nova-network-PREROUTING - [0:0]
:nova-network-floating-snat - [0:0]
:nova-network-snat - [0:0]
:nova-postrouting-bottom - [0:0]
-A PREROUTING -j nova-network-PREROUTING
-A OUTPUT -j nova-network-OUTPUT
-A POSTROUTING -j nova-network-POSTROUTING
-A POSTROUTING -j nova-postrouting-bottom
-A nova-network-POSTROUTING -s 192.168.2.0/24 -d 10.128.0.0/24 -j ACCEPT
-A nova-network-POSTROUTING -s 192.168.2.0/24 -d 192.168.2.0/24 -j ACCEPT
-A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.1.1:8773
-A nova-network-snat -j nova-network-floating-snat
-A nova-network-snat -s 192.168.2.0/24 -j SNAT --to-source 192.168.1.1
-A nova-postrouting-bottom -j nova-network-snat
COMMIT
# Completed on Fri Apr 8 19:02:27 2011
# Generated by iptables-save v1.4.4 on Fri Apr 8 19:02:27 2011
*filter
:INPUT ACCEPT [18191:1530656]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [35418:2835137]
:nova-filter-top - [0:0]
:nova-network-FORWARD - [0:0]
:nova-network-INPUT - [0:0]
:nova-network-OUTPUT - [0:0]
:nova-network-local - [0:0]
-A INPUT -j nova-network-INPUT
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-network-FORWARD
-A OUTPUT -j nova-filter-top
-A OUTPUT -j nova-network-OUTPUT
-A nova-filter-top -j nova-network-local
COMMIT
# Completed on Fri Apr 8 19:02:27 2011
=====================================

I tried everything , but still failed to get metadata

Nova-network create a bridge "br100" for flat network and add eth0 to br100.

===========ip addr show====on nova-network host==========
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 169.254.169.254/32 scope link lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0e:7b:d0:05:ef brd ff:ff:ff:ff:ff:ff
    inet6 fe80::20e:7bff:fed0:5ef/64 scope link
       valid_lft forever preferred_lft forever
3: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 00:19:d2:22:70:51 brd ff:ff:ff:ff:ff:ff
4: br100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 00:0e:7b:d0:05:ef brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.1/24 brd 192.168.2.255 scope global br100
    inet 192.168.1.2/24 brd 192.168.1.255 scope global br100
    inet6 fe80::7e:7aff:fe1f:1974/64 scope link
       valid_lft forever preferred_lft forever

=====================================

Could someone give me some clues about this issue? I face this problem several times.

I really want to know what's the problem over there.

In my knowing , instance try to get metadata while a instance run up through 169.254.169.254:80 .

and then send discovery from on br100 , at this moment , nova-netowork will assign a ip to instance. then try to get metadata. due to instance's gateway is nova-network's br100 , So that nova-network should route 169.254.169.254 request to nova-api server . That's the reason of iptable prerouting to $nova-api_ip:8773.

ok, I check everything . but instance still can not been ssh or ping . After check console-out-put, I saw the result as above.

Thanks you guys first. Plz correct my mistake.

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
Vish Ishaya
Solved:
2011-04-14
Last query:
2011-04-14
Last reply:
2011-04-08

This question was reopened

Best Vish Ishaya (vishvananda) said : #1

It actually isn't a metadata problem.. Dhcp is failing (Note the sending discover and No lease) it looks like it is trying to get an ip in this range:
> inet 192.168.2.1/24 brd 192.168.2.255 scope global br100

A few possibilities:

dnsmasq is not running properly (ps ax | grep dnsmasq should show exactly two dnsmasq processes with the exact same long command line)
you missed a flag like --dhcp_listen_address and so it is attempting to listen on 10.0.0.2 (check ps ax | grep dnsmasq) and see if it shows the wrong number --dhcp-range=10.0.0.2,static,120s) Fix this by setting the flag properly and also updating the networks table in the db.
likely you will see DHCP messages in /var/log/messages or /var/log/syslog with errors

Vish

On Apr 8, 2011, at 4:25 AM, Hugo Kou wrote:

> New question #152089 on OpenStack Compute (nova):
> https://answers.launchpad.net/nova/+question/152089
>
> There's my topology, It's a multi nodes archi that separate nova-network into a single box
> http://1.bp.blogspot.com/-KsV9TEoTjk4/TZ7cAlQMFEI/AAAAAAAACko/Jrjb3cczPcI/s1600/current.png
>
>
> ============euca-get-console-out-put==================
> bringing up loopback interface lo [ OK ]
> stty: \/dev\/console
> udhcpc (v1.17.2) started
> Sending discover...
> Sending discover...
> Sending discover...
> No lease, forking to background
> starting DHCP forEthernet interface eth0 [ OK ]
> cloud-setup: checking http:\/\/169.254.169.254\/2009-04-04\/meta-data\/instance-id
> wget: can't connect to remote host (169.254.169.254): Network is unreachable
> cloud-setup: failed 1\/30: up 10.46. request failed
> wget: can't connect to remote host (169.254.169.254): Network is unreachable
> cloud-setup: failed 2\/30: up 11.47. request failed
> wget: can't connect to remote host (169.254.169.254): Network is unreachable
> cloud-setup: failed 3\/30: up 12.49. request failed
> wget: can't connect to remote host (169.254.169.254): Network is unreachable
> cloud-setup: failed 4\/30: up 13.50. request failed
> wget: can't connect to remote host (169.254.169.254): Network is unreachable
> cloud-setup: failed 5\/30: up 14.52. request failed
> wget: can't connect to remote host (169.254.169.254): Network is unreachable
> cloud-setup: failed 6\/30: up 15.53. request failed
> wget: can't connect to remote host (169.254.169.254): Network is unreachable
> cloud-setup: failed 7\/30: up 16.55. request failed
> wget: can't connect to remote host (169.254.169.254): Network is unreachable
> cloud-setup: failed 8\/30: up 17.56. request failed
> wget: can't connect to remote host (169.254.169.254): Network is unreachable
> cloud-setup: failed 9\/30: up 18.58. request failed
>
> ==========iptables-save==on nova-network host==========
> *nat
> :PREROUTING ACCEPT [2:636]
> :OUTPUT ACCEPT [8:504]
> :POSTROUTING ACCEPT [8:504]
> :nova-network-OUTPUT - [0:0]
> :nova-network-POSTROUTING - [0:0]
> :nova-network-PREROUTING - [0:0]
> :nova-network-floating-snat - [0:0]
> :nova-network-snat - [0:0]
> :nova-postrouting-bottom - [0:0]
> -A PREROUTING -j nova-network-PREROUTING
> -A OUTPUT -j nova-network-OUTPUT
> -A POSTROUTING -j nova-network-POSTROUTING
> -A POSTROUTING -j nova-postrouting-bottom
> -A nova-network-POSTROUTING -s 192.168.2.0/24 -d 10.128.0.0/24 -j ACCEPT
> -A nova-network-POSTROUTING -s 192.168.2.0/24 -d 192.168.2.0/24 -j ACCEPT
> -A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.1.1:8773
> -A nova-network-snat -j nova-network-floating-snat
> -A nova-network-snat -s 192.168.2.0/24 -j SNAT --to-source 192.168.1.1
> -A nova-postrouting-bottom -j nova-network-snat
> COMMIT
> # Completed on Fri Apr 8 19:02:27 2011
> # Generated by iptables-save v1.4.4 on Fri Apr 8 19:02:27 2011
> *filter
> :INPUT ACCEPT [18191:1530656]
> :FORWARD ACCEPT [0:0]
> :OUTPUT ACCEPT [35418:2835137]
> :nova-filter-top - [0:0]
> :nova-network-FORWARD - [0:0]
> :nova-network-INPUT - [0:0]
> :nova-network-OUTPUT - [0:0]
> :nova-network-local - [0:0]
> -A INPUT -j nova-network-INPUT
> -A FORWARD -j nova-filter-top
> -A FORWARD -j nova-network-FORWARD
> -A OUTPUT -j nova-filter-top
> -A OUTPUT -j nova-network-OUTPUT
> -A nova-filter-top -j nova-network-local
> COMMIT
> # Completed on Fri Apr 8 19:02:27 2011
> =====================================
>
> I tried everything , but still failed to get metadata
>
> Nova-network create a bridge "br100" for flat network and add eth0 to br100.
>
> ===========ip addr show====on nova-network host==========
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
> inet 169.254.169.254/32 scope link lo
> inet6 ::1/128 scope host
> valid_lft forever preferred_lft forever
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
> link/ether 00:0e:7b:d0:05:ef brd ff:ff:ff:ff:ff:ff
> inet6 fe80::20e:7bff:fed0:5ef/64 scope link
> valid_lft forever preferred_lft forever
> 3: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
> link/ether 00:19:d2:22:70:51 brd ff:ff:ff:ff:ff:ff
> 4: br100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
> link/ether 00:0e:7b:d0:05:ef brd ff:ff:ff:ff:ff:ff
> inet 192.168.2.1/24 brd 192.168.2.255 scope global br100
> inet 192.168.1.2/24 brd 192.168.1.255 scope global br100
> inet6 fe80::7e:7aff:fe1f:1974/64 scope link
> valid_lft forever preferred_lft forever
>
> =====================================
>
> Could someone give me some clues about this issue? I face this problem several times.
>
> I really want to know what's the problem over there.
>
> In my knowing , instance try to get metadata while a instance run up through 169.254.169.254:80 .
>
> and then send discovery from on br100 , at this moment , nova-netowork will assign a ip to instance. then try to get metadata. due to instance's gateway is nova-network's br100 , So that nova-network should route 169.254.169.254 request to nova-api server . That's the reason of iptable prerouting to $nova-api_ip:8773.
>
> ok, I check everything . but instance still can not been ssh or ping . After check console-out-put, I saw the result as above.
>
> Thanks you guys first. Plz correct my mistake.
>
>
>
>
>
>
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

Hugo Kou (tonytkdk) said : #2

Thanks Vish~

While I check syslog as follow

===============/var/log/syslog==============
Apr 11 11:17:01 ubuntu2 CRON[14283]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Apr 11 11:17:16 ubuntu2 dnsmasq-dhcp[12622]: no address range available for DHCP request via br100
Apr 11 11:18:17 ubuntu2 dnsmasq-dhcp[12622]: last message repeated 7 times
Apr 11 11:18:40 ubuntu2 dnsmasq-dhcp[12622]: last message repeated 2 times
Apr 11 11:18:40 ubuntu2 dnsmasq[12622]: read /etc/hosts - 7 addresses
Apr 11 11:18:40 ubuntu2 dnsmasq-dhcp[12622]: read /var/lib/nova/networks/nova-br100.conf
Apr 11 11:18:42 ubuntu2 dnsmasq-dhcp[12622]: no address range available for DHCP request via br100
Apr 11 11:19:44 ubuntu2 dnsmasq-dhcp[12622]: last message repeated 20 times
Apr 11 11:20:45 ubuntu2 dnsmasq-dhcp[12622]: last message repeated 13 times
Apr 11 11:21:47 ubuntu2 dnsmasq-dhcp[12622]: last message repeated 14 times
=====================================

Yes, you are right!

Try to fix it now, I'll report it later.

HugoKuo

Hugo Kou (tonytkdk) said : #3

root@ubuntu2:/var/log# ps aux | grep dnsmasq

==============ps aux | grep dnsmasq=================
nobody 12622 0.0 0.0 4852 920 ? S Apr08 0:07 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.2.1 --except-interface=lo --dhcp-range=10.0.0.2,static,120s --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro
root 12623 0.0 0.0 4824 168 ? S Apr08 0:00 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.2.1 --except-interface=lo --dhcp-range=10.0.0.2,static,120s --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro

Hugo Kou (tonytkdk) said : #4

Thanks Vish Ishaya, that solved my question.

Hugo Kou (tonytkdk) said : #5

After add a flag
--flat_network_dhcp_start=192.168.2.2

====ps aux | grep dnsmasq=====
nobody 7901 0.0 0.0 4852 936 ? S 13:29 0:00 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.2.1 --except-interface=lo --dhcp-range=192.168.2.2,static,120s --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro
root 7902 0.0 0.0 4824 392 ? S 13:29 0:00 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.2.1 --except-interface=lo --dhcp-range=192.168.2.2,static,120s --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro
======================

OK for "e sending discover and No lease" .............

and I fall into another problem , I think it's metadata problem now.....
I close this one and create another one

[Success SSH a tty image but failed SSH UEC image]
https://answers.launchpad.net/nova/+question/152383

Hugo Kou (tonytkdk) said : #6

Thanks Vish Ishaya, that solved my question.

Salman (salmanmk) said : #7

Hi Guys,

Can you take a look at this:
https://answers.launchpad.net/nova/+question/195439

I am having a similar problem (that I haven't been able to solve) and I am using quantum plugin for network configuration.

Thanks,
Salman