Cannot connect to a running instance, even after the euca-authorize stuff

Asked by Brian Amedro on 2011-04-22

Hi !

I experience some difficulties do connect to the instances I've created with nova.
I can neither ping nor ssh the instance, even if I perform the euca-authorize stuff.

I use FlatDHCPManager.

Any help would be greatly appreciated !

Here are some details about my config and the problem :

==== /etc/nova/nova.conf ====
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--verbose
--libvirt_type=kvm
--network_manager=nova.network.manager.FlatDHCPManager
--flat_network_dhcp_start=138.96.126.200
--public_interface=eth0
--flat_injected=False

==== /etc/network/interfaces ====
# The loopback network interface
auto lo
iface lo inet loopback

# Networking for OpenStack Compute
auto br100
iface br100 inet dhcp
bridge_ports eth0
bridge_stp off
bridge_maxwait 0
bridge_fd 0

==== /sbin/ifconfig ====
br100 Link encap:Ethernet HWaddr 00:22:19:9a:6c:20
          inet addr:138.96.126.4 Bcast:138.96.126.255 Mask:255.255.255.0
          inet6 addr: fe80::222:19ff:fe9a:6c20/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:73420 errors:0 dropped:0 overruns:0 frame:0
          TX packets:23403 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:57123865 (57.1 MB) TX bytes:3052602 (3.0 MB)

eth0 Link encap:Ethernet HWaddr 00:22:19:9a:6c:20
          inet6 addr: fe80::222:19ff:fe9a:6c20/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:115824 errors:0 dropped:0 overruns:0 frame:0
          TX packets:24616 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:62186577 (62.1 MB) TX bytes:3385628 (3.3 MB)
          Interrupt:21 Memory:ea000000-ea012800

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:11291121 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11291121 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:783143897 (783.1 MB) TX bytes:783143897 (783.1 MB)

virbr0 Link encap:Ethernet HWaddr 26:dd:54:d5:85:df
          inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

vnet0 Link encap:Ethernet HWaddr fe:16:3e:1b:62:26
          inet6 addr: fe80::fc16:3eff:fe1b:6226/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:14 errors:0 dropped:0 overruns:0 frame:0
          TX packets:822 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:2948 (2.9 KB) TX bytes:110929 (110.9 KB)

==== # brctl show
bridge name bridge id STP enabled interfaces
br100 8000.0022199a6c20 no eth0
       vnet0
virbr0 8000.000000000000 yes

==== /var/log/libvirt/qemu/instance-00000019.log ====
2011-04-22 12:25:16.405: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name instance-00000019 -uuid bbe7e8a5-39b8-5b1d-e853-e6fe25ce934c -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000019.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot c -kernel /var/lib/nova/instances/instance-00000019/kernel -append root=/dev/vda console=ttyS0 -drive file=/var/lib/nova/instances/instance-00000019/disk,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,fd=17,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=02:16:3e:1b:62:26,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/instance-00000019/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -vnc 0.0.0.0:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/1

==== euca-get-console-output ====
<cut>
2011-04-22 10:26:20,267 - DataSourceEc2.py[WARNING]: waiting for metadata service at http:\/\/169.254.169.254\/2009-04-04\/meta-data\/instance-id
2011-04-22 10:26:20,269 - DataSourceEc2.py[WARNING]: 10:26:20 [ 1\/100]: url error [[Errno 101] Network is unreachable]
</cut>

==== curl http://169.254.169.254:8773/ ====
1.0
2007-01-19
2007-03-01
2007-08-29
2007-10-10
2007-12-15
2008-02-01
2008-09-01
2009-04-04

==== $ nova list ====
+----+-----------+--------+-----------+----------------+
| ID | Name | Status | Public IP | Private IP |
+----+-----------+--------+-----------+----------------+
| 25 | Server 25 | ACTIVE | | 138.96.126.201 |
+----+-----------+--------+-----------+----------------+

==== $ nova diagnostics 25 ====
n/a (HTTP 404)

==== # iptables -n -L ====
Chain INPUT (policy ACCEPT)
target prot opt source destination
nova-compute-INPUT all -- 0.0.0.0/0 0.0.0.0/0
nova-network-INPUT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:53
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:67
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:67

Chain FORWARD (policy ACCEPT)
target prot opt source destination
nova-filter-top all -- 0.0.0.0/0 0.0.0.0/0
nova-compute-FORWARD all -- 0.0.0.0/0 0.0.0.0/0
nova-network-FORWARD all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 192.168.122.0/24 state RELATED,ESTABLISHED
ACCEPT all -- 192.168.122.0/24 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
nova-filter-top all -- 0.0.0.0/0 0.0.0.0/0
nova-compute-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
nova-network-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0

Chain nova-compute-FORWARD (1 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0

Chain nova-compute-INPUT (1 references)
target prot opt source destination

Chain nova-compute-OUTPUT (1 references)
target prot opt source destination

Chain nova-compute-inst-25 (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
ACCEPT udp -- 138.96.126.1 0.0.0.0/0 udp spt:67 dpt:68
ACCEPT all -- 138.96.126.0/24 0.0.0.0/0
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
nova-compute-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0

Chain nova-compute-local (1 references)
target prot opt source destination
nova-compute-inst-25 all -- 0.0.0.0/0 138.96.126.201

Chain nova-compute-sg-fallback (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0

Chain nova-filter-top (2 references)
target prot opt source destination
nova-compute-local all -- 0.0.0.0/0 0.0.0.0/0
nova-network-local all -- 0.0.0.0/0 0.0.0.0/0

Chain nova-network-FORWARD (1 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0

Chain nova-network-INPUT (1 references)
target prot opt source destination

Chain nova-network-OUTPUT (1 references)
target prot opt source destination

Chain nova-network-local (1 references)
target prot opt source destination

==== # iptables-save ====
# Generated by iptables-save v1.4.10 on Fri Apr 22 12:53:16 2011
*mangle
:PREROUTING ACCEPT [11541338:853725355]
:INPUT ACCEPT [11537840:853174453]
:FORWARD ACCEPT [1640:305695]
:OUTPUT ACCEPT [11513629:800152874]
:POSTROUTING ACCEPT [11515269:800458569]
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Fri Apr 22 12:53:16 2011
# Generated by iptables-save v1.4.10 on Fri Apr 22 12:53:16 2011
*nat
:PREROUTING ACCEPT [301:54406]
:INPUT ACCEPT [2:96]
:OUTPUT ACCEPT [42:2919]
:POSTROUTING ACCEPT [135:15997]
:nova-compute-OUTPUT - [0:0]
:nova-compute-POSTROUTING - [0:0]
:nova-compute-PREROUTING - [0:0]
:nova-compute-floating-snat - [0:0]
:nova-compute-snat - [0:0]
:nova-network-OUTPUT - [0:0]
:nova-network-POSTROUTING - [0:0]
:nova-network-PREROUTING - [0:0]
:nova-network-floating-snat - [0:0]
:nova-network-snat - [0:0]
:nova-postrouting-bottom - [0:0]
-A PREROUTING -j nova-compute-PREROUTING
-A PREROUTING -j nova-network-PREROUTING
-A OUTPUT -j nova-compute-OUTPUT
-A OUTPUT -j nova-network-OUTPUT
-A POSTROUTING -j nova-compute-POSTROUTING
-A POSTROUTING -j nova-network-POSTROUTING
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
-A POSTROUTING -j nova-postrouting-bottom
-A nova-compute-snat -j nova-compute-floating-snat
-A nova-network-POSTROUTING -s 10.0.0.0/8 -d 10.128.0.0/24 -j ACCEPT
-A nova-network-POSTROUTING -s 10.0.0.0/8 -d 10.0.0.0/8 -j ACCEPT
-A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 138.96.126.4:8773
-A nova-network-snat -j nova-network-floating-snat
-A nova-network-snat -s 10.0.0.0/8 -j SNAT --to-source 138.96.126.4
-A nova-postrouting-bottom -j nova-compute-snat
-A nova-postrouting-bottom -j nova-network-snat
COMMIT
# Completed on Fri Apr 22 12:53:16 2011
# Generated by iptables-save v1.4.10 on Fri Apr 22 12:53:16 2011
*filter
:INPUT ACCEPT [449576:31640365]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [449045:31666556]
:nova-compute-FORWARD - [0:0]
:nova-compute-INPUT - [0:0]
:nova-compute-OUTPUT - [0:0]
:nova-compute-inst-25 - [0:0]
:nova-compute-local - [0:0]
:nova-compute-sg-fallback - [0:0]
:nova-filter-top - [0:0]
:nova-network-FORWARD - [0:0]
:nova-network-INPUT - [0:0]
:nova-network-OUTPUT - [0:0]
:nova-network-local - [0:0]
-A INPUT -j nova-compute-INPUT
-A INPUT -j nova-network-INPUT
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-compute-FORWARD
-A FORWARD -j nova-network-FORWARD
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -j nova-filter-top
-A OUTPUT -j nova-compute-OUTPUT
-A OUTPUT -j nova-network-OUTPUT
-A nova-compute-FORWARD -i br100 -j ACCEPT
-A nova-compute-FORWARD -o br100 -j ACCEPT
-A nova-compute-inst-25 -m state --state INVALID -j DROP
-A nova-compute-inst-25 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A nova-compute-inst-25 -s 138.96.126.1/32 -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A nova-compute-inst-25 -s 138.96.126.0/24 -j ACCEPT
-A nova-compute-inst-25 -p icmp -j ACCEPT
-A nova-compute-inst-25 -p tcp -m tcp --dport 22 -j ACCEPT
-A nova-compute-inst-25 -j nova-compute-sg-fallback
-A nova-compute-local -d 138.96.126.201/32 -j nova-compute-inst-25
-A nova-compute-sg-fallback -j DROP
-A nova-filter-top -j nova-compute-local
-A nova-filter-top -j nova-network-local
-A nova-network-FORWARD -i br100 -j ACCEPT
-A nova-network-FORWARD -o br100 -j ACCEPT
COMMIT
# Completed on Fri Apr 22 12:53:16 2011

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
Vish Ishaya
Solved:
2011-04-23
Last query:
2011-04-23
Last reply:
2011-04-22
Brian Amedro (brian-amedro) said : #1

I already saw https://answers.launchpad.net/nova/+question/145820 which looks like to be a similar problem, but I am already with FlatDHCPManager.
I also tried to connect to the VM through a serial console, but it fails:

==== $ virsh console instance-00000019 ====
Connected to domain instance-00000019
Escape character is ^]
error: internal error character device (null) is not using a PTY

==== $ cat /var/lib/nova/instances/instance-00000019/libvirt.xml ====
<domain type='kvm'>
    <name>instance-00000019</name>
    <memory>524288</memory>
    <os>
            <type>hvm</type>
            <kernel>/var/lib/nova/instances/instance-00000019/kernel</kernel>
                <cmdline>root=/dev/vda console=ttyS0</cmdline>
    </os>
    <features>
        <acpi/>
    </features>
    <vcpu>1</vcpu>
    <devices>
        <disk type='file'>
            <driver type='qcow2'/>
            <source file='/var/lib/nova/instances/instance-00000019/disk'/>
            <target dev='vda' bus='virtio'/>
        </disk>

        <interface type='bridge'>
            <source bridge='br100'/>
            <mac address='02:16:3e:1b:62:26'/>
            <!-- <model type='virtio'/> CANT RUN virtio network right now -->
            <filterref filter="nova-instance-instance-00000019-02163e1b6226">
                <parameter name="IP" value="138.96.126.201" />
                <parameter name="DHCPSERVER" value="138.96.126.1" />
            </filterref>
        </interface>
        <!-- The order is significant here. File must be defined first -->
        <serial type="file">
            <source path='/var/lib/nova/instances/instance-00000019/console.log'/>
            <target port='1'/>
        </serial>

        <console type='pty' tty='/dev/pts/2'>
            <source path='/dev/pts/2'/>
            <target port='0'/>
        </console>

        <serial type='pty'>
            <source path='/dev/pts/2'/>
            <target port='0'/>
        </serial>

        <graphics type='vnc' port='-1' autoport='yes' keymap='en-us' listen='0.0.0.0'/>
    </devices>
</domain>

Vish Ishaya (vishvananda) said : #2

i would guess problems with dnsmasq. It looks like you are trying to put instances on the same network as your host machine which may be difficult to get working if you have an external router or gateway responding to dnsmasq queries. I would check the process list for dnsmasq and make sure that there are exactly two running and that the command line for them is the same and it looks reasonable.

Vish

On Apr 22, 2011, at 3:58 AM, Brian Amedro wrote:

> Question #153827 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/153827
>
> Brian Amedro gave more information on the question:
> I already saw https://answers.launchpad.net/nova/+question/145820 which looks like to be a similar problem, but I am already with FlatDHCPManager.
> I also tried to connect to the VM through a serial console, but it fails:
>
> ==== $ virsh console instance-00000019 ====
> Connected to domain instance-00000019
> Escape character is ^]
> error: internal error character device (null) is not using a PTY
>
>
> ==== $ cat /var/lib/nova/instances/instance-00000019/libvirt.xml ====
> <domain type='kvm'>
> <name>instance-00000019</name>
> <memory>524288</memory>
> <os>
> <type>hvm</type>
> <kernel>/var/lib/nova/instances/instance-00000019/kernel</kernel>
> <cmdline>root=/dev/vda console=ttyS0</cmdline>
> </os>
> <features>
> <acpi/>
> </features>
> <vcpu>1</vcpu>
> <devices>
> <disk type='file'>
> <driver type='qcow2'/>
> <source file='/var/lib/nova/instances/instance-00000019/disk'/>
> <target dev='vda' bus='virtio'/>
> </disk>
>
> <interface type='bridge'>
> <source bridge='br100'/>
> <mac address='02:16:3e:1b:62:26'/>
> <!-- <model type='virtio'/> CANT RUN virtio network right now -->
> <filterref filter="nova-instance-instance-00000019-02163e1b6226">
> <parameter name="IP" value="138.96.126.201" />
> <parameter name="DHCPSERVER" value="138.96.126.1" />
> </filterref>
> </interface>
> <!-- The order is significant here. File must be defined first -->
> <serial type="file">
> <source path='/var/lib/nova/instances/instance-00000019/console.log'/>
> <target port='1'/>
> </serial>
>
> <console type='pty' tty='/dev/pts/2'>
> <source path='/dev/pts/2'/>
> <target port='0'/>
> </console>
>
> <serial type='pty'>
> <source path='/dev/pts/2'/>
> <target port='0'/>
> </serial>
>
> <graphics type='vnc' port='-1' autoport='yes' keymap='en-us' listen='0.0.0.0'/>
> </devices>
> </domain>
>
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

Brian Amedro (brian-amedro) said : #3

Hi vish, thanks for the reply !

You're right, I am trying to put instances on the same network as the host machine. This network already has a DHCP server on 138.96.126.1, which give IP for the range 138.96.126.1-199. Also, I cannot modify the config of this server.

Is there any way so the host machine respond to the VM requests instead of the external DHCP server ?
Looking at /var/lib/nova/instances/instance-00000019/libvirt.xml, it does not seems to be the case: <parameter name="DHCPSERVER" value="138.96.126.1" />

Also, and to be complete, even if I put --flat_network_dhcp_start=138.96.126.200 in my nova.config file, started instances was assigned with an IP in the wrong range. Thus, I modified by hand the database, in order to exclude some range by setting the 'reserved' column in the fixed_ips table.

As you asked, here is the list of dnsmasq processes.

nobody 1029 0.0 0.0 21688 1072 ? S 18:16 0:00 dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/libvirt/network/default.pid --conf-file= --except-interface lo --listen-address 192.168.122.1 --dhcp-range 192.168.122.2,192.168.122.254 --dhcp-lease-max=253 --dhcp-no-override

nobody 1417 0.0 0.0 24388 1112 ? S 18:16 0:00 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=138.96.126.1 --except-interface=lo --dhcp-range=138.96.126.200,static,120s --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro

root 1418 0.0 0.0 24256 416 ? S 18:16 0:00 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=138.96.126.1 --except-interface=lo --dhcp-range=138.96.126.200,static,120s --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro

$ cat /var/lib/nova/networks/nova-br100.conf
02:16:3e:12:e4:ad,i-0000001e.novalocal,138.96.126.200

Vish Ishaya (vishvananda) said : #4

Yes that is your problem. It attempts to give the first address in the subnet to your machine and it is conflicting with your external dns server. You could try using a smaller network, such as:
128.96.126.224/27 although your .1 server may still give out addresses and mess things up. Worth a shot.
You'll have to delete networks and fixed_ip tables and recreate the smaller range with nova-network network create
you should also change:
--fixed_range=128.96.224/27
--flat_network_dhcp_start=138.96.126.225

killall dnsmasq
iptables -F
iptabbles -t nat -F
restart nova-network

try again?

Vish

On Apr 22, 2011, at 7:52 AM, Brian Amedro wrote:

> Question #153827 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/153827
>
> Status: Answered => Open
>
> Brian Amedro is still having a problem:
> Hi vish, thanks for the reply !
>
> You're right, I am trying to put instances on the same network as the
> host machine. This network already has a DHCP server on 138.96.126.1,
> which give IP for the range 138.96.126.1-199. Also, I cannot modify the
> config of this server.
>
> Is there any way so the host machine respond to the VM requests instead of the external DHCP server ?
> Looking at /var/lib/nova/instances/instance-00000019/libvirt.xml, it does not seems to be the case: <parameter name="DHCPSERVER" value="138.96.126.1" />
>
> Also, and to be complete, even if I put
> --flat_network_dhcp_start=138.96.126.200 in my nova.config file, started
> instances was assigned with an IP in the wrong range. Thus, I modified
> by hand the database, in order to exclude some range by setting the
> 'reserved' column in the fixed_ips table.
>
> As you asked, here is the list of dnsmasq processes.
>
> nobody 1029 0.0 0.0 21688 1072 ? S 18:16 0:00 dnsmasq
> --strict-order --bind-interfaces --pid-
> file=/var/run/libvirt/network/default.pid --conf-file= --except-
> interface lo --listen-address 192.168.122.1 --dhcp-range
> 192.168.122.2,192.168.122.254 --dhcp-lease-max=253 --dhcp-no-override
>
> nobody 1417 0.0 0.0 24388 1112 ? S 18:16 0:00 dnsmasq
> --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-
> file=/var/lib/nova/networks/nova-br100.pid --listen-address=138.96.126.1
> --except-interface=lo --dhcp-range=138.96.126.200,static,120s --dhcp-
> hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin
> /nova-dhcpbridge --leasefile-ro
>
> root 1418 0.0 0.0 24256 416 ? S 18:16 0:00 dnsmasq
> --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-
> file=/var/lib/nova/networks/nova-br100.pid --listen-address=138.96.126.1
> --except-interface=lo --dhcp-range=138.96.126.200,static,120s --dhcp-
> hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin
> /nova-dhcpbridge --leasefile-ro
>
> $ cat /var/lib/nova/networks/nova-br100.conf
> 02:16:3e:12:e4:ad,i-0000001e.novalocal,138.96.126.200
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

Brian Amedro (brian-amedro) said : #5

I still have the same problem:
2011-04-22 17:29:42,257 - DataSourceEc2.py[WARNING]: waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id

2011-04-22 17:29:42,259 - DataSourceEc2.py[WARNING]: 17:29:42 [ 1/100]: url error [[Errno 101] Network is unreachable]

Just to remind, my config is now:
$ cat /etc/nova/nova.conf
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--verbose
--libvirt_type=kvm
--network_manager=nova.network.manager.FlatDHCPManager
--fixed_range=138.96.224/27
--flat_network_dhcp_start=138.96.126.225
--flat_interface=eth0
--flat_injected=False

and iptables-save gives

# Generated by iptables-save v1.4.10 on Fri Apr 22 19:31:57 2011
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [6:360]
:POSTROUTING ACCEPT [6:360]
:nova-compute-OUTPUT - [0:0]
:nova-compute-POSTROUTING - [0:0]
:nova-compute-PREROUTING - [0:0]
:nova-compute-floating-snat - [0:0]
:nova-compute-snat - [0:0]
:nova-network-OUTPUT - [0:0]
:nova-network-POSTROUTING - [0:0]
:nova-network-PREROUTING - [0:0]
:nova-network-floating-snat - [0:0]
:nova-network-snat - [0:0]
:nova-postrouting-bottom - [0:0]
-A PREROUTING -j nova-compute-PREROUTING
-A PREROUTING -j nova-network-PREROUTING
-A OUTPUT -j nova-compute-OUTPUT
-A OUTPUT -j nova-network-OUTPUT
-A POSTROUTING -j nova-compute-POSTROUTING
-A POSTROUTING -j nova-network-POSTROUTING
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
-A POSTROUTING -j nova-postrouting-bottom
-A nova-compute-snat -j nova-compute-floating-snat
-A nova-network-POSTROUTING -s 138.96.224.0/27 -d 10.128.0.0/24 -j ACCEPT
-A nova-network-POSTROUTING -s 138.96.224.0/27 -d 138.96.224.0/27 -j ACCEPT
-A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 138.96.126.4:8773
-A nova-network-snat -j nova-network-floating-snat
-A nova-network-snat -s 138.96.224.0/27 -j SNAT --to-source 138.96.126.4
-A nova-postrouting-bottom -j nova-compute-snat
-A nova-postrouting-bottom -j nova-network-snat
COMMIT
# Completed on Fri Apr 22 19:31:57 2011
# Generated by iptables-save v1.4.10 on Fri Apr 22 19:31:57 2011
*mangle
:PREROUTING ACCEPT [15392:1070864]
:INPUT ACCEPT [15390:1070800]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [15266:1065332]
:POSTROUTING ACCEPT [15266:1065332]
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Fri Apr 22 19:31:57 2011
# Generated by iptables-save v1.4.10 on Fri Apr 22 19:31:57 2011
*filter
:INPUT ACCEPT [8347:581314]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [8270:577902]
:nova-compute-FORWARD - [0:0]
:nova-compute-INPUT - [0:0]
:nova-compute-OUTPUT - [0:0]
:nova-compute-inst-33 - [0:0]
:nova-compute-local - [0:0]
:nova-compute-sg-fallback - [0:0]
:nova-filter-top - [0:0]
:nova-network-FORWARD - [0:0]
:nova-network-INPUT - [0:0]
:nova-network-OUTPUT - [0:0]
:nova-network-local - [0:0]
-A INPUT -j nova-compute-INPUT
-A INPUT -j nova-network-INPUT
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-compute-FORWARD
-A FORWARD -j nova-network-FORWARD
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -j nova-filter-top
-A OUTPUT -j nova-compute-OUTPUT
-A OUTPUT -j nova-network-OUTPUT
-A nova-compute-inst-33 -m state --state INVALID -j DROP
-A nova-compute-inst-33 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A nova-compute-inst-33 -s 138.96.126.1/32 -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A nova-compute-inst-33 -s 138.96.126.0/24 -j ACCEPT
-A nova-compute-inst-33 -p icmp -j ACCEPT
-A nova-compute-inst-33 -p tcp -m tcp --dport 22 -j ACCEPT
-A nova-compute-inst-33 -j nova-compute-sg-fallback
-A nova-compute-local -d 138.96.126.203/32 -j nova-compute-inst-33
-A nova-compute-sg-fallback -j DROP
-A nova-filter-top -j nova-compute-local
-A nova-filter-top -j nova-network-local
-A nova-network-FORWARD -i br100 -j ACCEPT
-A nova-network-FORWARD -o br100 -j ACCEPT
COMMIT
# Completed on Fri Apr 22 19:31:57 2011

Thanks for your time.

Brian Amedro (brian-amedro) said : #6

I have some evolution on the error. Now it is a "timed out" error, instead of a "Network is unreachable".

==== euca-get-console-output ====
2011-04-22 18:24:52,350 - DataSourceEc2.py[WARNING]: waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id
2011-04-22 18:24:52,353 - DataSourceEc2.py[WARNING]: 18:24:52 [ 1/100]: url error [timed out]
2011-04-22 18:24:55,360 - DataSourceEc2.py[WARNING]: 18:24:55 [ 2/100]: url error [timed out]
2011-04-22 18:24:58,367 - DataSourceEc2.py[WARNING]: 18:24:58 [ 3/100]: url error [timed out]
2011-04-22 18:25:01,373 - DataSourceEc2.py[WARNING]: 18:25:01 [ 4/100]: url error [timed out]
2011-04-22 18:25:02,380 - DataSourceEc2.py[WARNING]: 18:25:02 [ 5/100]: url error [[Errno 113] No route to host]
2011-04-22 18:25:01,373 - DataSourceEc2.py[WARNING]: 18:25:01 [ 4/100]: url error [timed out]
Hundred times...

Also, when I ping the instance, I have a response from an other IP :
$ nova list
+----+-----------+--------+-----------+----------------+
| ID | Name | Status | Public IP | Private IP |
+----+-----------+--------+-----------+----------------+
| 35 | Server 35 | ACTIVE | | 138.96.126.227 |
+----+-----------+--------+-----------+----------------+

$ ping 138.96.126.227
PING 138.96.126.227 (138.96.126.227) 56(84) bytes of data.
From 138.96.126.225 icmp_seq=1 Destination Host Unreachable
From 138.96.126.225 icmp_seq=2 Destination Host Unreachable
From 138.96.126.225 icmp_seq=3 Destination Host Unreachable
From 138.96.126.225 icmp_seq=5 Destination Host Unreachable

Network was created with:
# nova-manage network create 138.96.126.224/27 1 32

Best Vish Ishaya (vishvananda) said : #7

Yes so it isn't actually getting the right ip. I'm not sure if there is a way to get a config like this to work. You may have to switch to using private addressing (like the 10.x default range) and assigning floating ips to give your vms ips on the 138 network.

Vish

On Apr 22, 2011, at 9:48 AM, Brian Amedro wrote:

> Question #153827 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/153827
>
> Brian Amedro gave more information on the question:
> I have some evolution on the error. Now it is a "timed out" error,
> instead of a "Network is unreachable".
>
> ==== euca-get-console-output ====
> 2011-04-22 18:24:52,350 - DataSourceEc2.py[WARNING]: waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id
> 2011-04-22 18:24:52,353 - DataSourceEc2.py[WARNING]: 18:24:52 [ 1/100]: url error [timed out]
> 2011-04-22 18:24:55,360 - DataSourceEc2.py[WARNING]: 18:24:55 [ 2/100]: url error [timed out]
> 2011-04-22 18:24:58,367 - DataSourceEc2.py[WARNING]: 18:24:58 [ 3/100]: url error [timed out]
> 2011-04-22 18:25:01,373 - DataSourceEc2.py[WARNING]: 18:25:01 [ 4/100]: url error [timed out]
> 2011-04-22 18:25:02,380 - DataSourceEc2.py[WARNING]: 18:25:02 [ 5/100]: url error [[Errno 113] No route to host]
> 2011-04-22 18:25:01,373 - DataSourceEc2.py[WARNING]: 18:25:01 [ 4/100]: url error [timed out]
> Hundred times...
>
>
> Also, when I ping the instance, I have a response from an other IP :
> $ nova list
> +----+-----------+--------+-----------+----------------+
> | ID | Name | Status | Public IP | Private IP |
> +----+-----------+--------+-----------+----------------+
> | 35 | Server 35 | ACTIVE | | 138.96.126.227 |
> +----+-----------+--------+-----------+----------------+
>
> $ ping 138.96.126.227
> PING 138.96.126.227 (138.96.126.227) 56(84) bytes of data.
>> From 138.96.126.225 icmp_seq=1 Destination Host Unreachable
>> From 138.96.126.225 icmp_seq=2 Destination Host Unreachable
>> From 138.96.126.225 icmp_seq=3 Destination Host Unreachable
>> From 138.96.126.225 icmp_seq=5 Destination Host Unreachable
>
>
> Network was created with:
> # nova-manage network create 138.96.126.224/27 1 32
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

Brian Amedro (brian-amedro) said : #8

I will turn my config this way.
Private addressing + floating ips on the public network will do the trick.

Many thanks for your time Vish.

Brian Amedro (brian-amedro) said : #9

Thanks Vish Ishaya, that solved my question.

Max Schneider (henry2k) said : #10

btw. is it normal that the vnet0 gets an "fe:" mac-address while the vm gets a "02:" mac?