Cannot connect to a running instance, even after the euca-authorize stuff

I already saw https://answers.launchpad.net/nova/+question/145820 which looks like to be a similar problem, but I am already with FlatDHCPManager.
I also tried to connect to the VM through a serial console, but it fails:

==== $ virsh console instance-00000019 ====
Connected to domain instance-00000019
Escape character is ^]
error: internal error character device (null) is not using a PTY

==== $ cat /var/lib/nova/instances/instance-00000019/libvirt.xml ====
<domain type='kvm'>
    <name>instance-00000019</name>
    <memory>524288</memory>
    <os>
            <type>hvm</type>
            <kernel>/var/lib/nova/instances/instance-00000019/kernel</kernel>
                <cmdline>root=/dev/vda console=ttyS0</cmdline>
    </os>
    <features>
        <acpi/>
    </features>
    <vcpu>1</vcpu>
    <devices>
        <disk type='file'>
            <driver type='qcow2'/>
            <source file='/var/lib/nova/instances/instance-00000019/disk'/>
            <target dev='vda' bus='virtio'/>
        </disk>

        <interface type='bridge'>
            <source bridge='br100'/>
            <mac address='02:16:3e:1b:62:26'/>
            <!-- <model type='virtio'/> CANT RUN virtio network right now -->
            <filterref filter="nova-instance-instance-00000019-02163e1b6226">
                <parameter name="IP" value="138.96.126.201" />
                <parameter name="DHCPSERVER" value="138.96.126.1" />
            </filterref>
        </interface>
        <!-- The order is significant here. File must be defined first -->
        <serial type="file">
            <source path='/var/lib/nova/instances/instance-00000019/console.log'/>
            <target port='1'/>
        </serial>

        <console type='pty' tty='/dev/pts/2'>
            <source path='/dev/pts/2'/>
            <target port='0'/>
        </console>

        <serial type='pty'>
            <source path='/dev/pts/2'/>
            <target port='0'/>
        </serial>

        <graphics type='vnc' port='-1' autoport='yes' keymap='en-us' listen='0.0.0.0'/>
    </devices>
</domain>

i would guess problems with dnsmasq. It looks like you are trying to put instances on the same network as your host machine which may be difficult to get working if you have an external router or gateway responding to dnsmasq queries. I would check the process list for dnsmasq and make sure that there are exactly two running and that the command line for them is the same and it looks reasonable.

Vish

On Apr 22, 2011, at 3:58 AM, Brian Amedro wrote:

> Question #153827 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/153827
>
> Brian Amedro gave more information on the question:
> I already saw https://answers.launchpad.net/nova/+question/145820 which looks like to be a similar problem, but I am already with FlatDHCPManager.
> I also tried to connect to the VM through a serial console, but it fails:
>
> ==== $ virsh console instance-00000019 ====
> Connected to domain instance-00000019
> Escape character is ^]
> error: internal error character device (null) is not using a PTY
>
>
> ==== $ cat /var/lib/nova/instances/instance-00000019/libvirt.xml ====
> <domain type='kvm'>
> <name>instance-00000019</name>
> <memory>524288</memory>
> <os>
> <type>hvm</type>
> <kernel>/var/lib/nova/instances/instance-00000019/kernel</kernel>
> <cmdline>root=/dev/vda console=ttyS0</cmdline>
> </os>
> <features>
> <acpi/>
> </features>
> <vcpu>1</vcpu>
> <devices>
> <disk type='file'>
> <driver type='qcow2'/>
> <source file='/var/lib/nova/instances/instance-00000019/disk'/>
> <target dev='vda' bus='virtio'/>
> </disk>
>
> <interface type='bridge'>
> <source bridge='br100'/>
> <mac address='02:16:3e:1b:62:26'/>
> <!-- <model type='virtio'/> CANT RUN virtio network right now -->
> <filterref filter="nova-instance-instance-00000019-02163e1b6226">
> <parameter name="IP" value="138.96.126.201" />
> <parameter name="DHCPSERVER" value="138.96.126.1" />
> </filterref>
> </interface>
> <!-- The order is significant here. File must be defined first -->
> <serial type="file">
> <source path='/var/lib/nova/instances/instance-00000019/console.log'/>
> <target port='1'/>
> </serial>
>
> <console type='pty' tty='/dev/pts/2'>
> <source path='/dev/pts/2'/>
> <target port='0'/>
> </console>
>
> <serial type='pty'>
> <source path='/dev/pts/2'/>
> <target port='0'/>
> </serial>
>
> <graphics type='vnc' port='-1' autoport='yes' keymap='en-us' listen='0.0.0.0'/>
> </devices>
> </domain>
>
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

Hi vish, thanks for the reply !

You're right, I am trying to put instances on the same network as the host machine. This network already has a DHCP server on 138.96.126.1, which give IP for the range 138.96.126.1-199. Also, I cannot modify the config of this server.

Is there any way so the host machine respond to the VM requests instead of the external DHCP server ?
Looking at /var/lib/nova/instances/instance-00000019/libvirt.xml, it does not seems to be the case: <parameter name="DHCPSERVER" value="138.96.126.1" />

Also, and to be complete, even if I put --flat_network_dhcp_start=138.96.126.200 in my nova.config file, started instances was assigned with an IP in the wrong range. Thus, I modified by hand the database, in order to exclude some range by setting the 'reserved' column in the fixed_ips table.

As you asked, here is the list of dnsmasq processes.

nobody 1029 0.0 0.0 21688 1072 ? S 18:16 0:00 dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/libvirt/network/default.pid --conf-file= --except-interface lo --listen-address 192.168.122.1 --dhcp-range 192.168.122.2,192.168.122.254 --dhcp-lease-max=253 --dhcp-no-override

nobody 1417 0.0 0.0 24388 1112 ? S 18:16 0:00 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=138.96.126.1 --except-interface=lo --dhcp-range=138.96.126.200,static,120s --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro

root 1418 0.0 0.0 24256 416 ? S 18:16 0:00 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=138.96.126.1 --except-interface=lo --dhcp-range=138.96.126.200,static,120s --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro

$ cat /var/lib/nova/networks/nova-br100.conf
02:16:3e:12:e4:ad,i-0000001e.novalocal,138.96.126.200

Yes that is your problem. It attempts to give the first address in the subnet to your machine and it is conflicting with your external dns server. You could try using a smaller network, such as:
128.96.126.224/27 although your .1 server may still give out addresses and mess things up. Worth a shot.
You'll have to delete networks and fixed_ip tables and recreate the smaller range with nova-network network create
you should also change:
--fixed_range=128.96.224/27
--flat_network_dhcp_start=138.96.126.225

killall dnsmasq
iptables -F
iptabbles -t nat -F
restart nova-network

try again?

Vish

On Apr 22, 2011, at 7:52 AM, Brian Amedro wrote:

> Question #153827 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/153827
>
> Status: Answered => Open
>
> Brian Amedro is still having a problem:
> Hi vish, thanks for the reply !
>
> You're right, I am trying to put instances on the same network as the
> host machine. This network already has a DHCP server on 138.96.126.1,
> which give IP for the range 138.96.126.1-199. Also, I cannot modify the
> config of this server.
>
> Is there any way so the host machine respond to the VM requests instead of the external DHCP server ?
> Looking at /var/lib/nova/instances/instance-00000019/libvirt.xml, it does not seems to be the case: <parameter name="DHCPSERVER" value="138.96.126.1" />
>
> Also, and to be complete, even if I put
> --flat_network_dhcp_start=138.96.126.200 in my nova.config file, started
> instances was assigned with an IP in the wrong range. Thus, I modified
> by hand the database, in order to exclude some range by setting the
> 'reserved' column in the fixed_ips table.
>
> As you asked, here is the list of dnsmasq processes.
>
> nobody 1029 0.0 0.0 21688 1072 ? S 18:16 0:00 dnsmasq
> --strict-order --bind-interfaces --pid-
> file=/var/run/libvirt/network/default.pid --conf-file= --except-
> interface lo --listen-address 192.168.122.1 --dhcp-range
> 192.168.122.2,192.168.122.254 --dhcp-lease-max=253 --dhcp-no-override
>
> nobody 1417 0.0 0.0 24388 1112 ? S 18:16 0:00 dnsmasq
> --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-
> file=/var/lib/nova/networks/nova-br100.pid --listen-address=138.96.126.1
> --except-interface=lo --dhcp-range=138.96.126.200,static,120s --dhcp-
> hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin
> /nova-dhcpbridge --leasefile-ro
>
> root 1418 0.0 0.0 24256 416 ? S 18:16 0:00 dnsmasq
> --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-
> file=/var/lib/nova/networks/nova-br100.pid --listen-address=138.96.126.1
> --except-interface=lo --dhcp-range=138.96.126.200,static,120s --dhcp-
> hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin
> /nova-dhcpbridge --leasefile-ro
>
> $ cat /var/lib/nova/networks/nova-br100.conf
> 02:16:3e:12:e4:ad,i-0000001e.novalocal,138.96.126.200
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

I still have the same problem:
2011-04-22 17:29:42,257 - DataSourceEc2.py[WARNING]: waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id

2011-04-22 17:29:42,259 - DataSourceEc2.py[WARNING]: 17:29:42 [ 1/100]: url error [[Errno 101] Network is unreachable]

Just to remind, my config is now:
$ cat /etc/nova/nova.conf
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--verbose
--libvirt_type=kvm
--network_manager=nova.network.manager.FlatDHCPManager
--fixed_range=138.96.224/27
--flat_network_dhcp_start=138.96.126.225
--flat_interface=eth0
--flat_injected=False

and iptables-save gives

# Generated by iptables-save v1.4.10 on Fri Apr 22 19:31:57 2011
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [6:360]
:POSTROUTING ACCEPT [6:360]
:nova-compute-OUTPUT - [0:0]
:nova-compute-POSTROUTING - [0:0]
:nova-compute-PREROUTING - [0:0]
:nova-compute-floating-snat - [0:0]
:nova-compute-snat - [0:0]
:nova-network-OUTPUT - [0:0]
:nova-network-POSTROUTING - [0:0]
:nova-network-PREROUTING - [0:0]
:nova-network-floating-snat - [0:0]
:nova-network-snat - [0:0]
:nova-postrouting-bottom - [0:0]
-A PREROUTING -j nova-compute-PREROUTING
-A PREROUTING -j nova-network-PREROUTING
-A OUTPUT -j nova-compute-OUTPUT
-A OUTPUT -j nova-network-OUTPUT
-A POSTROUTING -j nova-compute-POSTROUTING
-A POSTROUTING -j nova-network-POSTROUTING
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
-A POSTROUTING -j nova-postrouting-bottom
-A nova-compute-snat -j nova-compute-floating-snat
-A nova-network-POSTROUTING -s 138.96.224.0/27 -d 10.128.0.0/24 -j ACCEPT
-A nova-network-POSTROUTING -s 138.96.224.0/27 -d 138.96.224.0/27 -j ACCEPT
-A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 138.96.126.4:8773
-A nova-network-snat -j nova-network-floating-snat
-A nova-network-snat -s 138.96.224.0/27 -j SNAT --to-source 138.96.126.4
-A nova-postrouting-bottom -j nova-compute-snat
-A nova-postrouting-bottom -j nova-network-snat
COMMIT
# Completed on Fri Apr 22 19:31:57 2011
# Generated by iptables-save v1.4.10 on Fri Apr 22 19:31:57 2011
*mangle
:PREROUTING ACCEPT [15392:1070864]
:INPUT ACCEPT [15390:1070800]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [15266:1065332]
:POSTROUTING ACCEPT [15266:1065332]
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Fri Apr 22 19:31:57 2011
# Generated by iptables-save v1.4.10 on Fri Apr 22 19:31:57 2011
*filter
:INPUT ACCEPT [8347:581314]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [8270:577902]
:nova-compute-FORWARD - [0:0]
:nova-compute-INPUT - [0:0]
:nova-compute-OUTPUT - [0:0]
:nova-compute-inst-33 - [0:0]
:nova-compute-local - [0:0]
:nova-compute-sg-fallback - [0:0]
:nova-filter-top - [0:0]
:nova-network-FORWARD - [0:0]
:nova-network-INPUT - [0:0]
:nova-network-OUTPUT - [0:0]
:nova-network-local - [0:0]
-A INPUT -j nova-compute-INPUT
-A INPUT -j nova-network-INPUT
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-compute-FORWARD
-A FORWARD -j nova-network-FORWARD
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -j nova-filter-top
-A OUTPUT -j nova-compute-OUTPUT
-A OUTPUT -j nova-network-OUTPUT
-A nova-compute-inst-33 -m state --state INVALID -j DROP
-A nova-compute-inst-33 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A nova-compute-inst-33 -s 138.96.126.1/32 -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A nova-compute-inst-33 -s 138.96.126.0/24 -j ACCEPT
-A nova-compute-inst-33 -p icmp -j ACCEPT
-A nova-compute-inst-33 -p tcp -m tcp --dport 22 -j ACCEPT
-A nova-compute-inst-33 -j nova-compute-sg-fallback
-A nova-compute-local -d 138.96.126.203/32 -j nova-compute-inst-33
-A nova-compute-sg-fallback -j DROP
-A nova-filter-top -j nova-compute-local
-A nova-filter-top -j nova-network-local
-A nova-network-FORWARD -i br100 -j ACCEPT
-A nova-network-FORWARD -o br100 -j ACCEPT
COMMIT
# Completed on Fri Apr 22 19:31:57 2011

Thanks for your time.

I have some evolution on the error. Now it is a "timed out" error, instead of a "Network is unreachable".

==== euca-get-console-output ====
2011-04-22 18:24:52,350 - DataSourceEc2.py[WARNING]: waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id
2011-04-22 18:24:52,353 - DataSourceEc2.py[WARNING]: 18:24:52 [ 1/100]: url error [timed out]
2011-04-22 18:24:55,360 - DataSourceEc2.py[WARNING]: 18:24:55 [ 2/100]: url error [timed out]
2011-04-22 18:24:58,367 - DataSourceEc2.py[WARNING]: 18:24:58 [ 3/100]: url error [timed out]
2011-04-22 18:25:01,373 - DataSourceEc2.py[WARNING]: 18:25:01 [ 4/100]: url error [timed out]
2011-04-22 18:25:02,380 - DataSourceEc2.py[WARNING]: 18:25:02 [ 5/100]: url error [[Errno 113] No route to host]
2011-04-22 18:25:01,373 - DataSourceEc2.py[WARNING]: 18:25:01 [ 4/100]: url error [timed out]
Hundred times...

Also, when I ping the instance, I have a response from an other IP :
$ nova list
+----+-----------+--------+-----------+----------------+
| ID | Name | Status | Public IP | Private IP |
+----+-----------+--------+-----------+----------------+
| 35 | Server 35 | ACTIVE | | 138.96.126.227 |
+----+-----------+--------+-----------+----------------+

$ ping 138.96.126.227
PING 138.96.126.227 (138.96.126.227) 56(84) bytes of data.
From 138.96.126.225 icmp_seq=1 Destination Host Unreachable
From 138.96.126.225 icmp_seq=2 Destination Host Unreachable
From 138.96.126.225 icmp_seq=3 Destination Host Unreachable
From 138.96.126.225 icmp_seq=5 Destination Host Unreachable

Network was created with:
# nova-manage network create 138.96.126.224/27 1 32

I will turn my config this way.
Private addressing + floating ips on the public network will do the trick.

Many thanks for your time Vish.

Thanks Vish Ishaya, that solved my question.

btw. is it normal that the vnet0 gets an "fe:" mac-address while the vm gets a "02:" mac?