Unable to ping or ssh Fedora 15 instance

Hi,

  l am using openstack cloud, setup at present has two servers. Base OS is ubuntu, on cloud controller(hostname - in01emt16) network, volume, scheduler services are running and there is one compute node(hostname - in01emt17).

  I am able to run ubuntu prepackaged images as well as custom built images from scratch including ubuntu natty. All my attempts to create CentOS, Fedora, RHEL6 custom images has failed.

I have on hand two issues, these are custom built images :

1. Fedora 15 instances boots & able to get euca-get-console-output, instance is running but unable to ping.

2. RHEL6 I could see the instance was "running" but couldn't ping the instance nor get euca-get-console-output.

I am refering to http://docs.openstack.org/cactus/openstack-compute/admin/content/creating-a-linux-image.html for creating images.

First I would like the Fedora issue to be fixed. Fedora 15 has changed the naming convention of NIC cards from eth0 to new standard, after going through some web pages disabled this feature by appending the following into your grub.conf kernel command line "biosdevname=0". There was no eth0, copied the ifcfg-lo to ifcfg-eth0 & made changes accordingly.

cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes

Disabled selinux, iptables, ip6tables.

Following are the configuration files :

Openstack controller :

 cat /etc/nova/nova.conf
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--verbose
--s3_host=10.144.199.85
--rabbit_host=192.168.3.1
--cc_host=192.168.3.1
--ec2_url=http://10.144.199.85:8773/services/Cloud
--fixed_range=192.168.0.0/16
--network_size=9
--FAKE_subdomain=ec2
--routing_source_ip=192.168.3.1
--sql_connection=mysql://root:nova@10.144.199.85/nova
--glance_host=192.168.3.1
--image_service=nova.image.glance.GlanceImageService
--iscsi_ip_prefix=192.168.3.
--network_manager=nova.network.manager.FlatDHCPManager
--flat_interface=eth1
--flat_injected=False
--public_interface=eth0
--network_host=10.144.199.85

 brctl show
bridge name bridge id STP enabled interfaces
br100 8000.d8d385e0062e no eth1
virbr0 8000.000000000000 yes

ip a sh
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 169.254.169.254/32 scope link lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether d8:d3:85:e0:06:2c brd ff:ff:ff:ff:ff:ff
    inet 10.144.199.85/24 brd 10.144.199.255 scope global eth0
    inet6 fe80::dad3:85ff:fee0:62c/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether d8:d3:85:e0:06:2e brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dad3:85ff:fee0:62e/64 scope link
       valid_lft forever preferred_lft forever
4: br100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether d8:d3:85:e0:06:2e brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.1/16 brd 192.168.255.255 scope global br100
    inet 192.168.3.1/25 brd 192.168.3.127 scope global br100
    inet6 fe80::dad3:85ff:fee0:62e/64 scope link
       valid_lft forever preferred_lft forever
5: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 86:5d:bd:40:cc:9b brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
nova-network-INPUT all -- anywhere anywhere
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps

Chain FORWARD (policy ACCEPT)
target prot opt source destination
nova-filter-top all -- anywhere anywhere
nova-network-FORWARD all -- anywhere anywhere
ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED
ACCEPT all -- 192.168.122.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
nova-filter-top all -- anywhere anywhere
nova-network-OUTPUT all -- anywhere anywhere

Chain nova-filter-top (2 references)
target prot opt source destination
nova-network-local all -- anywhere anywhere

Chain nova-network-FORWARD (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere

Chain nova-network-INPUT (1 references)
target prot opt source destination

Chain nova-network-OUTPUT (1 references)
target prot opt source destination

Chain nova-network-local (1 references)
target prot opt source destination

nova-compute node :

 cat /etc/nova/nova.conf
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--verbose
--s3_host=10.144.199.85
--rabbit_host=192.168.3.1
--cc_host=192.168.3.1
--ec2_url=http://10.144.199.85:8773/services/Cloud
--fixed_range=192.168.0.0/16
--network_size=8
--FAKE_subdomain=ec2
--routing_source_ip=192.168.3.2
--sql_connection=mysql://root:nova@10.144.199.85/nova
--glance_host=192.168.3.1
--image_service=nova.image.glance.GlanceImageService
--iscsi_ip_prefix=192.168.3.
--network_manager=nova.network.manager.FlatDHCPManager
--flat_interface=eth1
--flat_injected=False
--public_interface=eth0
--network_host=10.144.199.85

brctl show
bridge name bridge id STP enabled interfaces
br100 8000.1cc1def471aa no eth1
                                                                       vnet0
virbr0 8000.000000000000 yes

 ip a sh
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 1c:c1:de:f4:71:a8 brd ff:ff:ff:ff:ff:ff
    inet 10.144.199.86/24 brd 10.144.199.255 scope global eth0
    inet6 fe80::1ec1:deff:fef4:71a8/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 1c:c1:de:f4:71:aa brd ff:ff:ff:ff:ff:ff
    inet6 fe80::1ec1:deff:fef4:71aa/64 scope link
       valid_lft forever preferred_lft forever
4: br100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 1c:c1:de:f4:71:aa brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.2/16 brd 192.168.255.255 scope global br100
    inet6 fe80::1ec1:deff:fef4:71aa/64 scope link
       valid_lft forever preferred_lft forever
5: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 06:d9:47:38:ca:dc brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether fe:16:3e:0a:2a:02 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fe0a:2a02/64 scope link
       valid_lft forever preferred_lft forever

 euca-describe-instances
RESERVATION r-jkujc342 proj default
INSTANCE i-0000009e ami-0000008e 192.168.3.3 192.168.3.3 running None (proj, in01emt17) 0 m1.small 2011-07-04T09:19:20Z nova

euca-get-console-output i-0000009e

Welcome to Fedora release 15 (Lovelock)!

[ 1.845694] NET: Registered protocol family 10
[ 1.847164] systemd[1]: Set hostname to <localhost>.
Starting Collect Read-Ahead Data...
[ 1.940127] systemd-readahead-collect[315]: Disabling readahead collector due to execution in virtualized environment.Started Replay Read-Ahead Data.
Starting Bootup hack...
Starting Syslog Kernel Log Buffer Bridge...
Started Syslog Kernel Log Buffer Bridge.
Started Lock Directory.
Starting Media Directory...
Started Runtime Directory.
Starting Stdio Syslog Bridge...
Started Stdio Syslog Bridge.

Started Collect Read-Ahead Data.
Started Load legacy module configuration.
Started File System Check on Root Device.
Starting Remount Root FS...
Starting Remount API VFS...
Started Load Kernel Modules.
Started Set Up Additional Binary Formats.
Starting Apply Kernel Variables...
Starting Setup Virtual Console...
Starting Apply System Clock UTC Offset...
Started Apply Kernel Variables.
Started Media Directory.
Started Bootup hack.
Started Remount API VFS.
Started Apply System Clock UTC Offset.
Starting udev Kernel Device Manager...
Started udev Kernel Device Manager.
Starting udev Coldplug all Devices...
[ 2.110522] udev[338]: starting version 167
[ 2.128101] EXT4-fs (vda): re-mounted. Opts: (null)
Started Remount Root FS.
Starting Configure read-only root support...
Started udev Coldplug all Devices.
Starting udev Wait for Complete Device Initialization...
[ 2.434899] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004)
[ 2.458326] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
[ 2.458790] 8139cp 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, high) -> IRQ 11
[ 2.525702] 8139cp 0000:00:03.0: eth0: RTL-8139C+ at 0xf7926000, 02:16:3e:0a:2a:02, IRQ 11
[ 2.526994] microcode: CPU0 sig=0x623, pf=0x0, revision=0x0
[ 2.527496] microcode: Microcode Update Driver: v2.00 <email address hidden>, Peter Oruba
[ 2.537807] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0
[ 2.551528] 8139too: 8139too Fast Ethernet driver 0.9.28
Started udev Wait for Complete Device Initialization.
Starting Wait for storage scan...
Started Show Plymouth Boot Screen.
Started Wait for storage scan.
Starting Initialize storage subsystems (RAID, LVM, etc.)...
Starting Huge Pages File System...
Started Huge Pages File System.
Starting POSIX Message Queue File System...
Started POSIX Message Queue File System.
Started Initialize storage subsystems (RAID, LVM, etc.).
Starting Initialize storage subsystems (RAID, LVM, etc.)...
Started Initialize storage subsystems (RAID, LVM, etc.).
Started Mark the need to relabel after reboot.
Starting Enable all detected swap partitions...
Started Relabel all filesystems, if necessary.
Started Reconfigure the system on administrator request.
Starting Load Random Seed...
Starting Tell Plymouth To Write Out Runtime Data...
Starting Recreate Volatile Files and Directories...
Started Enable all detected swap partitions.
Started Load Random Seed.
Started Tell Plymouth To Write Out Runtime Data.
[ 2.954714] alsactlStarted Recreate Volatile Files and Directories.
Starting Console System Startup Logging...
Starting Restore Sound Card State...
Starting Bootup unhack...
Starting Job spooling tools...
Started Job spooling tools.
Starting Network Manager...
Starting Network Time Service...
Started Network Time Service.
Starting Avahi mDNS/DNS-SD Stack...
Starting System Logging Service...
Starting ABRT Automated Bug Reporting Tool...
Starting SYSV: Init script for live image....
Starting Command Scheduler...
Started Command Scheduler.
Starting irqbalance daemon...
Starting Machine Check Exception Logging Daemon...
Starting System Setup Keyboard...
Started System Setup Keyboard.
Starting D-Bus System Message Bus...
Started SYSV: Init script for live image..
Starting SYSV: sandbox, xguest and other apps that want to use pam_namespace require this script be run at boot. This service script does not actually run any service but sets up: /var/tmp, /tmp and home directories to be used by these tools. If you do not use sandbox, xguest or pam_namespace you can turn this service[320]: /sbin/alsactl: load_state:1586: Cannot open /var/lib/alsa/asound.state for reading: No such file or directory

Started Restore Sound Card State.
Started Bootup unhack.
Started irqbalance daemon.
Starting sandboxStarted Console System Startup Logging.
Started D-Bus System Message Bus.
[ 3.004229] ntpd[520]: ntpd 4.2.6p3@1.2290-o Fri May 6 16:27:05 UTC 2011 (1)
[ 3.006281] ntpd[520]: proto: precision = 0.567 usec
[ 3.007404] ntpd[520]: 0.0.0.0 c01d 0d kern kernel time sync enabled
[ 3.018723] ntpd[520]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
[ 3.026053] avahi-daemon[521]: Found user 'avahi' (UID 70) and group 'avahi' (GID 70).
[ 3.031347] avahi-daemon[521]: Successfully dropped root privileges.
[ 3.032378] avahi-daemon[521]: avahi-daemon 0.6.30 starting up.
[ 3.039152] ntpd[520]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
[ 3.040949] ntpd[520]: Listen and drop on 1 v6wildcard :: UDP 123
[ 3.041945] ntpd[520]: Listen normally on 2 lo 127.0.0.1 UDP 123
[ 3.052051] ntpd[520]: Listen normally on 3 lo ::1 UDP 123
[ 3.056679] ntpd[520]: peers refreshed
[ 3.057344] ntpd[520]: Listening on routing socket on fd #20 for interface updates
[ 3.060163] ntpd[520]: Deferring DNS for 0.fedora.pool.ntp.org 1
[ 3.063807] ntpd[520]: Deferring DNS for 1.fedora.pool.ntp.org 1
[ 3.064792] ntpd[520]: Deferring DNS for 2.fedora.pool.ntp.org 1
[ 3.065778] ntpd[520]: Deferring DNS for 3.fedora.pool.ntp.org 1
[ 3.066752] ntpd[ OK ]
[520]: 0.0.0.0 c016 06 restart
[ 3.076702] ntpd[520]: 0.0.0.0 c012 02 freq_set kernel 0.000 PPM
[ 3.077234] ntpd[520]: 0.0.0.0 c011 01 freq_not_set
[ 3.077659] NetworkManager[519]: <info> NetworkManager (version 0.8.9997-4.git20110620.fc15) is starting...
[ 3.078457] NetworkManager[519]: <info> Read config file /etc/NetworkManager/NetworkManager.conf
[ 3.079175] mcelog.setup[527]: read: No such device
[ 3.085689] /usr/sbin/crond[525]: (CRON) INFO (running with inotify support)
[ 3.113818] abrtd[523]: Init complete, entering main loop
Started Machine Check Exception Logging Daemon.
Started SYSV: sandbox, xguest and other apps that want to use pam_namespace require this script be run at boot. This service script does not actually run any service but sets up: /var/tmp, /tmp and home directories to be used by these tools. If you do not use sandbox, xguest or pam_namespace you can turn this service off..
Starting LSB: Port reservation utility...
Starting LSB: Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
Starting LSB: Starts and stops login iSCSI daemon....
Started ABRT Automated Bug Reporting Tool.
Started Avahi mDNS/DNS-SD Stack.
Stopping Syslog Kernel Log Buffer Bridge...
Stopped Syslog Kernel Log Buffer Bridge.
Started System Logging Service.
Starting portreserve: Started LSB: Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
[ OK ]
Started LSB: Port reservation utility.
Started D-Bus System Message Bus.
Started LSB: Starts and stops login iSCSI daemon..
Starting SYSV: This starts the Linux Auditing System Daemon, which collects security related events in a dedicated audit log. If this daemon is turned off, audit events will be sent to syslog....
Starting auditd: [ 3.297080] auditd (599): /proc/599/oom_adj is deprecated, please use /proc/599/oom_score_adj instead.
Started Network Manager.
[ 3.341582] eth0: link up, 100Mbps, full-duplex, lpa 0x05E1
[ OK ]
Started SYSV: This starts the Linux Auditing System Daemon, which collects security related events in a dedicated audit log. If this daemon is turned off, audit events will be sent to syslog..
Starting LSB: Installs coredump handler which saves segfault data...
Starting LSB: Starts and stops login and scanning of iSCSI devices....
Starting LSB: processor frequency scaling support...
Starting LSB: Start up the OpenSSH server daemon...
Starting LSB: Link Layer Discovery Protocol Agent Daemon...
Starting LSB: Mount and unmount network filesystems....
Starting LSB: Start and stop the MD software RAID monitor...
Starting LSB: The CUPS scheduler...
Starting LSB: start and stop sendmail...
Starting LSB: Watches system log for oops messages, creates ABRT dump directories for each oops...
Starting /etc/rc.local Compatibility...
Starting lldpad: Starting cups: Starting mdmonitor: Starting sshd: Started LSB: Watches system log for oops messages, creates ABRT dump directories for each oops.
Started LSB: Link Layer Discovery Protocol Agent Daemon.
[ 3.652114] 802.1Q VLAN Support v1.8 Ben Greear <email address hidden>
[ 3.652658] All bugs added by David S. Miller <email address hidden>
Started LSB: Installs coredump handler which saves segfault data.
Starting sendmail: [ OK ]
Started LSB: Start and stop the MD software RAID monitor.
[ OK ]
Started LSB: Start up the OpenSSH server daemon.
[ OK ]
Started LSB: The CUPS scheduler.
Started LSB: processor frequency scaling support.
[ OK ]
Starting sm-client: [ OK ]
Started LSB: start and stop sendmail.
AUTHORIZED_KEYS:
************************

************************
Started /etc/rc.local Compatibility.
Starting Wait for Plymouth Boot Screen to Quit...
Started LSB: Mount and unmount network filesystems..
Starting Permit User Sessions...
Started Permit User Sessions.
Started LSB: Starts and stops login and scanning of iSCSI devices..
Starting SYSV: Late init script for live image....
Started SYSV: Late init script for live image..

Fedora release 15 (Lovelock)
Kernel 2.6.38.8-32.fc15.i686 on an i686 (ttyS0)

localhost login:
root@in01emt16:~# euca-get-console-output i-0000009e > output
root@in01emt16:~# less output
root@in01emt16:~#
root@in01emt16:~# less output
[ OK ]
Started LSB: The CUPS scheduler.
Started LSB: processor frequency scaling support.
[ OK ]
Starting sm-client: [ OK ]
Started LSB: start and stop sendmail.
AUTHORIZED_KEYS:
************************

************************
Started /etc/rc.local Compatibility.
Starting Wait for Plymouth Boot Screen to Quit...
Started LSB: Mount and unmount network filesystems..
Starting Permit User Sessions..
Started Permit User Sessions.
Started LSB: Starts and stops login and scanning of iSCSI devices..
Starting SYSV: Late init script for live image....
Started SYSV: Late init script for live image..

Fedora release 15 (Lovelock)
Kernel 2.6.38.8-32.fc15.i686 on an i686 (ttyS0)

localhost login:

There is another question which is similar to mine - https://answers.launchpad.net/nova/+question/150776

Please provide ur insights, help me fix the issue. I have a deadline of couple of days to resolve it, any help or suggestion is highly appreciated.

Thanks,

Sudhir