OpenStack Compute (nova)

Cannot boot from volume with 2 devices

Asked by Gaurav Gupta

I am using the boot-from-volume extension to create an instance, where its root file system is on a nova volume. This works more of less (with some caveats). But when I try to boot an instance with 2 disks (each specified as a volume), the boot fails with a kernel panic. I am using KVM. I've narrowed it down to the libvirt configuration, where specifying just one device works and 2 causes the kernel panic.

Note, that none of the devices have a partition. The device "/dev/mapper/nova--volumes-volume--00000069" has the root file system and "/dev/mapper/nova--volumes-volume--0000006a" is empty. Both are formatted with ext3 filesystem.

Works:
---------
<domain type='kvm'>
    <name>instance-00000056</name>
    <memory>524288</memory>
    <os>
            <type>hvm</type>
            <kernel>/var/lib/nova/instances/instance-00000056/kernel</kernel>
                <cmdline>root=/dev/vda console=ttyS0</cmdline>
                <initrd>/var/lib/nova/instances/instance-00000056/ramdisk</initrd>
    </os>
    <features>
        <acpi/>
    </features>
    <vcpu>1</vcpu>
    <devices>
            <disk type='block'>
                <driver type='raw'/>
                <source dev='/dev/mapper/nova--volumes-volume--00000069'/>
                <target dev='vdb' bus='virtio'/>
            </disk>

        <interface type='bridge'>
            <source bridge='br100'/>
            <mac address='02:16:3e:0b:95:d3'/>
            <!-- <model type='virtio'/> CANT RUN virtio network right now -->
            <filterref filter="nova-instance-instance-00000056-02163e0b95d3">
                <parameter name="IP" value="10.10.0.13" />
                <parameter name="DHCPSERVER" value="10.10.0.1" />
            </filterref>
        </interface>
        ...

Fails:
------
<domain type='kvm'>
    <name>instance-00000056</name>
    <memory>524288</memory>
    <os>
            <type>hvm</type>
            <kernel>/var/lib/nova/instances/instance-00000056/kernel</kernel>
                <cmdline>root=/dev/vda console=ttyS0</cmdline>
                <initrd>/var/lib/nova/instances/instance-00000056/ramdisk</initrd>
    </os>
    <features>
        <acpi/>
    </features>
    <vcpu>1</vcpu>
    <devices>
            <disk type='block'>
                <driver type='raw'/>
                <source dev='/dev/mapper/nova--volumes-volume--00000069'/>
                <target dev='vdb' bus='virtio'/>
            </disk>
            <disk type='block'>
                <driver type='raw'/>
                <source dev='/dev/mapper/nova--volumes-volume--0000006a'/>
                <target dev='vda' bus='virtio'/>
            </disk>

        <interface type='bridge'>
            <source bridge='br100'/>
            <mac address='02:16:3e:0b:95:d3'/>
            <!-- <model type='virtio'/> CANT RUN virtio network right now -->
            <filterref filter="nova-instance-instance-00000056-02163e0b95d3">
                <parameter name="IP" value="10.10.0.13" />
                <parameter name="DHCPSERVER" value="10.10.0.1" />
            </filterref>
        </interface>
        ...

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
Gaurav Gupta
Solved:
Last query:
Last reply:
Revision history for this message
Gaurav Gupta (gagupta) said : 		#1

I forgot to mention. I am booting a CentOS 5.7 kernel with virto enabled.

Here are the last few lines of the console.log:
...
device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: <email address hidden>
Loading dm-log.ko module
Loading dm-region_hash.ko module
Loading dm-message.ko module
Loading dm-raid45.ko module
device-mapper: dm-raid45: initialized v0.2594l
Loading virtio.ko module
Loading virtio_ring.ko module
Loading virtio_pci.ko module
ACPI: PCI Interrupt 0000:00:04.0[A] -> Link [LNKD] -> GSI 11 (level, high) -> IRQ 11
ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
ACPI: PCI Interrupt 0000:00:05.0[A] -> Link [LNKA] -> GSI 10 (level, high) -> IRQ 10
ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10
ACPI: PCI Interrupt 0000:00:06.0[A] -> Link [LNKB] -> GSI 10 (level, high) -> IRQ 10
Loading virtio_blk.ko module
 vda: unknown partition table
 vdb: unknown partition table
Waiting for driver initialization.
Scanning and configuring dmraid supported devices
Trying to resume from LABEL=SWAP-sda2
Unable to access resume device (LABEL=SWAP-sda2)
Creating root device.
Mounting root filesystem.
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
Setting up other filesystems.
Setting up new root fs
setuproot: moving /dev failed: No such file or directory
no fstab.sys, mounting internal defaults
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
ERROR opening /dev/console: No such file or directory
Trying to use fd 0 instead.
WARNING: can't access (null)
exec of init ((null)) failed!!!: Bad address
Kernel panic - not syncing: Attempted to kill init!

Revision history for this message
Gaurav Gupta (gagupta) said : 		#2

I slept over the problem, tried this first thing in the morning and it worked :)...

If you look at the failed libvirt.xml file above, the device /dev/vdb is listed before /dev/vda. I suspected if for some reason the kernel is using /dev/vdb as the root filesystem (which will fail ofcourse). I reversed the order and listed /dev/vda before /dev/vdb, (destroyed, undefine and define the new libvirt.xml) and restarted the instance. It worked this time.

Now the problem is that libvirt.xml is generated automatically by Openstack compute. As a workaround I can try to manipulate the order in which I create block device mappings, but this would be a hack at best. This looks like a KVM bug to me. Any insight would be helpful.

Revision history for this message
Gaurav Gupta (gagupta) said : 		#3

I have a theory on why this might be happening here. It has to do with the KVM limitation of mapping devices and the device names. See this thread for details:
https://answers.launchpad.net/nova/+question/165695

In this case what's happening is that the hypervisor is attaching the device /dev/vdb first (since its listed first in the XML). But the guest picks up the device name /dev/vda for this device, because its available. Later the kernel tries to find the root filesystem on '/dev/vda' (because thats' what is selected as the root) and it fails for obvious reason.

I think it's a virsh bug. It should be smart about it and attach the devices in the right order.

Revision history for this message
Gaurav Gupta (gagupta) said : 		#4

Refer bug:
https://bugs.launchpad.net/nova/+bug/884984