XEN fails to load Dom0 kernel under 21.10 -- with manual FIX

Asked by Christopher Dutchyn

Upgrading to 2110 from 20.04 on a machine running the Xen hypervisor leads to a fail-to-boot into Xen, although it will boot into native Ubuntu.The error message is
   (XEN) ELF: not an ELF binary
   (XEN) ****************************************
   (XEN) Panic on CPU 0:
   (XEN) Could not set up DOM0 guest OS
   (XEN) ****************************************
   (XEN) Reboot in 5 seconds...

The problem is well-known: Xen does not understand all of the compression formats for vmlinuz, and cannot load newer kernels that are compressed with the newer formats. Redhat has a bug report (https://bugzilla.redhat.com/show_bug.cgi?id=1898662), ArchLinux has a fix (https://bbs.archlinux.org/viewtopic.php?id=118525). That solution is to ue Torvalds' `extract-vmlinux' script (https://github.com/torvalds/linux/blob/master/scripts/extract-vmlinux) to obtain an uncompressed kernel [a vmlinux-... file] and then correct the module2 line to use the uncompressed kernel in all the XEN-related /boot/grub/grub.cfg entries.

I put extract-vmlinux into /boot, and used it to extract vmlinux-5.13.0-19-generic from vmlinuz-5.13.0-19-generic -- beware, extract-vmlinux takes a filename, but extracts the kernel to stdout (!); so you need to type something like
        # cd /boot
        # ./extract-vmlinux ./vmlinuz-5.13.0-19.generic > ./vmlinux-5.13.0-19-generic
I did this for the 5.13.0-20 kernel too.

Changing the lines in /boot/grup/grub.cfg is simple, here's an example of what mine says now:
       multiboot2 /boot/xen-4.11-amd64.gz placeholder ucode=scan ${xen_rm_opts}
        echo 'Loading Linux 5.13.0-20-generic ...'
        #module2 /boot/vmlinuz-5.13.0-20-generic placeholder root=UUID=cc5800e2-d9d9-4138-b390-014f77aec5ea ro
        module2 /boot/vmlinux-5.13.0-20-generic placeholder root=UUID=cc5800e2-d9d9-4138-b390-014f77aec5ea ro
        echo 'Loading initial ramdisk ...'
        module2 --nounzip /boot/initrd.img-5.13.0-20-generic

The commented-out module2 line is what was present in the original generated grub.cfg; I duplicated the line, commented out the first instance, and changed the 'z' in vmlinuz to 'x' in the second instance.

This will need to be done manually anytime a new kernel is installed; at least until Xen is improved to understand the new kernel compression formats. Given the laissez-faire attitude espoused by Michael Young (a frequent contributor to Xen) in the Redhat report, "I was wonder how long it would be before someone else noticed that", I expect it may never be fixed. ArchLinux's solution (essentially what I described above) has been incorporated as a change into their automatic update system, so the uncompressed kernel is always available. Ubuntu could do the same thing.

The answer given below is irrelevent, appearing to be little more than a copy-paste of the top 2 hits from Google search with obvious keywords -- probably without even verifying that they affect the situation. Fortunately, a work-around is given above as a manual activity, and directs us to ArchLinux's automated fix. Note that the work-around can be effected once the system boots into Linux-without-Xen, which can be selected from the GRUB boot menu; so the system is repairable.

Question information

English Edit question
Ubuntu xen Edit question
No assignee Edit question
Solved by:
Christopher Dutchyn
Last query:
Last reply:
Revision history for this message
Bernard Stafford (bernard010) said :
Revision history for this message
Christopher Dutchyn (cdutchyn) said :

Ignore Mr Stafford's man-page links. The problem is XEN doesn't understand all possible kernel compressions when loading Dom0. It's not about deleting a (migrated) XEN image (that would a DomU activity), and it's not about creating/destroying/manipulating DomU instances either. Perhaps if he had read the problem and looked at the man-pages his search found, he could have saved distracting others with red-herrings. It might have even dawned on him that the problem statement
1) points out that XEN developers know this, and were even wondering when people would run over this pothole; and
2) gives a manual solution which needs to run whenever a new kernel is installed (and which ArchLinux offers a patch to automated).

Revision history for this message
Christopher Dutchyn (cdutchyn) said :

<s>Thank you for duplicating my posting; the more copies, the more eyeballs will see it — after all, we know that, despite their name, search engines can’t find find things on the web.</s> Continuing along that specious line of reasoning, you should reduplicate it hundreds or thousands of times — who cares about resources or originality: you might even get an award for volume.


> On Jan 26, 2022, at 02:15, Sébastien Chaumat <email address hidden> wrote:
> CAUTION: External to USask. Verify sender and use caution with links and attachments. Forward suspicious emails to <email address hidden>
> Your question #699338 on xen in Ubuntu changed:
> https://answers.launchpad.net/ubuntu/+source/xen/+question/699338
> Linked to bug: #1959076
> https://bugs.launchpad.net/bugs/1959076
> "xen 4.11 panic at boot because missing support for XZ compressed kernel"
> --
> You received this question notification because you asked the question.