Grub fails to recognise bpool filesystem when zstd compression is enabled

Asked by Nosehair

Ubuntu Mate 21.10

   [Edit added 27th July 2022]

    IMPORTANT -- This issue will also impact 22.04 and any other Debian-based distribution that has
                                 ZFS "zstd" compression available along with Grub2 versions up-to and including
                                 2.06.

                                 Setting ZFS compression to "zstd" on "bpool" will cause the system to become
                                 un-bootable and non-recoverable (just setting compression to "lz4" or "off" will
                                 -not- fix it).

                                 All on-disk data is unaffected and readable from a booted system (but remains
                                 inaccessible by Grub2).

Root and boot on ZFS (standard rpool/bpool from the installer when the system was initially built).

After changing the ZFS compression method from lz4 to zstd on bpool, grub failed to recognise the filesystem and the system was left in an un-bootable state. Booting from the live-CD and changing the compression method back to lz4 did not fix the issue. "grub-probe" reports "Unknown filesystem".

To be clear, this issue seems to stem from my changing the compression format on bpool from lz4 to zstd. The machine in question is normally running 24/7, so the problem -appeared- to be triggered by a normal run of Software Updater on June 24th 2022 (which triggered an initramfs rebuild and hence a reboot). Although the problem was triggered by it, there doesn't seem to have been any problem at all with the Software Update itself (which confused me for quite a while).

The boot failure is characterized by EFI successfully calling grub and the normal grub menu appearing, but any attempt to boot from the default kernel, back-up kernel, or previous ZFS saved snapshots from the grub history sub-menu all fail with "no filesystem found" messages.

Adding an external USB disk and copying the pools over with zfs send-receive and attempting to re-install grub on that external disk allso fails in exactly the same way.

The bpool and rpool pools can be imported successfully from the live-CD and all data is intact (a scrub runs without errors and when mounted the filesystems are all normal). As mentioned above, all of the system data can be successfully save-receive copied to a local disk or to a remote system without errors.

The only "fix" that I've found so far (to enable use of the machine) is to use ZFSBootMenu instead of grub. Obviously this isn't something that most users would want to do. I know it's a tall order (and beyond the mandate of the Ubuntu maintainers), but it would be nice to push a request for zstd support upstream to the grub team.

Question information

Language:
English Edit question
Status:
Expired
For:
Ubuntu grub2 Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Manfred Hampl (m-hampl) said :
#1

I find somewhat contradictory information:

Bug #1751517 seems to cover that topic (status: wishlist)

According to https://bugs.archlinux.org/task/63235 grub 2.04 is supposed to contain the required software, but apparently needs some kind of reinstall after the related configuration change.

Remark: Ubuntu 21.10 will be EOL in less than a week's time. You should upgrade to 22.04 in due course.

Revision history for this message
Nosehair (sbi-gaijin) said :
#2

Manfred,
   Many thanks for your reply to my question.

   Unfortunately, looking through the news info file for the grub2
project, I can only find a mention of lz4 for ZFS being added in
version 2.02 and nothing related to ZFS for 2.04 or 2.06 (which is the
current version shipped with 22.04).

   The Arch-Linux project refers to BTRFS (not ZFS) and, while I can
find references to zstd in the grub-core/fs/btrfs.c file, I can't find
any reference to it in the grub-core/fs/zfs files (although lz4 is
there), so it looks as though this problem may still be present in
22.04.

   I'm writing this on a 22.04 install (root on ZFS), but I haven't
had the chance to test the issue yet.

   Again. many thanks for taking the time to reply; I appreciate it.

                                    -John-

On Sun, 10 Jul 2022 at 18:51, Manfred Hampl
<email address hidden> wrote:
>
> Your question #702434 on Ubuntu changed:
> https://answers.launchpad.net/ubuntu/+question/702434
>
> Status: Open => Answered
>
> Manfred Hampl proposed the following answer:
> I find somewhat contradictory information:
>
> Bug #1751517 seems to cover that topic (status: wishlist)
>
> According to https://bugs.archlinux.org/task/63235 grub 2.04 is supposed
> to contain the required software, but apparently needs some kind of
> reinstall after the related configuration change.
>
> Remark: Ubuntu 21.10 will be EOL in less than a week's time. You should
> upgrade to 22.04 in due course.
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
> https://answers.launchpad.net/ubuntu/+question/702434/+confirm?answer_id=0
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/ubuntu/+question/702434
>
> You received this question notification because you asked the question.

Revision history for this message
Nosehair (sbi-gaijin) said :
#3

As far as I can tell from the upstream Debian changelog for grub2, the zstd library was added in Nov 2018 and implemented in the
 (grub) BTRFS code at the same time.

I don't see anything in the changelog or in the grub-core/fs/zfs directory to indicate that it has been implemented for ZFS, so it looks as though this issue might still persist through into 22.04.

In partial answer to my own question, I'd say that Ubuntu users had better stick with lz4 on their bpool filesystems for now.

              -John-

Revision history for this message
Manfred Hampl (m-hampl) said :
#4

I can't comment any further. You know more about the topic yourself than I do.

Revision history for this message
Nosehair (sbi-gaijin) said :
#5

Just a little bit more on this subject... The folks at OpenZFS/Debian/Ubuntu are waaay ahead of me (as always) and there is actually a little helper shipped under /usr/share/zfs/compatibility.d (in Ubuntu, anyway) which contains a "grub2" compatibility file. Because it's baked into ZFS, you don't even need to type in the long path name; just add "-o compatibility=grub2" as an argument when creating a new pool and it -should- ensure that your new pool only uses those features which are compatible with Grub2.

           zpool create -o compatibility=grub2 <NEW POOL NAME>

Revision history for this message
Launchpad Janitor (janitor) said :
#6

This question was expired because it remained in the 'Open' state without activity for the last 15 days.