[20.04 FEAT] Support/enhancement of NVMe IPL

Bug #1902179 reported by bugproxy
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Release Notes for Ubuntu
Fix Released
Undecided
Unassigned
Ubuntu on IBM z Systems
Fix Released
High
Skipper Bug Screeners
linux (Ubuntu)
Fix Released
Medium
Unassigned
Focal
Fix Released
Medium
Unassigned
Groovy
Fix Released
Medium
Unassigned
Hirsute
Fix Released
Medium
Unassigned
s390-tools (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Released
Wishlist
Canonical Foundations Team
Groovy
Fix Released
Undecided
Unassigned
Hirsute
Fix Released
Undecided
Unassigned

Bug Description

SRU Justification: (focal)
==================

[Impact]

* The base for being able to IPL (boot) NVMe devices on s390x was introduced with kernel 5.8.

* This got now requested (for hardware enablement reasons) for Ubuntu 20.04 LTS as well.

* On top a brand new commit got upstream accepted that introduces support for NVMe IPL kernel parameters, which is not yet in groovy.

[Fix]

* cherry pick of commit 3737e8ee4f2fc7e77994d1a8bd618a9dda5a5514 3737e8ee4f2f "s390: nvme ipl"

* cherry pick of commit 23a457b8d57dc8d0cc1dbd1882993dd2fcc4b0c0 23a457b8d57d "s390: nvme reipl"
  does not apply cleanly, hence the following backport:
  https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1902179/+attachment/5430310/+files/0002-s390-nvme-reipl.patch

* cherry pick of commit d9f12e48d08ec08ace574050a838e001e442ee38 d9f12e48d08e "s390/ipl: support NVMe IPL kernel parameters"

[Test Case]

* IBM z15 or LinuxONE III hardware is needed with an NVMe device attached to a LPAR.

* Install the patched kernel on focal/20.04 installation and make sure that zipl re-ran (the patched version of zipl with the s390-tools commit mentioned in this LP bug - or take the s390-tools version for groovy for testing purposes).

* If everything is in place (means patched kernel, as well as patched s390-tools/zipl) an installation from scratch on an NVMe devices should be possible - in case everything needed landed on an updated image.
  With the 20.04.2 image that should be the case.

[Regression Potential]

* There is a certain regression risk with these patches, because:

* the 'zipl' (s390x-specific) boot-loader is touched

* if something is wrong there, in worst-case systems where the modified zipl ran may no longer be bootable!

* The modifications are targetted towards nvme devices ('blkext' driver), but they are closely related to zFCP devices and share some code parts,

* hence in worst case they could have an impact on zFCP devices, too.

* But this is very unlikely, since a (largely) separate IPL type 'nvme' got introduced and NVMe ipl is handled in separate case statements and functions.

* The patches are all upstream accepted (the first two with 5.8, that last with v5.10-rc1, hence the latter one is as of today in linux-next).

* A patched focal kernel was build and shared for further testing. I did some regression testing with the patched kernel on non-NVMe systems - the NVMe based tests need to be done by IBM (due to the lack of hardware).

* All modifications are limited to the s390x architecture and there again to the unique way of booting aka IPL (arch/s390/include/asm/ipl.h, arch/s390/include/uapi/asm/ipl.h, arch/s390/kernel/ipl.c and arch/s390/boot/ipl_parm.c).

[Other]

* The general NVMe ipl (boot) functionality in given with 3737e8ee4f2f "s390: nvme ipl" and 23a457b8d57d "s390: nvme reipl" and is already proven to work with groovy.

* New for groovy AND focal is only "s390/ipl: support NVMe IPL kernel parameters".

* The entire set of commits/patches is only new for focal.

* The SRU for SRUing "s390/ipl: support NVMe IPL kernel parameters" to groovy/20.10 was handled by a separate request.

_________________________

SRU Justification: (groovy)
==================

[Impact]

* The basics for being able to IPL (boot) from NVMe devices on s390x were introduced with kernel 5.8.

* This was tested and is proven to work with groovy.

* Now a patch was requested to be added to groovy that introduces support for NVMe IPL kernel parameters.

[Fix]

* d9f12e48d08ec08ace574050a838e001e442ee38 d9f12e48d08e "s390/ipl: support NVMe IPL kernel parameters"

[Test Case]

* IBM z15 or LinuxONE III hardware is needed with an NVMe device attached to a LPAR.

* Just check if NVMe kernel parameters can be passed over.

* Due to the lack of hardware this test needs to be done by IBM.

[Regression Potential]

* There is a certain regression risk with this patch, since:

* The handling of 'scpdata' is changed, and with that the way how kernel cmd-line parameters are extracted from the NVMe IPL block, that is passed by the firmware to the kernel at boot time.

* If broken such a hand over will not work for NVMe anymore - which wouldn't be a very big problem for now, since booting w/o still works fine (as it does today).

* But in worst case it could break the hand over of cmd-line parameters for FCP devices, since some code is shared or even harm ipl in general.

* The patch is upstream accepted (with v5.10-rc1 - as of today in linux-next) and a patched groovy kernel was build and shared for further testing.

* All modifications are limited to the s390x architecture and there again to the unique way of booting aka IPL (s390/boot/ipl*).

[Other]

* This is in preparation for getting IPL (boot) from NVMe device support on s390x backported to focal (for hardware enablement reasons).

* Without having this patch in groovy, one may face a regression during upgrade from groovy to focal.

* Since it's planned to have the hirsute kernel on 5.1x, it will have this patch sooner or later.

* However, since the today's hirsute kernel is just based on groovy, I've added hirsute to the SRU.

Related branches

bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-188732 severity-high targetmilestone-inin2004
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
importance: Undecided → High
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
Changed in linux (Ubuntu):
assignee: Skipper Bug Screeners (skipper-screen-team) → Frank Heimes (fheimes)
summary: - [20.04 FEAT|Support of NVMe IPL with Ubuntu 20.04
+ [20.04 FEAT]Support of NVMe IPL with Ubuntu 20.04 (kernel+s390-tools)
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: New → Incomplete
Revision history for this message
Frank Heimes (fheimes) wrote : Re: [20.04 FEAT]Support of NVMe IPL with Ubuntu 20.04 (kernel+s390-tools)

The s390x NVMe kernel support went upstream with kernel 5.8 (LP 1886792), so it's already in groovy (and with that in hirsute).

The s390-tools 2.13 and 2.14 came with NMVe IPL support (for zipl), hence this was done for groovy (and with that for hirsute) based on LP 1884721.

Hence I marked groovy and hirsute as already fix released.

Changed in linux (Ubuntu Groovy):
status: New → Fix Released
Changed in linux (Ubuntu Hirsute):
status: New → Fix Released
Changed in s390-tools (Ubuntu Groovy):
status: New → Fix Released
Changed in s390-tools (Ubuntu Hirsute):
status: New → Fix Released
Steve Langasek (vorlon)
tags: added: fr-890
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2020-11-02 03:00 EDT-------
Kernel git-commits
3737e8ee4f2f s390: nvme ipl
23a457b8d57d s390: nvme reipl
d9f12e48d08e s390/ipl: support NVMe IPL kernel parameters

s390-tools git.commits

commit 1b65b23b4398 ("zipl: Support nvme devices") (went in v2.13.0)
(https://github.com/ibm-s390-tools/s390-tools/commit/1b65b23b43985cb8a1da2ef399ec6def31bbcc69)

commit 0472b5ea5c97 ("ipl-tools: Add nvme device support to
lsreipl/chreipl") (went in v2.14.0)
(https://github.com/ibm-s390-tools/s390-tools/commit/
0472b5ea5c97f2c59a938deebe53b7f27e8a9a32)

commit 71b36d17f019 ("zipl: Fix NVMe partition and base device detection")
(went in v2.14.0; not sure if this is required)
(https://github.com/ibm-s390-tools/s390-tools/commit/
71b36d17f019c9e2cf218351520d5b55a6c2d479)

Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: Incomplete → Triaged
Changed in s390-tools (Ubuntu Focal):
assignee: nobody → Canonical Foundations Team (canonical-foundations)
Changed in linux (Ubuntu Focal):
assignee: nobody → Frank Heimes (fheimes)
Changed in linux (Ubuntu Hirsute):
assignee: Frank Heimes (fheimes) → nobody
Revision history for this message
Frank Heimes (fheimes) wrote : Re: [20.04 FEAT]Support of NVMe IPL with Ubuntu 20.04 (kernel+s390-tools)

I'm changing the kernel groovy entry back to triaged, since I noticed (during verification of the recently communicated kernel commits, that one is brand new, and just arrived in 5.10-rc1, hence is not in groovy master-next.

Changed in linux (Ubuntu Groovy):
status: Fix Released → Triaged
Revision history for this message
Frank Heimes (fheimes) wrote :

Commit d9f12e48d08e "s390/ipl: support NVMe IPL kernel parameters" cherry picks cleanly on groovy master-next, and I've build a patched groovy kernel that can be used for further testing:
https://people.canonical.com/~fheimes/lp1902179/groovy/

However, cherry picking from focal was not successful.
Commit 23a457b8d57d "s390: nvme reipl" does not apply cleanly on focal master-next (but the other two kernel commits do).

Frank Heimes (fheimes)
summary: - [20.04 FEAT]Support of NVMe IPL with Ubuntu 20.04 (kernel+s390-tools)
+ [20.04 FEAT] Support/enhancement of NVMe IPL
Revision history for this message
Frank Heimes (fheimes) wrote :

Kernel SRU request submitted for groovy:
https://lists.ubuntu.com/archives/kernel-team/2020-November/thread.html#114477
changing status to 'In Progress' for groovy.

Changed in linux (Ubuntu Groovy):
status: Triaged → In Progress
description: updated
Revision history for this message
bugproxy (bugproxy) wrote : nvme-ipl kernel patch 0001 of 0003 for focal/master-next

------- Comment (attachment only) From <email address hidden> 2020-11-02 12:08 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : nvme-ipl kernel patch 0002 of 0003 for focal/master-next

------- Comment (attachment only) From <email address hidden> 2020-11-02 12:08 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : nvme-ipl kernel patch 0003 of 0003 for focal/master-next

------- Comment (attachment only) From <email address hidden> 2020-11-02 12:09 EDT-------

Revision history for this message
Frank Heimes (fheimes) wrote :

I've created a patched focal kernel (cherry pick of 3737e8ee4f2f "s390: nvme ipl", backport of 23a457b8d57d "s390: nvme reipl" and again cherry pick of d9f12e48d08e "s390/ipl: support NVMe IPL kernel parameters", based on master-next) that can now be used for further testing:
https://people.canonical.com/~fheimes/lp1902179/focal/

Frank Heimes (fheimes)
information type: Private → Public
Frank Heimes (fheimes)
Changed in linux (Ubuntu Hirsute):
status: Fix Released → In Progress
Stefan Bader (smb)
Changed in linux (Ubuntu Hirsute):
importance: Undecided → Medium
Changed in linux (Ubuntu Groovy):
importance: Undecided → Medium
Frank Heimes (fheimes)
description: updated
Revision history for this message
Frank Heimes (fheimes) wrote :

Kernel SRU request submitted for focal:
https://lists.ubuntu.com/archives/kernel-team/2020-November/thread.html#114528
changing status to 'In Progress' for focal.

Changed in linux (Ubuntu Focal):
status: New → In Progress
assignee: Frank Heimes (fheimes) → nobody
importance: Undecided → Medium
Changed in ubuntu-z-systems:
status: Triaged → In Progress
Ian May (ian-may)
Changed in linux (Ubuntu Groovy):
status: In Progress → Fix Committed
Ian May (ian-may)
Changed in linux (Ubuntu Focal):
status: In Progress → Fix Committed
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2020-11-05 12:23 EDT-------
We were finally able to cobble together a system to perform some testing on the provided Ubuntu test kernel.
Here is what was tested (using Debian s390-tools 2.14):

- Using zipl to install the bootloader to NVMe works as expected.
- The kernel IPLs and the IPL type is properly identified via lsreipl
- chreipl/reboot to the same NVMe device works as expected.
- chreipl/reboot to a DASD works as expected.

What could not easily be tested in our environment:
- chreipl/reboot from the DASD back to the NVMe device.
- chreipl involving an FCP device.

Though, knowing what has changed in the code, and knowing how reipl works, I have no reason to suspect that either of these test cases could cause any issues. I would be confident to give a thumbs up here.

Revision history for this message
Frank Heimes (fheimes) wrote :

That a reasonable test with a good coverage - thx for that!

Changed in s390-tools (Ubuntu Focal):
importance: Undecided → Wishlist
Revision history for this message
Frank Heimes (fheimes) wrote :

The following two commits commit 1b65b23b4398 and 0472b5ea5c97 do not apply cleanly to:
s390-tools | 2.12.0-0ubuntu3.1
and some of the conflicts don't seem to be obvious to solve.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-11-16 11:16 EDT-------
Canonical,

are there public s390-tools repositories with the Ubuntu versions available
that we can mirror internally and which we could then use for potential
backport work (such as this feature seems to require)?

Please let me know so that we can provide you with backport patches
that apply cleanly on your version of s390-tools (v2.12.0).

Revision history for this message
Frank Heimes (fheimes) wrote :

Hi Jan, yes (almost) all Ubuntu packages are hosted nowadays at our Launchpad git (at least all packages in main, incl. s390-tools).
Please see: https://code.launchpad.net/ubuntu/+source/s390-tools
If you are logged in with a Launchpad account, you will directly see at the top a comd-line that allows to clone the package sources, like:
" Get this repository:
  git clone https://git.launchpad.net/ubuntu/+source/s390-tools "
If not logged in you'll see a more generic page, but again listing the URLs that can be used for cloning, here:
git://git.launchpad.net/ubuntu/+source/s390-tools
git+ssh://git.launchpad.net/ubuntu/+source/s390-tools
https://git.launchpad.net/ubuntu/+source/s390-tools

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-11-16 14:19 EDT-------
(In reply to comment #60)
> Hi Jan, yes (almost) all Ubuntu packages are hosted nowadays at our
> Launchpad git (at least all packages in main, incl. s390-tools).
> Please see: https://code.launchpad.net/ubuntu/+source/s390-tools
> If you are logged in with a Launchpad account, you will directly see at the
> top a comd-line that allows to clone the package sources, like:
> " Get this repository:
> git clone https://git.launchpad.net/ubuntu/+source/s390-tools "
> If not logged in you'll see a more generic page, but again listing the URLs
> that can be used for cloning, here:
> git://git.launchpad.net/ubuntu/+source/s390-tools
> git+ssh://git.launchpad.net/ubuntu/+source/s390-tools
> https://git.launchpad.net/ubuntu/+source/s390-tools

I've cloned the repo. Which branch should I re-base the patches on? I was able to find the following likely candidates:

origin/ubuntu/focal
origin/ubuntu/focal-devel
origin/ubuntu/focal-proposed
origin/ubuntu/focal-updates

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
tags: added: verification-needed-groovy
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-groovy' to 'verification-done-groovy'. If the problem still exists, change the tag 'verification-needed-groovy' to 'verification-failed-groovy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Frank Heimes (fheimes)
Changed in linux (Ubuntu Hirsute):
status: In Progress → Fix Committed
bugproxy (bugproxy)
tags: removed: verification-needed-focal verification-needed-groovy
Frank Heimes (fheimes)
tags: added: verification-done-focal verification-done-groovy
bugproxy (bugproxy)
tags: removed: verification-done-focal verification-done-groovy
bugproxy (bugproxy)
tags: added: verification-done-focal verification-done-groovy
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-11-17 16:23 EDT-------
I've attached the three backported s390-tools patches for focal-proposed. They received a basic compile test and a very quick "it runs" sanity check.

If we need more testing at this stage, let me know and I'll see about getting it tested properly. This may take a day or two as the environment is very difficult to set up without installer support for NVMe devices.

Revision history for this message
bugproxy (bugproxy) wrote : nvme-ipl s390-tools patch 0001 of 0003 for focal-proposed

------- Comment (attachment only) From <email address hidden> 2020-11-17 16:20 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : nvme-ipl s390-tools patch 0002 of 0003 for focal-proposed

------- Comment (attachment only) From <email address hidden> 2020-11-17 16:21 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : nvme-ipl s390-tools patch 0003 of 0003 for focal-proposed

------- Comment (attachment only) From <email address hidden> 2020-11-17 16:21 EDT-------

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (78.9 KiB)

This bug was fixed in the package linux - 5.4.0-56.62

---------------
linux (5.4.0-56.62) focal; urgency=medium

  * focal/linux: 5.4.0-56.62 -proposed tracker (LP: #1905300)

  * CVE-2020-4788
    - selftests/powerpc: rfi_flush: disable entry flush if present
    - powerpc/64s: flush L1D on kernel entry
    - powerpc/64s: flush L1D after user accesses
    - selftests/powerpc: entry flush test

linux (5.4.0-55.61) focal; urgency=medium

  * focal/linux: 5.4.0-55.61 -proposed tracker (LP: #1903175)

  * Update kernel packaging to support forward porting kernels (LP: #1902957)
    - [Debian] Update for leader included in BACKPORT_SUFFIX

  * Avoid double newline when running insertchanges (LP: #1903293)
    - [Packaging] insertchanges: avoid double newline

  * EFI: Fails when BootCurrent entry does not exist (LP: #1899993)
    - efivarfs: Replace invalid slashes with exclamation marks in dentries.

  * CVE-2020-14351
    - perf/core: Fix race in the perf_mmap_close() function

  * raid10: Block discard is very slow, causing severe delays for mkfs and
    fstrim operations (LP: #1896578)
    - md: add md_submit_discard_bio() for submitting discard bio
    - md/raid10: extend r10bio devs to raid disks
    - md/raid10: pull codes that wait for blocked dev into one function
    - md/raid10: improve raid10 discard request
    - md/raid10: improve discard request for far layout
    - dm raid: fix discard limits for raid1 and raid10
    - dm raid: remove unnecessary discard limits for raid10

  * Bionic: btrfs: kernel BUG at /build/linux-
    eTBZpZ/linux-4.15.0/fs/btrfs/ctree.c:3233! (LP: #1902254)
    - btrfs: drop unnecessary offset_in_page in extent buffer helpers
    - btrfs: extent_io: do extra check for extent buffer read write functions
    - btrfs: extent-tree: kill BUG_ON() in __btrfs_free_extent()
    - btrfs: extent-tree: kill the BUG_ON() in insert_inline_extent_backref()
    - btrfs: ctree: check key order before merging tree blocks

  * Ethernet no link lights after reboot (Intel i225-v 2.5G) (LP: #1902578)
    - igc: Add PHY power management control

  * Undetected Data corruption in MPI workloads that use VSX for reductions on
    POWER9 DD2.1 systems (LP: #1902694)
    - powerpc: Fix undetected data corruption with P9N DD2.1 VSX CI load emulation
    - selftests/powerpc: Make alignment handler test P9N DD2.1 vector CI load
      workaround

  * [20.04 FEAT] Support/enhancement of NVMe IPL (LP: #1902179)
    - s390: nvme ipl
    - s390: nvme reipl
    - s390/ipl: support NVMe IPL kernel parameters

  * uvcvideo: add mapping for HEVC payloads (LP: #1895803)
    - media: uvcvideo: Add mapping for HEVC payloads

  * Focal update: v5.4.73 upstream stable release (LP: #1902115)
    - ibmveth: Switch order of ibmveth_helper calls.
    - ibmveth: Identify ingress large send packets.
    - ipv4: Restore flowi4_oif update before call to xfrm_lookup_route
    - mlx4: handle non-napi callers to napi_poll
    - net: fec: Fix phy_device lookup for phy_reset_after_clk_enable()
    - net: fec: Fix PHY init after phy_reset_after_clk_enable()
    - net: fix pos incrementment in ipv6_route_seq_next
    - net/smc: fix valid DMBE buffer sizes
    - net...

Changed in linux (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2020-12-01 08:05 EDT-------
(In reply to comment #74)
> @Jason, seems to be some build failures. Can you please have a look.

It looks like there are two versions of this patch applied:
[PATCH] s390-tools: zipl: Fix NVMe partition and base device detection

Not sure how this happened. Here is the compiler output:

disk.c:174:12: error: redefinition of ?blkext_get_partnum?
174 | static int blkext_get_partnum(dev_t dev)
| ^~~~~~~~~~~~~~~~~~
disk.c:92:12: note: previous definition of ?blkext_get_partnum? was here
92 | static int blkext_get_partnum(dev_t dev)
| ^~~~~~~~~~~~~~~~~~

disk.c:199:12: error: redefinition of ?blkext_is_base_device?
199 | static int blkext_is_base_device(dev_t dev)
| ^~~~~~~~~~~~~~~~~~~~~
disk.c:117:12: note: previous definition of ?blkext_is_base_device? was here
117 | static int blkext_is_base_device(dev_t dev)
| ^~~~~~~~~~~~~~~~~~~~~

disk.c:213:12: error: redefinition of ?blkext_get_base_dev?
213 | static int blkext_get_base_dev(dev_t dev, dev_t *base_dev)
| ^~~~~~~~~~~~~~~~~~~
disk.c:131:12: note: previous definition of ?blkext_get_base_dev? was here
131 | static int blkext_get_base_dev(dev_t dev, dev_t *base_dev)
| ^~~~~~~~~~~~~~~~~~~

Can someone on Canonical's end manually verify this,and perhaps exclude the
above patch from their build if it seems to already be present?

I created the backport patches on top of the following repo/branch:
git://git.launchpad.net/ubuntu/+source/s390-tools
focal-proposed

And on this branch, this patch was missing, which is why I included it here.
Was I using the wrong repo/branch for this backport?

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (50.5 KiB)

This bug was fixed in the package linux - 5.8.0-31.33

---------------
linux (5.8.0-31.33) groovy; urgency=medium

  * groovy/linux: 5.8.0-31.33 -proposed tracker (LP: #1905299)

  * Groovy 5.8 kernel hangs on boot on CPUs with eLLC (LP: #1903397)
    - drm/i915: Mark ininitial fb obj as WT on eLLC machines to avoid rcu lockup
      during fbdev init

  * CVE-2020-4788
    - selftests/powerpc: rfi_flush: disable entry flush if present
    - powerpc/64s: flush L1D on kernel entry
    - powerpc/64s: flush L1D after user accesses
    - selftests/powerpc: entry flush test

linux (5.8.0-30.32) groovy; urgency=medium

  * groovy/linux: 5.8.0-30.32 -proposed tracker (LP: #1903194)

  * Update kernel packaging to support forward porting kernels (LP: #1902957)
    - [Debian] Update for leader included in BACKPORT_SUFFIX

  * Avoid double newline when running insertchanges (LP: #1903293)
    - [Packaging] insertchanges: avoid double newline

  * EFI: Fails when BootCurrent entry does not exist (LP: #1899993)
    - efivarfs: Replace invalid slashes with exclamation marks in dentries.

  * raid10: Block discard is very slow, causing severe delays for mkfs and
    fstrim operations (LP: #1896578)
    - md: add md_submit_discard_bio() for submitting discard bio
    - md/raid10: extend r10bio devs to raid disks
    - md/raid10: pull codes that wait for blocked dev into one function
    - md/raid10: improve raid10 discard request
    - md/raid10: improve discard request for far layout
    - dm raid: fix discard limits for raid1 and raid10
    - dm raid: remove unnecessary discard limits for raid10

  * Bionic: btrfs: kernel BUG at /build/linux-
    eTBZpZ/linux-4.15.0/fs/btrfs/ctree.c:3233! (LP: #1902254)
    - btrfs: extent_io: do extra check for extent buffer read write functions
    - btrfs: extent-tree: kill BUG_ON() in __btrfs_free_extent()
    - btrfs: extent-tree: kill the BUG_ON() in insert_inline_extent_backref()
    - btrfs: ctree: check key order before merging tree blocks

  * Tiger Lake PMC core driver fixes (LP: #1899883)
    - platform/x86: intel_pmc_core: update TGL's LPM0 reg bit map name
    - platform/x86: intel_pmc_core: fix bound check in pmc_core_mphy_pg_show()
    - platform/x86: pmc_core: Use descriptive names for LPM registers
    - platform/x86: intel_pmc_core: Fix TigerLake power gating status map
    - platform/x86: intel_pmc_core: Fix the slp_s0 counter displayed value

  * drm/i915/dp_mst - System would hang during the boot up. (LP: #1902469)
    - Revert "UBUNTU: SAUCE: drm/i915/display: Fix null deref in
      intel_psr_atomic_check()"
    - drm/i915: Fix encoder lookup during PSR atomic check

  * Undetected Data corruption in MPI workloads that use VSX for reductions on
    POWER9 DD2.1 systems (LP: #1902694)
    - powerpc: Fix undetected data corruption with P9N DD2.1 VSX CI load emulation
    - selftests/powerpc: Make alignment handler test P9N DD2.1 vector CI load
      workaround

  * [20.04 FEAT] Support/enhancement of NVMe IPL (LP: #1902179)
    - s390/ipl: support NVMe IPL kernel parameters

  * uvcvideo: add mapping for HEVC payloads (LP: #1895803)
    - media: uvcvideo: Add mapping for HEVC payloads

  * risc-v 5.8 ...

Changed in linux (Ubuntu Groovy):
status: Fix Committed → Fix Released
Changed in s390-tools (Ubuntu Focal):
milestone: none → ubuntu-20.04.2
Revision history for this message
Frank Heimes (fheimes) wrote :

I've now created a PPA with a patched focal s390-tools package, but only added:
1b65b23b4398 "zipl: Support nvme devices"
and
0472b5ea5c97 "ipl-tools: Add nvme device support to lsreipl/chreipl"
but NOT
71b36d17f019 "zipl: Fix NVMe partition and base device detection".

It's available for testing over here:
https://launchpad.net/~fheimes/+archive/ubuntu/lp1902179

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 5.8.0-36.40+21.04.1

---------------
linux (5.8.0-36.40+21.04.1) hirsute; urgency=medium

  * Packaging resync (LP: #1786013)
    - update dkms package versions

  [ Ubuntu: 5.8.0-36.40 ]

  * debian/scripts/file-downloader does not handle positive failures correctly
    (LP: #1878897)
    - [Packaging] file-downloader not handling positive failures correctly

  [ Ubuntu: 5.8.0-35.39 ]

  * Packaging resync (LP: #1786013)
    - update dkms package versions
  * CVE-2021-1052 // CVE-2021-1053
    - [Packaging] NVIDIA -- Add the NVIDIA 460 driver

 -- Kleber Sacilotto de Souza <email address hidden> Thu, 07 Jan 2021 11:57:30 +0100

Changed in linux (Ubuntu Hirsute):
status: Fix Committed → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2021-01-14 09:00 EDT-------
chreipl currently fails on Ubuntu 20.04 when given a device node (e.g. chreipl /dev/nvme0n1p1). This is because chreipl cannot find the expected sysfs directory structure. We are investigating the best way to fix this. If the Ubuntu installer can still complete an install to NVMe devices without this particular invocation of chreipl then we recommend going forward with the code as-is and we can provide a bug fix at a later date. I suspect this should be fine, as chreipl should only affect the reboot following OS installation.

It should be noted that chreipl works when given the devices function id (FID) parameter instead of the device node. It only fails when it needs to look up the device's FID given the device node.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Is the NVMe drive with or without nvme-multipath support?
Does it help to disable multipath?

Please see the below pullrequest from Canonical employee in efivar project that explains the issue a bit, and the various paths that the NVMe devices can have.

https://github.com/rhboot/efivar/pull/158

This also affected efibootmgr project for the EFI platforms with newer kernels, as shipped in focal.

You can also try booting with the following kernel cmdline option appended in the param file i.e. `nvme_core.multipath=N` to get the "non-multipath nvme0 nodes in the sysfs".

BTW https://www.ibm.com/support/knowledgecenter/STSLR9_8.3.1/com.ibm.fs9100_831.doc/svc_fc_nvme_multipathconfighosts.html has an excellent summary of nvme multipath situation in various distros.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2021-01-15 08:35 EDT-------
(In reply to comment #82)
> Is the NVMe drive with or without nvme-multipath support?
> Does it help to disable multipath?
>
> Please see the below pullrequest from Canonical employee in efivar project
> that explains the issue a bit, and the various paths that the NVMe devices
> can have.
>
> https://github.com/rhboot/efivar/pull/158
>
> This also affected efibootmgr project for the EFI platforms with newer
> kernels, as shipped in focal.
>
> You can also try booting with the following kernel cmdline option appended
> in the param file i.e. `nvme_core.multipath=N` to get the "non-multipath
> nvme0 nodes in the sysfs".
>
> BTW
> https://www.ibm.com/support/knowledgecenter/STSLR9_8.3.1/com.ibm.fs9100_831.
> doc/svc_fc_nvme_multipathconfighosts.html has an excellent summary of nvme
> multipath situation in various distros.

Yes, multipath is the problem here. When multipath is enabled we have a different sysfs structure. I'll have to work up a fix.

Can you move ahead with the code as it is, and then we can provide a bug fix patch when ready? This chreipl bug should not impede NVMe installation or a normal boot process.

Revision history for this message
Frank Heimes (fheimes) wrote :

Ok, so 'chreipl' is used by the subiquity installer to prepare for the post-install reboot.
This means that a fix for chreipl is indeed to have proper NVMe support in place, incl. installer.

There is now just a little bit of time pressure, since this is relevant for install time, it should not only land in the Ubuntu archives, but also on the ISO image, and 20.04.2 is coming very soon...
So I guess we would need it in the next couple of days to have a little chance to pick it up for focal/20.04.2 - otherwise it might slip to the .3 release (the install time support).

Revision history for this message
Frank Heimes (fheimes) wrote :

After some additional discussions the situation is as following:
With picking the current s390-tools fix (as it was used so far here) we would be on the same level (regarding NVMe support) than groovy.
The NVMe testing on groovy was done on a device without multipath.
The issue that cropped up during the testing on focal now is because an NVMe device with multipath was used.
But there is still value of getting the fixes picked up as they are (especially for a certain team).

However, getting NVMe devices supported also the multipath way and by the installer, the fix is of course still needed (chreipl).
But since this fix is not only needed for focal, but at least also for groovy (and probably also for hirsute), I think it's fine to drive this missing bit a later stage, based on a separate ticket and dedicated s390-tools package SRU.

(Please keep in mind that this is all s390x specific!)

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2021-01-22 10:02 EDT-------
Hi Frank,

we've now internally confirmed that setting the "nvme-core.multipath=0" Kernel
parameter as a workaround makes "chreipl node" succeed with NVMes that otherwise break the current chreipl due to its lack of multipath support.

Do you think it would be possible to set this Kernel parameter for the installer?
As far as I understand this option doesn't necessarily have to be set for the installed system since the IPL information is stored in firmware as FID
and namespace so only a manual "chreipl node /dev/nvmeXYZ" from
the installed system would risk breaking the IPL information and there
the user should get a crash and can instead use "chreipl nvme -i <fid> -s <nsid>". Normal "zipl" as done by a kernel update should not be a problem, in fact I've got an Ubuntu 20.04 installed on an NVMe (via KVM ;-) /dev/vda on an NVMe) here and kernel updates worked fine.

Alternatively there is also the config option CONFIG_NVME_MULTIPATH if
we would set that to "n" for Z that should also work.

Either way we'll of course work on a proper fix for chreipl.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Ubuntu supports NVMe Multipath on all architectures in Focal. I know that some other distributions do not.

Setting nvme-core.multipath=0 for s390x isos only, or changing kernel config to disable multipath to "n" on s390x only, would mean regression on s390x as compared with amd64/arm64/ppc64le.

That would be suboptimal.

Do you have no plans to support multipath NVMe and are customers who buy expensive multipath capable NVMe drives will not be able to use multipath capability with them? Or for example is multipath handled separately on Z with NVMe? (ie. HMC mediated)

My preference would be to see a chreipl fix upstream in s390-tools to add support for multipath nvme drives, and SRU that too.

Revision history for this message
Frank Heimes (fheimes) wrote :

So for now I think we should pick up the patches as they are,
since they already provide value and we will be at a similar level like on groovy,
mention the current situation in the release notes and pointing to "nvme-core.multipath=0"
and SRUing a chreipl fix as soon as it becomes available (probably for H, G and F then).

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2021-01-22 10:39 EDT-------
(In reply to comment #87)
> Ubuntu supports NVMe Multipath on all architectures in Focal. I know that
> some other distributions do not.
>
> Setting nvme-core.multipath=0 for s390x isos only, or changing kernel config
> to disable multipath to "n" on s390x only, would mean regression on s390x as
> compared with amd64/arm64/ppc64le.
>
> That would be suboptimal.
>
> Do you have no plans to support multipath NVMe and are customers who buy
> expensive multipath capable NVMe drives will not be able to use multipath
> capability with them? Or for example is multipath handled separately on Z
> with NVMe? (ie. HMC mediated)
>
> My preference would be to see a chreipl fix upstream in s390-tools to add
> support for multipath nvme drives, and SRU that too.

Oh, we absolutely want to support NVMe multipathing and I don't see
a general reason to disable it in the config as I don't know
about any other issues with it. This is only about working
around the current "chreipl node /dev/nvmeXYZ" bug which as far as I
understand breaks the final reboot from the installer to the
installed system. We're already working on a chreipl fix but as far
as I understand the installer ISOs are released only for .x releases
and running the installer with multipathing off should only impact
the installer's performance or am I missing something? So
this is definitely meant as a temporary workaround.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2021-01-22 11:01 EDT-------
(In reply to comment #89)
> So for now I think we should pick up the patches as they are,
> since they already provide value and we will be at a similar level like on
> groovy,
> mention the current situation in the release notes and pointing to
> "nvme-core.multipath=0"
> and SRUing a chreipl fix as soon as it becomes available (probably for H, G
> and F then).

Sounds good, thank you!

Revision history for this message
Frank Heimes (fheimes) wrote :

MP is open

Revision history for this message
Frank Heimes (fheimes) wrote :
Changed in s390-tools (Ubuntu Focal):
status: New → Triaged
Frank Heimes (fheimes)
Changed in s390-tools (Ubuntu Focal):
status: Triaged → In Progress
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello bugproxy, or anyone else affected,

Accepted s390-tools into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools/2.12.0-0ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in s390-tools (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-focal
removed: verification-done-focal
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2021-01-27 08:03 EDT-------
ok, same behavior like with the PPA package

Revision history for this message
Frank Heimes (fheimes) wrote :

Ok thx - adjusting the tag(s) accordingly ...

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package s390-tools - 2.12.0-0ubuntu3.2

---------------
s390-tools (2.12.0-0ubuntu3.2) focal; urgency=medium

  * debian/patches/s390-tools-sru-lp1903984-focal.patch
    zcryptstats: Fix handling of partial results with many domains
    Thanks to Ingo Franzki (LP: #1903984)
  * debian/patches/s390-tools-sru-lp1908371-focal.patch:
    zipl command isn't working correctly in combination with the -M
    (respectively --mvdump) option.
    cherry-picking 4 commits from s390-tools v2.15.1 to v2.14
    Thanks to Stefan Haberland and Sven Schnelle (LP: #1908371)
  * debian/patches/s390-tools-sru-lp1902179-focal.patch:
    Enabling IPL (boot) from NVMe devices by adding 2 upstream commits
    from s390-tools v2.13/2.14 as backports to this v2.12 base.
    Thanks to Jason J. Herne (LP: #1902179)

 -- Frank Heimes <email address hidden> Fri, 22 Jan 2021 15:52:19 +0100

Changed in s390-tools (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for s390-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: In Progress → Fix Released
Revision history for this message
Frank Heimes (fheimes) wrote :

The following entry will be added to the 20.04.2 Release Notes (about LP 1902179):
"
Ubuntu Server 20.04.2 on IBM Z and LinuxONE (s390x) is currently only able to boot/IPL from NVMe devices that are not represented as multipath devices. In case a NVMe device is used on s390x that is multipath capable, the multipath option needs to be switched off with the help of this kernel parameter: 'nvme-core.multipath=0', otherwise 'chreipl' will not work properly.
"

Revision history for this message
Frank Heimes (fheimes) wrote :

A known-issue entry was added to the release notes: https://wiki.ubuntu.com/FocalFossa/ReleaseNotes
Hence closing this ticket.

Changed in ubuntu-release-notes:
status: New → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2021-02-25 07:33 EDT-------
IBM Bugzilla status->closed Fix Released with 20.04

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.