Ubuntu
linux package

zcrypt DD: Toleration for new IBM Z Crypto Hardware - (Backport to Ubuntu 20.04)

Bug #1954680 reported by bugproxy
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Fix Released
High
Skipper Bug Screeners
linux (Ubuntu)
Fix Released
High
Frank Heimes
Focal
Fix Released
Undecided
Canonical Kernel Team
Hirsute
Invalid
Undecided
Canonical Kernel Team
Impish
Fix Released
Undecided
Frank Heimes
Jammy
Fix Released
High
Frank Heimes

Bug Description

SRU Justification:
==================

[Impact]

 * CEX8 hardware CryptoExpress adapter shall support quantum-safe crypto
   and therefore require nowadays message sizes > 12kB.

 * This change here is mainly required to support EP11 responses to admin requests at zNext
   which due to QS certificates can grow larger than 12kB.

 * It's to cover a minimal patch to provide toleration support for this feature
   which shall be back-ported to all distribution releases in service at zNext

 * This SRU requests belongs to the hardware enablement case.

[Fix]

 * bd39654a2282 bd39654a2282c1a51c044575a6bc00d641d5dfd1 "s390/AP: support new dynamic AP bus size limit"

[Test Plan]

 * An Ubuntu 20.04 (respectively 21.04) LPAR or z/VM guest is needed
   that has access to at least one online crypto domain.

 * Ideally using a CEX8 adapter (but can be too early to get one).

 * Then get the patched kernel installed (see PPA below).

 * And look for the /sys/devices/ap/cardxx/max_msg_size sysfs attributes.

 * On top IBM has some more in-depth zcrypt tests (see also LP#1933805).

[Where problems could occur]

 * First of all the modification are limited to:
   the zcrypt driver ("/drivers/s390/crypto/ap_*.*" and
   "/drivers/s390/crypto/zcrypt_*.*")
   hence are s390x platform specific and crypto specific and
   should even affect CEX8 cards only.
   So in case anything fails, it's limited to s390x cryptography,
   which usually allows sw fall-backs.

 * The function signature of ap_queue_info and ap_test_queue got modified,
   which may lead to issues if called with the old signatures,
   but that would be identified by the test compile already.

 * Some minor new structures like 'info', 'ml' got introduced,
   but are properly declared and initialized.

 * The way ap_queue_info and ap_card_create get filled and used was changed,
   therefore in some code areas slightly different data might be expected,
   if not properly adapted to the new way.
   But a verification test will prove this.

 * The actual msg length is now handled based on bufsize rather than len
   and with that zq is calculated in a different way (using zcrypt_queue_alloc)
   which may cause some side effects if not properly (alloc)
   or not thoroughly done.

 * in _zcrypt_send_cprb and _zcrypt_send_ep11_cprb some additional calculations
   and checks (if-stmts) were introduced, but they look sane.

 * New code to identify older cards got added, since message sizes > 12kB
   are supported by CEX8 and higher only.
   The dispatcher responsible for choosing the right card and queue is aware
   of the individual card AP bus message limit.
   But already at the user space tools it should be ensured that the right
   card is used.

 * Nevertheless, the patch is not small, hence s390x hardware crypto
   zcrypt driver needs to be properly re-tested.

[Other Info]

 * The above commit/patch is upstream accepted with 5.14.

 * Impish's Kernel 5.13 was already patched, based on LP#1933805.

 * With that there is already a certain level of testing that was done
   based on Impish (since the zcrypt driver is largely the same now with
   these cherry-picks).

 * Hence the SRU is only needed for Focal
   and Hirsute (just to avoid regressions on upgrades).
__________

Toleration support for new IBM Z crypto hardware - Backport to focal (20.04)

Patch from kernel 5.14:

Summary: s390/AP: support new dynamic AP bus size limit
Description: This patch provides support for new dynamic AP bus
             message limit with the existing zcrypt device driver
             and AP bus core code. There is support for a new
             field 'ml' from the TAPQ query which indicates the
             per card AP bus message size limit. This TAPQ
             improvement will come with an updated firmware and
             this patch exploits this new field and thus makes
             it possible to send/receive AP messages greater
             than the current limit of 12K.
Upstream-ID: bd39654a2282c1a51c044575a6bc00d641d5dfd1

See original description
bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-195656 severity-high targetmilestone-inin2004
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
Revision history for this message
Frank Heimes (fheimes) wrote (last edit ):

Incl. in jammy (since jammy is 5.15), hence updating 'affects jammy' entry to 'Fix Released'.

Changed in ubuntu-z-systems:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
Changed in linux (Ubuntu):
assignee: Skipper Bug Screeners (skipper-screen-team) → Frank Heimes (fheimes)
Changed in linux (Ubuntu Jammy):
status: New → Fix Released
importance: Undecided → High
Changed in ubuntu-z-systems:
importance: Undecided → High
Frank Heimes (fheimes)
description: updated
Revision history for this message
Frank Heimes (fheimes) wrote :

commit "s390/AP: support new dynamic AP bus size limit" is already in impish as d5428ba5ba6b due to LP#1933805 - "[21.10 FEAT] zcrypt DD: CEX8 toleration", hence updating 'affects impish# entry to 'Fix Released'.

Changed in linux (Ubuntu Impish):
status: New → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

Incl. in jammy (since jammy is 5.15), hence updating 'affects jammy' entry to 'Fix released'.

Revision history for this message
Frank Heimes (fheimes) wrote :

I'm glad to see that cherry-picking bd39654a2282 "s390/AP: support new dynamic AP bus size limit" to hirsute and focal applies cleanly.

Changed in linux (Ubuntu Hirsute):
status: New → Confirmed
Changed in linux (Ubuntu Focal):
status: New → Confirmed
Changed in ubuntu-z-systems:
status: New → Confirmed
Revision history for this message
Frank Heimes (fheimes) wrote :

patched test kernels are currently build at: https://launchpad.net/~fheimes/+archive/ubuntu/lp1954680
and will be available in a few hours.

Frank Heimes (fheimes)
description: updated
Revision history for this message
Frank Heimes (fheimes) wrote :

SRU request submitted to the Ubuntu kernel team mailing list for focal and hirsute:
https://lists.ubuntu.com/archives/kernel-team/2021-December/thread.html#126511
Changing status to 'In Progress' for focal and hirsute.

Changed in linux (Ubuntu Hirsute):
status: Confirmed → In Progress
Changed in linux (Ubuntu Impish):
assignee: nobody → Frank Heimes (fheimes)
Changed in linux (Ubuntu Focal):
status: Confirmed → In Progress
Changed in linux (Ubuntu Hirsute):
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
Changed in linux (Ubuntu Focal):
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
Changed in ubuntu-z-systems:
status: Confirmed → In Progress
Frank Heimes (fheimes)
information type: Private → Public
Kleber Sacilotto de Souza (kleber-souza)
Changed in linux (Ubuntu Hirsute):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Focal):
status: In Progress → Fix Committed
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.11.0-47.52 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-hirsute' to 'verification-done-hirsute'. If the problem still exists, change the tag 'verification-needed-hirsute' to 'verification-failed-hirsute'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-hirsute
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2022-01-13 09:38 EDT-------
I am about to set up a system and will try to verify asap.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2022-01-13 11:17 EDT-------
unable to install a system with ubuntu 20.04 or whatever ... looks like jet another 'improvement' in your installation system. Maybe we should start selling teddy bears instead of claiming to be a company able to provide 'software solutions'.

Revision history for this message
Frank Heimes (fheimes) wrote :

Hi Harald, I'm sorry to read that you have problems with installing Ubuntu 20.04.
But let's be constructive please and let us know more details, so that we can look at solving the situation.

Which Ubuntu image did you used for the installation?
The very latest 20.04 ISO image is always available from here:
https://cdimage.ubuntu.com/releases/20.04/release/
(There are also ready to use Cloud images available).

Please notice that the way to install 20.04 differs from the previous Ubuntu releases,
see more details here: https://ubuntu.com/server/docs/installation

Where did you want to install: LPAR, z/VM or KVM?

Have you tried an interactive or non-interactive (autoinstall)?

And especially: Can you share the logs?
The interactive installer is a live installer, and even if you run into a problem, you can usually enter the installer shell (which give you access to a fully-fledged Ubuntu/Linux system).
Ideally share the entire /var/log/installer folder and (in case not empty) /var/crash.

Revision history for this message
Frank Heimes (fheimes) wrote :

And since the installation issue is independent from this particular LP bug here, I suggest to open a separate LP for it.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2022-01-14 03:50 EDT-------
Sorry for this confusion about my last comment.
The installation issue is clearly not related to Cannonical and I apologize for this.

However, I finally managed to have a Ubuntu 20.04 running on my system with some tricks. Then I picked the kernel from here: https://launchpad.net/ubuntu/focal/s390x/linux-image-5.4.0-94-generic/5.4.0-94.106 as the bug suggests, installed and rebooted. So now I am running kernel:

root@a35lp66:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

root@a35lp66:~# uname -a
Linux a35lp66 5.4.0-94-generic #106-Ubuntu SMP Thu Jan 6 23:57:26 UTC 2022 s390x s390x s390x GNU/Linux

But I am missing the sysfs attributes
/sys/devices/ap/cardxx/max_msg_size
for all crypto cards. So is patch
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bd39654a2282c1a51c044575a6bc00d641d5dfd1
really integrated into this kernel ???

Revision history for this message
Frank Heimes (fheimes) wrote :

Hi Harald, glad that you now have a test-system at hand (and thx for the clarification).

Well, so far this LP tickets asks (in LP comment #8: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1954680/comments/7) for verification on hirsute.
focal/20.04 need to be verified soon too, but the 'ubuntu-kernel-bot' did not left a comment regarding focal verification, yet.

I just checked if the code landed in the focal master-next tree:
$ git log --oneline --grep "s390/AP: support new dynamic AP bus size limit"
f65525ab6604 s390/AP: support new dynamic AP bus size limit
and it is and it's tagged to be included in kernel 'Ubuntu-5.4.0-95', which is the next upcoming kernel:
$ git tag --contains f65525ab6604
Ubuntu-5.4.0-95.107

But 'Ubuntu-5.4.0-95' is still in preparation and not yet ready, hence did not arrived in the archives yet (will btw. first land in -proposed):
$ rmadison -a s390x linux-generic | grep focal
 linux-generic | 5.4.0.26.32 | focal | s390x
 linux-generic | 5.4.0.94.98 | focal-security | s390x
 linux-generic | 5.4.0.94.98 | focal-updates | s390x
(-proposed is not yet listed)

So I apologize, that currently on hirsute/21.04 verification is possible (you may just leave the focal system sitting there for a while ...?!)

For hirsute the kernel 5.11.0-47 is the one that incl. the requested patch (according to comment #7) - and I just double checked the hirsute master-next tree:
$ git log --oneline --grep "s390/AP: support new dynamic AP bus size limit"
c0c6e76a5dd8 s390/AP: support new dynamic AP bus size limit
$ git tag --contains c0c6e76a5dd8
Ubuntu-5.11.0-47.52
$ rmadison -a s390x linux-generic | grep hirsute
 linux-generic | 5.11.0.16.17 | hirsute | s390x
 linux-generic | 5.11.0.46.46 | hirsute-security | s390x
 linux-generic | 5.11.0.46.46 | hirsute-updates | s390x
 linux-generic | 5.11.0.47.47 | hirsute-proposed | s390x
So the hirsute kernel is indeed ready to test from proposed (btw. please ignore the right-most digits of the kernel version, they just incl. build info).

And to get a kernel installed and running from the "-proposed" pocket of the archive, the following is needed on top of a standard installation:
1)
enable proposed with:
"sudo add-apt-repository "deb http://us.ports.ubuntu.com/ubuntu-ports/ $(lsb_release -sc)-proposed main"
(for the src use "sudo add-apt-repository "deb-src http://us.ports.ubuntu.com/ubuntu-ports/ $(lsb_release -sc)-proposed main")
2)
Do an "sudo apt update" (if not automatically triggered) and you will find new packages listed by:
"apt list --upgradable"
3)
You can just install all of them (e.g. with "sudo apt full-upgrade"), or just selectively install an updated package, like the kernel (e.g. "sudo apt install linux-generic").
4)
Reboot to activate the new kernel (and check with 'uname' or 'apt-cache policy linux-generic' after reboot and before testing).

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2022-01-14 05:21 EDT-------
Thanks for this clarification.
For Ubuntu 21.10 I already did the verification (I think it was launchpad 1933805) in September last year.
So I'll wait until something for focal is available.

Thanks

Frank Heimes (fheimes)
tags: added: verification-done-hirsute
removed: verification-needed-hirsute
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.4.0-97.110 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2022-01-24 04:38 EDT-------
Verified this with an update from the 'proposed' repository:

uname -a shows:
Linux a35lp66 5.4.0-97-generic #110-Ubuntu SMP Thu Jan 13 18:22:14 UTC 2022 s390x
s390x s390x GNU/Linux
and I can see the new sysfs attributes /sys/devices/ap/card??/max_msg_size.

Verification done, Thanks

Revision history for this message
Frank Heimes (fheimes) wrote :

Many thx Harald! (adjusting the tags now).

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (31.9 KiB)

This bug was fixed in the package linux - 5.4.0-97.110

---------------
linux (5.4.0-97.110) focal; urgency=medium

  * icmp_redirect from selftests fails on F/kvm (unary operator expected)
    (LP: #1938964)
    - selftests: icmp_redirect: pass xfail=0 to log_test()

  * Focal: CIFS stable updates (LP: #1954926)
    - cifs: use the expiry output of dns_query to schedule next resolution
    - cifs: set a minimum of 120s for next dns resolution
    - cifs: To match file servers, make sure the server hostname matches

  * seccomp_bpf in seccomp from ubuntu_kernel_selftests failed to build on B-5.4
    (LP: #1896420)
    - SAUCE: selftests/seccomp: fix "storage size of 'md' isn't known" build issue
    - SAUCE: selftests/seccomp: Fix s390x regs not defined issue

  * system crash when removing ipmi_msghandler module (LP: #1950666)
    - ipmi: Move remove_work to dedicated workqueue
    - ipmi: msghandler: Make symbol 'remove_work_wq' static

  * zcrypt DD: Toleration for new IBM Z Crypto Hardware - (Backport to Ubuntu
    20.04) (LP: #1954680)
    - s390/AP: support new dynamic AP bus size limit

  * [UBUNTU 20.04] KVM hardware diagnose data improvements for guest kernel -
    kernel part (LP: #1953334)
    - s390/setup: diag 318: refactor struct
    - s390/kvm: diagnose 0x318 sync and reset
    - KVM: s390: remove diag318 reset code
    - KVM: s390: add debug statement for diag 318 CPNC data

  * Updates to ib_peer_memory requested by Nvidia (LP: #1947206)
    - SAUCE: RDMA/core: Updated ib_peer_memory

  * Include Infiniband Peer Memory interface (LP: #1923104)
    - IB: Allow calls to ib_umem_get from kernel ULPs
    - SAUCE: RDMA/core: Introduce peer memory interface

  * Focal update: v5.4.162 upstream stable release (LP: #1954834)
    - arm64: zynqmp: Do not duplicate flash partition label property
    - arm64: zynqmp: Fix serial compatible string
    - ARM: dts: NSP: Fix mpcore, mmc node names
    - scsi: lpfc: Fix list_add() corruption in lpfc_drain_txq()
    - arm64: dts: hisilicon: fix arm,sp805 compatible string
    - RDMA/bnxt_re: Check if the vlan is valid before reporting
    - usb: musb: tusb6010: check return value after calling
      platform_get_resource()
    - usb: typec: tipd: Remove WARN_ON in tps6598x_block_read
    - arm64: dts: qcom: msm8998: Fix CPU/L2 idle state latency and residency
    - arm64: dts: freescale: fix arm,sp805 compatible string
    - ASoC: SOF: Intel: hda-dai: fix potential locking issue
    - clk: imx: imx6ul: Move csi_sel mux to correct base register
    - ASoC: nau8824: Add DMI quirk mechanism for active-high jack-detect
    - scsi: advansys: Fix kernel pointer leak
    - firmware_loader: fix pre-allocated buf built-in firmware use
    - ARM: dts: omap: fix gpmc,mux-add-data type
    - usb: host: ohci-tmio: check return value after calling
      platform_get_resource()
    - ARM: dts: ls1021a: move thermal-zones node out of soc/
    - ARM: dts: ls1021a-tsn: use generic "jedec,spi-nor" compatible for flash
    - ALSA: ISA: not for M68K
    - tty: tty_buffer: Fix the softlockup issue in flush_to_ldisc
    - MIPS: sni: Fix the build
    - scsi: target: Fix ordered tag handling
    - scsi: target: Fix al...

Changed in linux (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Frank Heimes (fheimes) wrote :

Changing the "affects hirsute" entry to Invalid, since hirsute reached it's end of life on January the 20th.
With that all other releases in service are Fix Released and with that the project entry itself, hence closing as Fixed Released.

Changed in linux (Ubuntu Hirsute):
status: Fix Committed → Invalid
Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-ibm-5.4/5.4.0-1014.15~18.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-bionic
Revision history for this message
Frank Heimes (fheimes) wrote :

This bug was for 20.04 GA kernel plain Ubuntu - hence verification on linux-ibm-5.4/5.4.0-1014 does not apply here.
However, I'm updating the tags to verification-done-bionic, just to unblock the process.

tags: added: verification-done-bionic
removed: verification-needed-bionic
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.