[UBUNTU 20.04] KVM: PV: ext call delivered twice when receiver in PSW wait

Bug #1995941 reported by bugproxy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Fix Released
Medium
Skipper Bug Screeners
linux (Ubuntu)
Invalid
Undecided
Unassigned
Focal
Fix Released
Medium
Canonical Kernel Team

Bug Description

SRU Justification:
==================

[Impact]

 * In a secure execution guest, the external interrupt for the SIGP
   external call order is delivered twice to a VCPU even though it was
   only sent once.

 * Under PV (protected virtualization), external call interrupts are
   delivered by the SIGP interpretation facility, without KVM's
   involvement.
   But, if the receiving CPU is in enabled wait, KVM needs to wake the
   receiving CPU such that the interrupt can be delivered.
   Hence, in this case, the SIGP external call order causes
   an interception.

 * In response, KVM only needs to wake the receiving VCPU.
   Interrupt delivery is then handled by the SIGP interpretation facility.

 * KVM wrongly assumed it also needs to request injection for the
   external call interrupt after the respective intercept, causing the
   interrupt to be delivered twice:
   * once through the SIGP interpretation facility
   * and once through the interrupt injection control by KVM.

 * Solution is to add appropriate special handling for 108 external
   call intercepts.

[Fix]

 * c3f0e5fd2d33 c3f0e5fd2d33d80c5a5a8b5e5d2bab2841709cc8
   "KVM: s390: pv: don't present the ecall interrupt twice"

[Test Case]

 * Have an Secure Execution (PV) environment setup on an
   IBM z15 or LinuxONE III LPAR using Ubuntu Server 20.04 (latest).

 * Apply kvm-unit-test submitted upstream:
   "[kvm-unit-tests PATCH v1 0/4] s390x: add tests for SIGP call \
    orders in enabled wait"

 * Run the smp_PV kvm-unit-test: ./run_tests.sh smp_PV

 * Check logs/smp_PV.log.
   If system is affected, the following line can be found:
   "ABORT: smp: psw wait: ecall: Unexpected external call interrupt \
    (code 0x1202): on cpu 1 at 0x11958"

 * If the system is not affected, the line should look like this:
   "PASS: smp: psw wait: ecall: received"

[Regression Potential / What can go wrong]

 * The handle_pv_notification can be wrong and misleading
   in case 'ret' is not handled correctly.

 * trace_kvm_s390_handle_sigp_pei might not be called correctly,
   now after the if condition.

 * In worst case the external interrupt could not be delivered
   at all or still too often.

[Other]

 * The fix/patch c3f0e5fd2d33 got upstream accepted with kernel v6.0,
   so it not only needs to be applied to 20.04/5.4, but also to 22.04/5.15
   and 22.10/5.19.

 * But the patch got properly tagged for upstream stable:
   Cc: <email address hidden> # 5.7
   Fixes: da24a0cc58ed ("KVM: s390: protvirt: Instruction emulation")

 * And with that it got already picked up and is included in:
   22.04 with Ubuntu-5.15.0-53.59 (currently in jammy-proposed)
   22.10 with Ubuntu-5.19.0-16.16 means incl. in the release kernel.

 * So the only Ubuntu release that is affected is 20.04/focal.
__________

Description: KVM: PV: ext call delivered twice when receiver in PSW wait
Symptom: In a secure execution guest, the external interrupt for the
               SIGP external call order is delivered twice to a VCPU even
               though it was only sent once.
Problem: Under PV, external call interrupts are delivered by the SIGP
               interpretation facility, without KVM's involvement. But, if the
               receiving CPU is in enabled wait, KVM needs to wake the
               receiving CPU such that the interrupt can be delivered. Hence,
               in this case, the SIGP external call order causes an
               interception. In response, KVM only needs to wake the receiving
               VCPU. Interrupt delivery is then handled by the SIGP
               interpretation facility.

               KVM wrongfuly assumed it also needs to request injection for the
               external call interrupt after the respective intercept, causing
               the interrupt to be delivered twice: once through the SIGP
               interpretation facility and once through the interrupt injection
               control by KVM.
Solution: Add appropriate special handling for 108 external call
               intercepts.
Reproduction: 0. Apply kvm-unit-test submitted upstream
                  ("[kvm-unit-tests PATCH v1 0/4] s390x: add tests for SIGP
                  call orders in enabled wait").
               1. Run the smp_PV kvm-unit-test:
                    ./run_tests.sh smp_PV
               2. Check logs/smp_PV.log. If system is affected, the following
                  line can be found:
                    ABORT: smp: psw wait: ecall: Unexpected external call \
                     interrupt (code 0x1202): on cpu 1 at 0x11958
                  If the system is not affected, the line should look like
                  this:
                    PASS: smp: psw wait: ecall: received

Preventive: yes
Author: Nico Boehr <email address hidden>
Component: kernel

CVE References

bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-199408 severity-medium targetmilestone-inin---
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2022-11-08 05:32 EDT-------
Fix is upstream available:
Upstream-ID: c3f0e5fd2d33d80c5a5a8b5e5d2bab2841709cc8
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c3f0e5fd2d33d80c5a5a8b5e5d2bab2841709cc8

Revision history for this message
Frank Heimes (fheimes) wrote :

Hello Nico, thanks for raising this.

The fix/patch c3f0e5fd2d33 "KVM: s390: pv: don't present the ecall interrupt twice" got upstream accepted with kernel v6.0, so I assume that it not only needs to be applied to 20.04/5.4, but also to 22.04/5.15 and 22.10/5.19.
And I've noticed that it got nice and properly tagged for upstream stable:
Cc: <email address hidden> # 5.7
Fixes: da24a0cc58ed ("KVM: s390: protvirt: Instruction emulation")

And with that it got already picked up and is included in:
22.04 with Ubuntu-5.15.0-53.59 (currently in jammy-proposed)
22.10 with Ubuntu-5.19.0-16.16 means already incl. in the release kernel.

So the only Ubuntu release that is affected is 20.04/focal, so I've set that as affected target series...

Changed in linux (Ubuntu):
status: New → Invalid
Changed in linux (Ubuntu Focal):
status: New → Triaged
Changed in ubuntu-z-systems:
status: New → Triaged
Changed in linux (Ubuntu Focal):
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
Changed in linux (Ubuntu):
assignee: Skipper Bug Screeners (skipper-screen-team) → nobody
Changed in ubuntu-z-systems:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
Frank Heimes (fheimes)
description: updated
description: updated
Revision history for this message
Frank Heimes (fheimes) wrote :

SRU request submitted to the Ubuntu kernel team mailing list for focal:
https://lists.ubuntu.com/archives/kernel-team/2022-November/thread.html#134671
Changing status to 'In Progress' for focal.

Changed in linux (Ubuntu Focal):
assignee: Skipper Bug Screeners (skipper-screen-team) → Canonical Kernel Team (canonical-kernel-team)
status: Triaged → In Progress
Changed in ubuntu-z-systems:
status: Triaged → In Progress
importance: Undecided → Medium
Changed in linux (Ubuntu Focal):
importance: Undecided → Medium
Revision history for this message
Frank Heimes (fheimes) wrote :

A test kernel is currently being build at this PPA:
https://launchpad.net/~fheimes/+archive/ubuntu/lp1995941

Stefan Bader (smb)
Changed in linux (Ubuntu Focal):
status: In Progress → Fix Committed
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.4.0-136.153 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-focal-linux verification-needed-focal
bugproxy (bugproxy)
tags: added: targetmilestone-inin2004
removed: targetmilestone-inin---
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2022-12-09 09:39 EDT-------
Hi, sorry this took a while.

The bug was successfully verified:

~/kvm-unit-tests# dpkg -l | grep linux-image-5.4.0-136-generic
ii linux-image-5.4.0-136-generic 5.4.0-136.153 s390x Signed kernel image generic
~/kvm-unit-tests# uname -a
Linux t35lp63.lnxne.boe 5.4.0-136-generic #153-Ubuntu SMP Thu Nov 24 15:57:18 UTC 2022 s390x s390x s390x GNU/Linux
~/kvm-unit-tests# ./run_tests.sh smp_PV
PASS smp_PV (67 tests, 2 skipped)
~/kvm-unit-tests# grep 'psw wait: ecall:' logs/smp_PV.log
PASS: smp: psw wait: ecall: received

Revision history for this message
Frank Heimes (fheimes) wrote :

Thank you Nico for your verification! I'm going to adjust the tags accordingly.

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-xilinx-zynqmp/5.4.0-1020.24 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-focal-linux-xilinx-zynqmp verification-needed-focal
removed: verification-done-focal
Revision history for this message
Frank Heimes (fheimes) wrote :

This bug was not opened against linux-xilinx-zynqmp.
So I'm updating the verification tag just to unblock the further process.

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (20.0 KiB)

This bug was fixed in the package linux - 5.4.0-136.153

---------------
linux (5.4.0-136.153) focal; urgency=medium

  * focal/linux: 5.4.0-136.153 -proposed tracker (LP: #1997835)

  * Expose built-in trusted and revoked certificates (LP: #1996892)
    - [Packaging] Expose built-in trusted and revoked certificates

  * [UBUNTU 20.04] KVM: PV: ext call delivered twice when receiver in PSW wait
    (LP: #1995941)
    - KVM: s390: pv: don't present the ecall interrupt twice

  * [UBUNTU 20.04] boot: Add s390x secure boot trailer (LP: #1996071)
    - s390/boot: add secure boot trailer

  * Fix rfkill causing soft blocked wifi (LP: #1996198)
    - platform/x86: hp_wmi: Fix rfkill causing soft blocked wifi

  * md: Replace snprintf with scnprintf (LP: #1993315)
    - md: Replace snprintf with scnprintf

  * input/keyboard: the keyboard on some Asus laptops can't work (LP: #1992266)
    - ACPI: resource: Skip IRQ override on Asus Vivobook K3402ZA/K3502ZA
    - ACPI: resource: Add ASUS model S5402ZA to quirks

  * Focal update: v5.4.218 upstream stable release (LP: #1995530)
    - mm: pagewalk: Fix race between unmap and page walker
    - perf tools: Fixup get_current_dir_name() compilation
    - firmware: arm_scmi: Add SCMI PM driver remove routine
    - dmaengine: xilinx_dma: cleanup for fetching xlnx,num-fstores property
    - dmaengine: xilinx_dma: Report error in case of dma_set_mask_and_coherent API
      failure
    - ARM: dts: fix Moxa SDIO 'compatible', remove 'sdhci' misnomer
    - scsi: qedf: Fix a UAF bug in __qedf_probe()
    - net/ieee802154: fix uninit value bug in dgram_sendmsg
    - um: Cleanup syscall_handler_t cast in syscalls_32.h
    - um: Cleanup compiler warning in arch/x86/um/tls_32.c
    - arch: um: Mark the stack non-executable to fix a binutils warning
    - usb: mon: make mmapped memory read only
    - USB: serial: ftdi_sio: fix 300 bps rate for SIO
    - mmc: core: Replace with already defined values for readability
    - mmc: core: Terminate infinite loop in SD-UHS voltage switch
    - rpmsg: qcom: glink: replace strncpy() with strscpy_pad()
    - nilfs2: fix leak of nilfs_root in case of writer thread creation failure
    - nilfs2: replace WARN_ONs by nilfs_error for checkpoint acquisition failure
    - ceph: don't truncate file in atomic_open
    - random: clamp credited irq bits to maximum mixed
    - ALSA: hda: Fix position reporting on Poulsbo
    - efi: Correct Macmini DMI match in uefi cert quirk
    - USB: serial: qcserial: add new usb-id for Dell branded EM7455
    - random: restore O_NONBLOCK support
    - random: avoid reading two cache lines on irq randomness
    - random: use expired timer rather than wq for mixing fast pool
    - Input: xpad - add supported devices as contributed on github
    - Input: xpad - fix wireless 360 controller breaking after suspend
    - Linux 5.4.218

  * Focal update: v5.4.217 upstream stable release (LP: #1995528)
    - xfs: fix misuse of the XFS_ATTR_INCOMPLETE flag
    - xfs: introduce XFS_MAX_FILEOFF
    - xfs: truncate should remove all blocks, not just to the end of the page
      cache
    - xfs: fix s_maxbytes computation on 32-bit kernels
    - xfs: fix IOCB_NOWAIT h...

Changed in linux (Ubuntu Focal):
status: Fix Committed → Fix Released
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-iot/5.4.0-1011.13 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-focal-linux-iot verification-needed-focal
removed: verification-done-focal
Revision history for this message
Frank Heimes (fheimes) wrote :

This bug was not opened against linux-iot, hence the new request for verification is not valid.
I'm updating the verification tags just to unblock the further process...

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-aws/5.4.0-1095.103 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-focal-linux-aws verification-needed-focal
removed: verification-done-focal
Frank Heimes (fheimes)
tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-azure/5.4.0-1102.108 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-focal-linux-azure verification-needed-focal
removed: verification-done-focal
Frank Heimes (fheimes)
tags: added: verification-done-focal
removed: verification-needed-focal
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.