18.10 kernel does not appear to validate kernel module signatures correctly

Bug #1798863 reported by Daniel Dadap
38
This bug affects 9 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Medium
Seth Forshee
Cosmic
Fix Released
Medium
Seth Forshee
shim (Ubuntu)
Invalid
Medium
Mathieu Trudel-Lapierre
Cosmic
Invalid
Medium
Mathieu Trudel-Lapierre

Bug Description

SRU Justification

Impact: An bug in the secure boot lockdown patches in the 18.10 kernel causes the results of module signature verification to be ignored, allowing modules with no signature or an invalid signature to be loaded. A second bug results in the MOK not being trusted for signing modules, but this bug has been masked by the first bug.

Fix: These bugs should be fixed together to avoid regressions in dkms module loading under secure boot. First, fix the latter bug by trusting keys in the kernel's secondary keyring for module signing. Then fix the former bug by removing code related to trusting IMA signatures for loading modules under kernel lockdown.

Test Case: Confirm the following behavior under kernel lockdown:

  1) Unsigned modules cannot be loaded.

  2) Modules signed with an untrusted key cannot be loaded.

  3) Modules signed with the kernel's ephemeral build-time key can be loaded.

  4) Modules signed with a MOK which has been enrolled with shim can be loaded.

I have tested to verify these conditions with the proposed fixes.

Regression Potential: This restores the behavior from previous Ubuntu releases, so no regressions are expected wrt those releases. In some cases modules that were loading under lockdown might no longer load, but this is the intended behavior.

---

On a system with Ubuntu 18.10, with secure boot enabled, and a key enrolled in the MOK database, I am observing the following peculiar behaviors:

* Signature verification appears to be disabled, and cannot be enabled again. It appeared to be enabled previously, as loading of unsigned modules was failing, and `mokutil --enable-validation` runs without incident; however, upon the next boot when attempting to confirm the change, MokManager prints an error message "Unable to delete Secure Boot state" after completing the password challenge.
* As a result of signature verification being disabled, modules signed with untrusted keys taint the kernel instead of failing to load outright.
* Regardless of signature verification being enabled or not, it seems that the key enrolled in the MOK is not being used for validating kernel module signatures. Modules signed with the key still fail the signature verification test and taint the kernel, even though the key is visible in the output of `mokutil --list-enrolled`, and testing the key with `mokutil --test-key` shows that it's enrolled. Also, a message acknowledging the key appears in dmesg: Loaded UEFI:MokListRT cert 'nvidia-installer generated signing key: 90c957eb56dfb04d8734d54fb614ef5af6c69318' linked to secondary sys keyring
* Also, somewhat strangely, in this state with module signature verification not being enforced, attempting to load a completely unsigned kernel module suceeds, and doesn't even log a kernel message about a missing/invalid signature, or taint the kernel.

Apport report attached, which includes dmesg log showing the kernel acknowledging the key enrolled in the MOK database, and a signature verification failure and subsequent successful loading of a module signed with that key:

 [ 4.234093] Loaded UEFI:MokListRT cert 'nvidia-installer generated signing key: 90c957eb56dfb04d8734d54fb614ef5af6c69318' linked to secondary sys keyring
...
 [ 6.628452] nvidia: module verification failed: signature and/or required key missing - tainting kernel
...
 [ 6.637252] nvidia-nvlink: Nvlink Core is being initialized, major device number 238
 [ 6.637507] nvidia 0000:01:00.0: enabling device (0006 -> 0007)
 [ 6.637620] nvidia 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
 [ 6.737216] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 410.66 Wed Oct 10 12:01:53 CDT 2018 (using threaded interrupts)

This system dual-boots Ubuntu 18.04 and Ubuntu 18.10: when booted into Ubuntu 18.04, signatures made with the same key are recognized as valid. Hence, I suspect that something changed in the Ubuntu 18.10 kernel which is causing signature verification to function in an unexpected way.
---
ProblemType: Bug
ApportVersion: 2.20.10-0ubuntu13
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: danix 1729 F.... pulseaudio
DistroRelease: Ubuntu 18.10
InstallationDate: Installed on 2018-10-23 (0 days ago)
InstallationMedia: Ubuntu 18.10 "Cosmic Cuttlefish" - Release amd64 (20181017.3)
NonfreeKernelModules: nvidia_uvm nvidia
Package: linux (not installed)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.18.0-10-generic root=/dev/mapper/ubuntu--vg-root ro quiet splash nouveau.modeset=0
ProcVersionSignature: Ubuntu 4.18.0-10.11-generic 4.18.12
RelatedPackageVersions:
 linux-restricted-modules-4.18.0-10-generic N/A
 linux-backports-modules-4.18.0-10-generic N/A
 linux-firmware 1.175
Tags: cosmic
Uname: Linux 4.18.0-10-generic x86_64
UnreportableReason: This report is about a package that is not installed.
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: False
dmi.bios.date: 03/20/2018
dmi.bios.vendor: HP
dmi.bios.version: P70 Ver. 01.18
dmi.board.name: 8270
dmi.board.vendor: HP
dmi.board.version: KBC Version 46.67
dmi.chassis.type: 10
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:bvrP70Ver.01.18:bd03/20/2018:svnHP:pn:pvr:rvnHP:rn8270:rvrKBCVersion46.67:cvnHP:ct10:cvr:
dmi.product.family: 103C_5336AN HP ZBook
dmi.sys.vendor: HP

Revision history for this message
Daniel Dadap (ddadap) wrote :
description: updated
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1798863

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

This could be a duplicate of bug 1798940

Changed in linux (Ubuntu):
importance: Undecided → Medium
Revision history for this message
Daniel Dadap (ddadap) wrote : AlsaInfo.txt

apport information

tags: added: apport-collected cosmic
description: updated
Revision history for this message
Daniel Dadap (ddadap) wrote : CRDA.txt

apport information

Revision history for this message
Daniel Dadap (ddadap) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Daniel Dadap (ddadap) wrote : IwConfig.txt

apport information

Revision history for this message
Daniel Dadap (ddadap) wrote : Lspci.txt

apport information

Revision history for this message
Daniel Dadap (ddadap) wrote : Lsusb.txt

apport information

Revision history for this message
Daniel Dadap (ddadap) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Daniel Dadap (ddadap) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Daniel Dadap (ddadap) wrote : ProcModules.txt

apport information

Revision history for this message
Daniel Dadap (ddadap) wrote : PulseList.txt

apport information

Revision history for this message
Daniel Dadap (ddadap) wrote : RfKill.txt

apport information

Revision history for this message
Daniel Dadap (ddadap) wrote : UdevDb.txt

apport information

Revision history for this message
Daniel Dadap (ddadap) wrote : WifiSyslog.txt

apport information

Seth Forshee (sforshee)
Changed in linux (Ubuntu Cosmic):
assignee: nobody → Seth Forshee (sforshee)
status: Incomplete → Confirmed
Seth Forshee (sforshee)
Changed in linux (Ubuntu Cosmic):
status: Confirmed → In Progress
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

There's a bit I don't understand:

"* Signature verification appears to be disabled, and cannot be enabled again. It appeared to be enabled previously, as loading of unsigned modules was failing, and `mokutil --enable-validation` runs without incident; however, upon the next boot when attempting to confirm the change, MokManager prints an error message "Unable to delete Secure Boot state" after completing the password challenge."

Sure, there are other issues at play here, but every test I've done of this has been working just fine, and at that point we're not actually talking to the kernel at all. Could we be hitting an issue with the firmware on this specific HP machine? Is this something you see on other hardware (if you can test)? If you toggle validation in 18.04, do things work then?

I wonder if you're hitting an issue here due to versioning of the different EFI binaries; if you have 18.10 and 18.04 installed, I wonder which version of shim and mokmanager are actually being used.

As for what happens at the kernel level, I don't know. If the MOK key is available there (it certainly seemed to be), then I concur there would have to be a bug in the keyring handling.

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Opening a task for shim; I want to check whether this is doing what it is supposed to, at least on a new install.

Changed in shim (Ubuntu Cosmic):
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Mathieu Trudel-Lapierre (cyphermox)
Revision history for this message
Seth Forshee (sforshee) wrote :

I can clarify what's happening in the kernel. There are two bugs, and one is masking the other.

The first bug is that we don't use the secondary keyring for verifying module signatures. The secondary keyring is where the MOK ends.

The second bug is that we aren't enforcing that modules must be signed when under lockdown. So even though signature verification fails due to the first bug the module is still allowed to load.

I have patches for both bugs.

Revision history for this message
Steve Langasek (vorlon) wrote :

Based on Seth's response there is no reason to suspect shim here.

Changed in shim (Ubuntu Cosmic):
status: In Progress → Invalid
Revision history for this message
Tyler Hicks (tyhicks) wrote :

I've requested a CVE for this issue. I wanted to provide some more context as other Linux distributions will likely be reading this bug report once the CVE assignment occurs.

This flaw is introduced by certain configuration options in combination with this out-of-tree patch from the Lockdown patchset:

  https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/cosmic/commit/?id=03c7de9e956395f3b36f86f89b62780ad9501eef

We do not force module signatures (CONFIG_MODULE_SIG_FORCE=n), we enable IMA-appraise (CONFIG_IMA_APPRAISE=y), and we do not use the built-in IMA secure_boot policy snippet by default. Therefore, no signature verification is performed when a module is loaded via the finit_module(2) syscall.

Revision history for this message
Tyler Hicks (tyhicks) wrote :

I need to make a correction to the last sentence of my last comment. Signature verification is performed but the result is effectively ignored due to the configuration options mentioned earlier in that comment.

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Use CVE-2018-18653.

Revision history for this message
Daniel Dadap (ddadap) wrote :

Cool, glad you were able to track down the problems. Sorry if my report that module signature verification was disabled and couldn't be re-enabled was misleading. That's what I thought was happening; I didn't think to imagine that the enforcement of the "valid signature required" policy wasn't taking place at all.

Seth Forshee (sforshee)
description: updated
Stefan Bader (smb)
Changed in linux (Ubuntu Cosmic):
status: In Progress → Fix Committed
Revision history for this message
Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-cosmic' to 'verification-done-cosmic'. If the problem still exists, change the tag 'verification-needed-cosmic' to 'verification-failed-cosmic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-cosmic
Thanh Tung (tung9484)
Changed in linux (Ubuntu Cosmic):
status: Fix Committed → Fix Released
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

Hi Thanh Tung,

Please do not change manually a series release task to 'Fix Released', that is done automatically by a bot when the package hits -updates.

Thank you.

Changed in linux (Ubuntu Cosmic):
status: Fix Released → Fix Committed
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

Hi Daniel Dadap,

Could you please verify if the Cosmic kernel currently in -proposed fixes the issue?

Thank you.

Revision history for this message
Daniel Dadap (ddadap) wrote :

Yes, I do see the expected behavior now with signed modules, both when the signing key is enrolled in the MOK (module loads, no verification error) and when it is not enrolled in the MOK (module fails to load due to verification error.) However, the behavior is not quite what I expect when a module is unsigned. The module fails to load, which is expected, but there is no error message in dmesg indicating a missing key, which can make it tricky to determine why the module failed to load, since the failure message printed by modprobe/insmod is simply "Operation not permitted". (It seems the ENOKEY failure is not getting propagated to the user-facing tool.)

Revision history for this message
Daniel Dadap (ddadap) wrote :

Just to confirm, this is with the 4.18.0-12-generic x86_64 kernel from cosmic-proposed.

Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

Hi Daniel Dadap,

Thank you for your feedback.

Revision history for this message
Seth Forshee (sforshee) wrote :

Daniel: A message is printed for only the first time a module signature verification fails for any reason, so if you tested the "signed with key not enrolled in the MOK" and then "not signed" cases in the same boot you will only see a message for the first one. If you reboot and then retest the "not signed" case by itself you should see a message.

Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

Based on the above comments, I'm marking verification done for Cosmic. Thank you!

tags: added: verification-done-cosmic
removed: verification-needed-cosmic
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (39.7 KiB)

This bug was fixed in the package linux - 4.18.0-12.13

---------------
linux (4.18.0-12.13) cosmic; urgency=medium

  * linux: 4.18.0-12.13 -proposed tracker (LP: #1802743)

  * [FEAT] Guest-dedicated Crypto Adapters (LP: #1787405)
    - s390/zcrypt: Add ZAPQ inline function.
    - s390/zcrypt: Review inline assembler constraints.
    - s390/zcrypt: Integrate ap_asm.h into include/asm/ap.h.
    - s390/zcrypt: fix ap_instructions_available() returncodes
    - KVM: s390: vsie: simulate VCPU SIE entry/exit
    - KVM: s390: introduce and use KVM_REQ_VSIE_RESTART
    - KVM: s390: refactor crypto initialization
    - s390: vfio-ap: base implementation of VFIO AP device driver
    - s390: vfio-ap: register matrix device with VFIO mdev framework
    - s390: vfio-ap: sysfs interfaces to configure adapters
    - s390: vfio-ap: sysfs interfaces to configure domains
    - s390: vfio-ap: sysfs interfaces to configure control domains
    - s390: vfio-ap: sysfs interface to view matrix mdev matrix
    - KVM: s390: interface to clear CRYCB masks
    - s390: vfio-ap: implement mediated device open callback
    - s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl
    - s390: vfio-ap: zeroize the AP queues
    - s390: vfio-ap: implement VFIO_DEVICE_RESET ioctl
    - KVM: s390: Clear Crypto Control Block when using vSIE
    - KVM: s390: vsie: Do the CRYCB validation first
    - KVM: s390: vsie: Make use of CRYCB FORMAT2 clear
    - KVM: s390: vsie: Allow CRYCB FORMAT-2
    - KVM: s390: vsie: allow CRYCB FORMAT-1
    - KVM: s390: vsie: allow CRYCB FORMAT-0
    - KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-1
    - KVM: s390: vsie: allow guest FORMAT-1 CRYCB on host FORMAT-2
    - KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-2
    - KVM: s390: device attrs to enable/disable AP interpretation
    - KVM: s390: CPU model support for AP virtualization
    - s390: doc: detailed specifications for AP virtualization
    - KVM: s390: fix locking for crypto setting error path
    - KVM: s390: Tracing APCB changes
    - s390: vfio-ap: setup APCB mask using KVM dedicated function
    - [Config:] Enable CONFIG_S390_AP_IOMMU and set CONFIG_VFIO_AP to module.

  * Bypass of mount visibility through userns + mount propagation (LP: #1789161)
    - mount: Retest MNT_LOCKED in do_umount
    - mount: Don't allow copying MNT_UNBINDABLE|MNT_LOCKED mounts

  * CVE-2018-18955: nested user namespaces with more than five extents
    incorrectly grant privileges over inode (LP: #1801924) // CVE-2018-18955
    - userns: also map extents in the reverse map to kernel IDs

  * kdump fail due to an IRQ storm (LP: #1797990)
    - SAUCE: x86/PCI: Export find_cap() to be used in early PCI code
    - SAUCE: x86/quirks: Add parameter to clear MSIs early on boot
    - SAUCE: x86/quirks: Scan all busses for early PCI quirks

  * crash in ENA driver on removing an interface (LP: #1802341)
    - SAUCE: net: ena: fix crash during ena_remove()

  * Ubuntu 18.04.1 - [s390x] Kernel panic while stressing network bonding
    (LP: #1797367)
    - s390/qeth: reduce hard-coded access to ccw channels
    - s390/qeth: sanitize strings in debug messages

  * Add checksum offload and T...

Changed in linux (Ubuntu Cosmic):
status: Fix Committed → Fix Released
Revision history for this message
Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-bionic
Revision history for this message
Daniel Dadap (ddadap) wrote :

I'm confused about the above message. This bug never affected the kernel in Bionic AFAIK. Or is this referring to the HWE kernel for Bionic from Cosmic for 18.04.2? In that case, why isn't this change already included in the HWE kernel as it was imported from Cosmic, rather than needing to be brought in through -proposed first?

Revision history for this message
Andy Whitcroft (apw) wrote :

This bug was erroneously marked for verification in bionic; verification is not required and verification-needed-bionic is being removed.

tags: added: kernel-fixup-verification-needed-bionic verification-done-bionic
removed: verification-needed-bionic
Revision history for this message
Mohd Abbas (mohd-abbas) wrote :

Still facing the same problem!

Machine: MSI GV62 7RD
Processor: Intel® Core™ i7-7700HQ CPU @ 2.80GHz × 8
Graphics: GeForce GTX 1050/PCIe/SSE2
Operating System: Ubuntu 18.10
Kernel: 4.18.0-15-generic

Revision history for this message
Steve Langasek (vorlon) wrote :

That is unlikely to be true. Please file a new bug report and describe in your own words what is happening and why you believe it is a bug.

Seth Forshee (sforshee)
Changed in linux (Ubuntu):
status: In Progress → Fix Released
Brad Figg (brad-figg)
tags: added: cscc
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.