x86: mm: early boot problem on i386 with KPTI enabled

This change was made by a bot.

You'll probably want the boot.log?

I have two old Dell computers which experience a problem similar to what Michael describes. They work fine at Linux 4.15.0-47 but at 4.15.0-48 they show a corrupted display while booting. I can revert to 4.15.0-47 and await a fix.

I will attach the output file of ubuntu-bug made in 4.15.0-48 recovery mode.

Here is the Ubuntu-bug data collection for my other Dell the exibits the same problem.

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

Update:
HP Laptop: I have upgraded BIOS on the HP6710b laptop to F16, which included upgrade of video BIOS. It had no effect on the boot behaviour.

ASUS Laptop: I have realised that I CAN get the grub menu - its just that the screen is black and I can´t see it. (ie: I can navigate the grub menu using arrow/enter keys and select various options from memory of where they are in the menu tree). Behaviour is essentially the same as HP laptop - it will only boot successfully into 4.15.0-48 after a previous failed boot. A first boot fails.

Note that I tried a fresh install from the mini.iso on the asus laptop. Boot problem is the same on a fresh install of 4.15.0-48, not just upgrading from earlier version through software updater.

Ubunutu-bug output of the asus filed a new bug report Bug #1828764

This problem affected all Intel CPUs we have, except for Atoms

This problem also exists in kernel 4.15.0-50 version on Ubuntu 18.04.

I am seeing a similar problem.

Linux Mint 18.3 XFCE, Lenovo Thinkpad T60 i686 / ATI Radeon RV515 GPU

After upgrade to kernel 4.15.0-48, I get a black screen and frozen keyboard once the boot splash screen exits, when the login screen should appear. There is no mouse pointer, and I can't escape with ctrl-alt-f1, ctrl-alt-backspace, etc. If I leave the system running for a while, there is occasional disk activity, and syslog entries with later time stamps, indicating that the kernel is still running, but the X server is frozen.

Same thing with latest kernel 4.15.0-50.

I can boot into the previous kernel 4.15.0-47 with no problems.

If I boot into run level 3 (terminal only, no GUI) by editing the Grub kernel command line, the system will boot, but enters the same frozen state if I then start the GUI with 'sudo service lightdm start' (which works fine with the previous 4.15.0-47 kernel).

If I add "nomodeset" to the kernel command line, the system will boot into the GUI. In this case Xorg.0.log shows that the X server unloads the Radeon driver (which will not run without KMS) and loads the fallback VESA driver instead. I.e., on my computer the bug affects the Radeon driver but not the VESA driver. (However, this is not a solution for me because the VESA driver doesn't handle my 1400x1050 screen resolution correctly.)

One other thing - the bug happens maybe 9 times out of 10 but not every time. On the occasional successful boot, once the GUI comes up without freezing, everything seems to work OK.

I have three different machines running Xubuntu 18.04 i386. All are affected.

After upgrade to kernel 4.15.0-48 the boot process will stop to an almost black screen with a few multi-coloured lines.
Problem still persists with kernel 4.15.0-50.

Everything works fine with kernel 4.15.0-47.

I have a Lenovo G500 (32 bit laptop about 5 years old).

I am seeing the same behavior on my system. Booting 4.15.0-47-generic works fine, but both
4.15.0-48-generic and 4.15.0-50-generic usually hang at boot time. If I enter grub at boot and select 4.15.0-50-generic, it usually hangs at this:

Loading Linux 4.15.0-50-generic ...
Loading initial ramdisk ...

If I let the system just boot up w/o going into grub I don't see the above. I usually either get a solid blank purple screen or a blank purple screen with a bar of random colored pixels across the middle of the screen.

Now yesterday I was able to get 4.15.0-50-generic to boot all they way into the GUI after booting it from the grub menu and was able to use the system for the rest of the day, no problems. So it seems to be a startup only problem.

When in the hung state, I can't get to the console screen via the usual Fn-Ctrl-Alt-F3.

I have the same issue on 3 Laptops (2 HP nc6230 and 1 Asus, all 32bit systems), only solution so far is to revert to Linux 4.15.0-47. I could not find any useful hint in any of the logs.
One strange thing is that only every second boot fails. After hard power-off a hanging boot the next boot works fine one time, but the next one again hangs.
I could not even get a text console with Ctrl-Alt-F1 in case it hangs.

I have precisely the same issue:
Dell Latitude D530
Intel graphics GM965/GL960
Lubuntu 18.04 (32 bit)
Kernel 4.15.0-50

This problem also exists in kernel 4.15.0-51 version on Ubuntu 18.04 (i386).

This might be a duplicate of 1830433, could you give it a try with this test kernel and see if the problem is still happening?

https://kernel.ubuntu.com/~arighi/LP-1827884/

Thanks!

On 2019-07-02 16:36:27, Andrea Righi wrote:
> This might be a duplicate of 1830433, could you give it a try with this
> test kernel and see if the problem is still happening?
>
> https://kernel.ubuntu.com/~arighi/LP-1827884/

For everyone who is able to try out that test build, please remember to
remove the "mitigations=off" and "nopti" kernel command line
parameters that you've added to work around this bug. We want you to
test that kernel without using the workaround.

If you report successful results when trying the test kernel, please
include the output from the following commands:

 $ cat /proc/cmdline

 $ cat /sys/devices/system/cpu/vulnerabilities/meltdown

The output from those commands will help us make sure that the bug is
fixed rather than a workaround giving us false hope.

My hope is that Andrea's work on bug 1830433 ends up fixing the problems
that you all are seeing in this bug. Thanks in advance for any testing
that you can provide.

I've tried. It's still happening.

Status changed to 'Confirmed' because the bug affects multiple users.

I`ve tried kernel test version, the problem is still happening.

So, it looks like bug 1830433 didn't fix this problem... assuming it's still an x86/mm issue, it'd be interesting to test another kernel (I've applied a few more x86/mm-related fixes).

This new test kernel is available at the same place:

https://kernel.ubuntu.com/~arighi/LP-1827884/

As correctly pointed out by Tyler, for the test you should remove "mitigations=off" and "nopti" and in case of success, please report the output of:

 $ cat /proc/cmdline

 $ cat /sys/devices/system/cpu/vulnerabilities/meltdown

Thanks again for all the tests!

Seems I used wrong kernel for previous test. Sorry.
This new kernel works, but xorg doesn't start.
I'm testing on Lubuntu xenial. How to add "Xenial" under the "Bionic" on the top of this thread?

I`ve tried second kernel test version on ubuntu 18.04(bionic), the problem still exists.

I've rechecked the last test version. Previous problem is gone, but xorg doesn't start. Xorg doesn't start even with the "nopti" option.

BOOT_IMAGE=/@/boot/vmlinuz-4.15.0-54-generic root=UUID=50c3a930-a3e9-4444-b4ea-1646d276c7c6 ro rootflags=subvol=@ ipv6.disable=1 zswap.enabled=0 raid=noautodetect priority=low video=SVIDEO-1:d module_blacklist=r8169,mii,msr,jfs,xfs,bluetooth,hfs,hfsplus,ufs,minix,ntfs,joydev,mac_hid

Mitigation: PTI

But I use Lubuntu Xenial. Probably this is why xorg doesn't start with the last test version. I can upload dmesg or something else if it is needed.

Thanks for the tests and the feedback @dima2017 and @valia0906! Could you also post the output of the following commands?

$ uname -r

$ cat /proc/cmdline

$ cat /sys/devices/system/cpu/vulnerabilities/meltdown

My output:
uname -a
Linux TEPL-ET2013 4.15.0-54-generic #58+lp1827884 SMP Wed Jul 3 10:05:45 UTC 2019 i686 i686 i686 GNU/Linux

cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.15.0-54-generic root=UUID=63b3d2c0-b9a9-4112-97eb-de501efd9205 ro quiet splash

cat /sys/devices/system/cpu/vulnerabilities/meltdown
Mitigation: PTI

Thanks Andrea, much appreciated.

Have installed your latest test kernel on my ASUS W1N Pentium M (banias) machine running 18.04 Xubuntu, and it appears to work! Have shutdown normally and rebooted several times with no problems. (Note: unable to test on the HO6710b machine that originally reported the bug - its still on long term loan to my brother)

Output of commands...

uname -a
Linux asusw1na 4.15.0-54-generic #58+lp1827884 SMP Wed Jul 3 10:05:45 UTC 2019 i686 i686 i686 GNU/Linux

cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.15.0-54-generic root=UUID=9e042291-28d3-4f6b-a455-488355b84473 ro forcepae

cat /sys/devices/system/cpu/vulnerabilities/meltdown
Mitigation: PTI

______

NOTE: I'm still a Linux newbie. I have a couple of questions about what happens next, and grateful if anyone could offer some guidance...

I installed 3 of the test kernel deb packages:
   linux-headers-4.15.0-54_4.15.0-54.58+lp1827884_all.deb
   linux-headers-4.15.0-54-generic_4.15.0-54.58+lp1827884_i386.deb
   linux-image-4.15.0-54-generic_4.15.0-54.58+lp1827884_i386.deb
Should I be installing any others to make sure everything else and all my software applications etc still work? (Just a basic end user of the generic version)

And what happens next once the bug fix is confirmed? Will it roll into the automatic updates? Do I have to uninstall the test kernel to still get those updates?

Thanks for any advice on the newbie questions. And Thanks again Andrea for the fix!

Michael, first of all thanks for testing. It seems that (in your case at least) the problem was fixed by the extra commits that I backported into 4.15.0-54, so I think it's worth to have them applied.

To answer your questions, 1) you may want to install also linux-modules-4.15.0-54-generic_4.15.0-54.58+lp1827884_i386.deb and 2) the next step is to send an SRU email to the kernel-team mailing list (I'll do that) requesting to include these fixes. If the request is accepted, during the next SRU cycle these fixes will be applied to the "official" kernel.

In the meantime you can safely use the test kernel if it works and, as soon as the new kernel comes out, you'll get it via the regular automatic updates.

Hi Andrea,

Thanks a lot for the test kernel.

https://kernel.ubuntu.com/~arighi/LP-1827884/

I tried it on Laptop ASUS X51 R.

Results:

1. It boots just fine (without nopti boot option.)
2. However, after boot Screen Resolution is fixed to 1024x768.
   Native LCD-Resolution with working ATI Radeon Graphics would be 1280x800 (and adjustable)
   This is the same effect to when I use my current kernel 4.15.0-51 in recovery mode.

   Boot with 4.15.0.51 using nopti-Kernel-Option results in graphics OK.

Thanks a lot,
Thomas

Status changed to 'Confirmed' because the bug affects multiple users.

It flooded this before reboot:

IP: print_modules+0x40/0xbd
*pdpt = 0000000036445001 *pde = 000000000de36063 *pte = 0000000000000000
Thread overran stack, or stack corrupted
Oops: 000 [a1672] PREEMPI SMP PTI
Modules linked in:
BUG: unable to handle kernel paging request at fffffffc

I don't know which string is first.
I can see some video artifacts before rebooting.

(I just copied this message from #1827725)

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1827725

I could see that without "quiet" boot option.

The problem also present in kernel version 4.15.0.55.57.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

I can test it on xenial. Is the fix released in xenial proposed?

I-ve tested bionic-proposed kernel version 4.15.0.56.58, problem is still present.

I have been searching for this bug-report for a few weeks now, and I can confirm that I too am experiencing this bug.
After updating from Ubuntu 16.04 LTS to Ubuntu 18.04 LTS, upon first boot I experienced a black screen on my laptop (HP Presario V4000 from 2014). The backlight was still on and so was the fan, though there was no hard-drive activity light. After waiting for around half an hour, I cut the power and rebooted.
Upon the second boot, everything worked fine, leading me to beleive it was simply a problem after updating. The next day, I booted the laptop again and experienced the black screen bug. I soon figured out that it occured every second boot (Roughly, sometimes I would experience the bug twice in a row).
I decided to try “Advanced options for Ubuntu”, and selected “Ubuntu, with Linux 4.15.0-45-generic”. This worked first try. After testing multiple times, I never got a failed boot.
Unfortunately I can confirm that this bug still occurs occasionally when I select:
“Ubuntu, with Linux 4.15.0-54-generic” or
“Ubuntu, with Linux 4.15.0-55-generic”.

I’m fine continuing with what I’m currently doing, but I need to navigate to this option upon every boot, and if a family member were to attempt to use this laptop, they would probably experience the black screen and think they broke something.

I’ll wait until a fix comes out for this, until then I’ll wait and see if anthing else happens.

Anthony, I had that problem with fresh Lubuntu 16.04 on Toshiba Sattelite L300. I don't know why but after a while the problem disappeared. Try "sudo update-initramfs -ck all" and removing of "quiet" option from /etc/default/grub (and then "sudo update-grub").
And if "nopti" option doesn't help I think you should make a separate bug report.

I think this may be solved using the following fix: https://lists.ubuntu.com/archives/kernel-team/2019-July/thread.html

I've prepared some test kernel debs for you to download and install to see if this fixes the issue. The .deb packages can be found at: https://kernel.ubuntu.com/~cking/lp1827884/

I`ve tested kernel from: https://kernel.ubuntu.com/~cking/lp1827884/, the problems seems to be fixed, but what about kernel version 4.15.0?

I've uploaded a 4.15.0 kernel with the same patches applied: https://kernel.ubuntu.com/~cking/lp1827884/4.15

I`ve tested 4.15.0 kernel from: https://kernel.ubuntu.com/~cking/lp1827884/4.15, the problem seems to be fixed.

Dima, in comment #75 you mentioned that you experienced issues with Xenial. What kernel version were you running? We believe this issue is tied to PTI for 32-bit but that feature was only backported to the 4.15 kernel, not the 4.4. kernel.

Juerg, I used xenial-hwe kernel (4.15). Kernel 4.4 works fine.

Why this report is a duplicate of https://bugs.launchpad.net/bugs/1838115 ? That report is more recent than this one.

I've tested https://kernel.ubuntu.com/~cking/lp1827884/4.15 on xenial. It works. Xorg starts too.

user@user:~$ cat /proc/cmdline
BOOT_IMAGE=/@/boot/vmlinuz-4.15.0-57-generic root=UUID=50c3a930-a3e9-4444-b4ea-1646d276c7c6 ro rootflags=subvol=@ ipv6.disable=1 zswap.enabled=0 raid=noautodetect priority=low video=SVIDEO-1:d module_blacklist=r8169,mii,msr,jfs,xfs,bluetooth,hfs,hfsplus,ufs,minix,ntfs,joydev,mac_hid
user@user:~$
user@user:~$ uname -a
Linux user 4.15.0-57-generic #63~lp1827884 SMP Mon Jul 29 15:10:18 UTC 2019 i686 i686 i686 GNU/Linux
user@user:~$

Dima, the tracking part of this problem is a little misleading. The Xenial 4.15 HWE kernel is basically the Bionic 4.15 kernel recompiled on Xenial and we track those problems only for the Bionic series (even though he problem technically showed up in Xenial). That's why I removed the Xenial tags and references.

Also, although 1838115 is newer, we marked this bug as duplicate of 1838115 because we fixed 1838115 and the changelog for that fix references that bug. We just later learned that it also fixes this issue.

I didn't know it. Thank you.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

So we use this tags for repositories only? I thought it can be used for a fix in the bug report thread too.

This bug was fixed in the package linux - 4.15.0-58.64

---------------
linux (4.15.0-58.64) bionic; urgency=medium

  * unable to handle kernel NULL pointer dereference at 000000000000002c (IP:
    iget5_locked+0x9e/0x1f0) (LP: #1838982)
    - Revert "ovl: set I_CREATING on inode being created"
    - Revert "new primitive: discard_new_inode()"

linux (4.15.0-57.63) bionic; urgency=medium

  * CVE-2019-1125
    - x86/cpufeatures: Carve out CQM features retrieval
    - x86/cpufeatures: Combine word 11 and 12 into a new scattered features word
    - x86/speculation: Prepare entry code for Spectre v1 swapgs mitigations
    - x86/speculation: Enable Spectre v1 swapgs mitigations
    - x86/entry/64: Use JMP instead of JMPQ
    - x86/speculation/swapgs: Exclude ATOMs from speculation through SWAPGS

  * Packaging resync (LP: #1786013)
    - update dkms package versions

linux (4.15.0-56.62) bionic; urgency=medium

  * bionic/linux: 4.15.0-56.62 -proposed tracker (LP: #1837626)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log
    - [Packaging] update helper scripts

  * CVE-2019-2101
    - media: uvcvideo: Fix 'type' check leading to overflow

  * hibmc-drm Causes Unreadable Display for Huawei amd64 Servers (LP: #1762940)
    - [Config] Set CONFIG_DRM_HISI_HIBMC to arm64 only
    - SAUCE: Make CONFIG_DRM_HISI_HIBMC depend on ARM64

  * Bionic: support for Solarflare X2542 network adapter (sfc driver)
    (LP: #1836635)
    - sfc: make mem_bar a function rather than a constant
    - sfc: support VI strides other than 8k
    - sfc: add Medford2 (SFC9250) PCI Device IDs
    - sfc: improve PTP error reporting
    - sfc: update EF10 register definitions
    - sfc: populate the timer reload field
    - sfc: update MCDI protocol headers
    - sfc: support variable number of MAC stats
    - sfc: expose FEC stats on Medford2
    - sfc: expose CTPIO stats on NICs that support them
    - sfc: basic MCDI mapping of 25/50/100G link speeds
    - sfc: support the ethtool ksettings API properly so that 25/50/100G works
    - sfc: add bits for 25/50/100G supported/advertised speeds
    - sfc: remove tx and MCDI handling from NAPI budget consideration
    - sfc: handle TX timestamps in the normal data path
    - sfc: add function to determine which TX timestamping method to use
    - sfc: use main datapath for HW timestamps if available
    - sfc: only enable TX timestamping if the adapter is licensed for it
    - sfc: MAC TX timestamp handling on the 8000 series
    - sfc: on 8000 series use TX queues for TX timestamps
    - sfc: only advertise TX timestamping if we have the license for it
    - sfc: simplify RX datapath timestamping
    - sfc: support separate PTP and general timestamping
    - sfc: support second + quarter ns time format for receive datapath
    - sfc: support Medford2 frequency adjustment format
    - sfc: add suffix to large constant in ptp
    - sfc: mark some unexported symbols as static
    - sfc: update MCDI protocol headers
    - sfc: support FEC configuration through ethtool
    - sfc: remove ctpio_dmabuf_start from stats
    - sfc: stop the TX queue before pushing new buffers

  * [18.04 FEAT] zKVM: Add hardwar...