suspend only works once on ThinkPad X1 Carbon gen 7

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1865570

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Looks like it is an issue on the e1000e driver

[ 909.916364] PM: pci_pm_suspend(): e1000e_pm_suspend+0x0/0x80 [e1000e] returns -2
[ 909.916367] PM: dpm_run_callback(): pci_pm_suspend+0x0/0x150 returns -2
[ 909.916369] PM: Device 0000:00:1f.6 failed to suspend async: error -2
[ 911.183052] PM: Some devices failed to suspend, or early wake event detected

Please test latest mainline kernel:
https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.6-rc4/

5.6.0-050600rc4-generic
Same behaviour:

First time works:
[ 122.113542] PM: suspend entry (s2idle)
[ 122.118823] Filesystems sync: 0.005 seconds
[ 122.120954] Freezing user space processes ... (elapsed 0.002 seconds) done.
[ 122.123333] OOM killer disabled.
[ 122.123334] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[ 122.124606] printk: Suspending console(s) (use no_console_suspend to debug)
[ 122.125072] wlp0s20f3: deauthenticating from 18:d6:c7:61:95:c2 by local choice (Reason: 3=DEAUTH_LEAVING)
[ 122.230038] e1000e: EEE TX LPI TIMER: 00000011
[ 122.345108] ACPI: EC: interrupt blocked
[ 138.117673] ACPI: EC: interrupt unblocked
[ 140.917449] iwlwifi 0000:00:14.3: Applying debug destination EXTERNAL_DRAM
[ 141.031854] iwlwifi 0000:00:14.3: Applying debug destination EXTERNAL_DRAM
[ 141.096775] iwlwifi 0000:00:14.3: FW already configured (0) - re-configuring
[ 143.796468] e1000e 0000:00:1f.6 enp0s31f6: PHY Wakeup cause - Unicast Packet
[ 144.466557] e1000e 0000:00:1f.6 enp0s31f6: Hardware Error
[ 145.195597] OOM killer enabled.
[ 145.195597] Restarting tasks ... done.
[ 145.213016] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915])
[ 145.239043] PM: suspend exit

Second time fails to suspend:
[ 169.696561] PM: suspend entry (s2idle)
[ 169.734111] Filesystems sync: 0.037 seconds
[ 169.734688] Freezing user space processes ... (elapsed 0.002 seconds) done.
[ 169.736790] OOM killer disabled.
[ 169.736791] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[ 169.738032] printk: Suspending console(s) (use no_console_suspend to debug)
[ 169.738318] wlp0s20f3: deauthenticating from 18:d6:c7:61:95:c2 by local choice (Reason: 3=DEAUTH_LEAVING)
[ 169.839564] e1000e: EEE TX LPI TIMER: 00000011
[ 171.730682] PM: pci_pm_suspend(): e1000e_pm_suspend+0x0/0x80 [e1000e] returns -2
[ 171.730685] PM: dpm_run_callback(): pci_pm_suspend+0x0/0x170 returns -2
[ 171.730687] PM: Device 0000:00:1f.6 failed to suspend async: error -2
[ 172.997917] PM: Some devices failed to suspend, or early wake event detected
[ 173.010160] iwlwifi 0000:00:14.3: Applying debug destination EXTERNAL_DRAM
[ 173.126554] iwlwifi 0000:00:14.3: Applying debug destination EXTERNAL_DRAM
[ 173.193691] iwlwifi 0000:00:14.3: FW already configured (0) - re-configuring
[ 173.222986] OOM killer enabled.
[ 173.222987] Restarting tasks ...
[ 173.235209] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915])
[ 173.235267] done.
[ 173.266890] PM: suspend exit

Suspend works with eoan's 5.3.0-18-generic (same system, just booted into the old kernel):

First time successful:
[ 66.725822] PM: suspend entry (s2idle)
[ 66.731748] Filesystems sync: 0.005 seconds
[ 66.732769] Freezing user space processes ... (elapsed 0.002 seconds) done.
[ 66.735102] OOM killer disabled.
[ 66.735103] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[ 66.736358] printk: Suspending console(s) (use no_console_suspend to debug)
[ 66.737562] wlp0s20f3: deauthenticating from 18:d6:c7:61:95:c2 ...

Could the culprit be enabled "wake on LAN" functionality of e1000e, that was not enabled in 5.3?

Would it be possible for you to do a kernel bisection?

First, find the last -rc kernel works and the first -rc kernel doesn’t work from http://kernel.ubuntu.com/~kernel-ppa/mainline/

Then,
$ sudo apt build-dep linux
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
$ cd linux
$ git bisect start
$ git bisect good $(the working version you found)
$ git bisect bad $(the non-working version found)
$ make localmodconfig
$ make -j`nproc` deb-pkg
Install the newly built kernel, then reboot with it.
If it still have the same issue,
$ git bisect bad
Otherwise,
$ git bisect good
Repeat to "make -j`nproc` deb-pkg" until you find the offending commit.

@kaihengfeng, unfortunately that does not work.
I could not find time to do git bisect (each kernel build takes a long time), but it would not help if I did. I did a "bisect" over the packages in https://kernel.ubuntu.com/~kernel-ppa/mainline/. The problem is, between the version with working suspend and the version with suspend that only works once, there are several version in which suspend instantly hangs the system:

5.4.24-050424-generic suspend works
5.5.0-050500rc1-generic kernel freeze on the first attempt to suspend
5.5.0-050500rc4-generic kernel freeze on the first attempt to suspend
5.5.0-050500rc5-generic kernel freeze on the first attempt to suspend
5.5.0-050500rc6-generic kernel freeze on the first attempt to suspend
5.5.0-050500rc7-generic suspend works only the first time

Some system does not work well with s2idle, but they might work with regular suspend (s3).

Please try "echo deep | sudo tee /sys/power/mem_sleep" to switch to s3, and verify whether this is s2idle related or a power management issue with e1000e driver.

5.4.0-16/17 are abandoned for issues of s2idle, please ignore it.

And for e1000e errors, not reproduced on my X1 Carbon 7.

Please update the kernel.

If you still reproduce this issue, lease test the latest e1000e driver from intel:
https://downloadcenter.intel.com/download/15817

Please file a bug in bugzilla.kernel.org too if driver above failed too.

And for mainline kernel suspend hang that is because of it lacks of audio sof supports.

> 5.4.0-16/17 are abandoned for issues of s2idle, please ignore it.

I do not understand that.

With the current "official" kernel, the problem still exists:
5.4.0-17-generic suspend works only the first time

The only kernel that works for me is 5.4.24-050424-generic from https://kernel.ubuntu.com/~kernel-ppa/mainline/

@crosser,

5.4.0-17 is a Ubuntu kernel version.
5.4.24 is mainline stable kernel version.

If 5.4.24 is good, that means it is a regression of Ubuntu kernel 5.4.0-17 who is from proposed kernel not a official release in updates channel.

Now Ubuntu kernel 5.4.0-18 has replaced 5.4.0-17 in proposed channel.
You can try it first.

May blame to this patch:
https://lists.ubuntu.com/archives/kernel-team/2020-January/106800.html

JFYI, with the current "proposed" kernel behaviour is the same (problematic):
5.4.0-18-generic suspend works only the first time

To check the theory from comment #13, I built kernel from mainline at tag v5.6-rc5. Out of the box, it behaves the same way as all recent kernels: suspend works only the first time. I could not revert commit f15bb6d (there were later changes), instead I commented out the calls to e1000e_s0ix_entry_flow() and e1000e_s0ix_exit_flow()

diff --git a/drivers/net/ethernet/intel/e1000e/netdev.c b/drivers/net/ethernet/intel/e1000e/netdev.c
index db4ea58bac82..53f8ed97d282 100644
--- a/drivers/net/ethernet/intel/e1000e/netdev.c
+++ b/drivers/net/ethernet/intel/e1000e/netdev.c
@@ -6860,8 +6860,8 @@ static int e1000e_pm_suspend(struct device *dev)
   e1000e_pm_thaw(dev);

  /* Introduce S0ix implementation */
- if (hw->mac.type >= e1000_pch_cnp)
- e1000e_s0ix_entry_flow(adapter);
+ /*if (hw->mac.type >= e1000_pch_cnp)
+ e1000e_s0ix_entry_flow(adapter);*/

  return rc;
 }
@@ -6875,8 +6875,8 @@ static int e1000e_pm_resume(struct device *dev)
  int rc;

  /* Introduce S0ix implementation */
- if (hw->mac.type >= e1000_pch_cnp)
- e1000e_s0ix_exit_flow(adapter);
+ /* if (hw->mac.type >= e1000_pch_cnp)
+ e1000e_s0ix_exit_flow(adapter);*/

  rc = __e1000_resume(pdev);
  if (rc)

And voila, suspend works normally every time now!
Apparently S0ix implementation is indeed the culprit.

A test kernel can be found here:
https://people.canonical.com/~khfeng/lp1865570/focal/

I can confirm that suspend works normally with the kernel from comment #16
Thank you!

https://lkml.org/lkml/2020/3/19/16

Thank you for going the extra mile in resolving this issue.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Hi

Please verify the issue with this patch:

https://<email address hidden>/

@mapengyu, I've rebuilt e1000e.ko with your patch applied to 5.6.0-rc5 tree, and suspend works normally for me:

Linux journex 5.6.0-rc5+ #2 SMP Sun Mar 15 00:09:19 CET 2020 x86_64 x86_64 x86_64 GNU/Linux

First suspend:
[ 66.578525] PM: suspend entry (s2idle)
[ 66.582870] Filesystems sync: 0.004 seconds
[ 66.583640] Freezing user space processes ... (elapsed 0.002 seconds) done.
[ 66.585841] OOM killer disabled.
[ 66.585842] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[ 66.587085] printk: Suspending console(s) (use no_console_suspend to debug)
[ 66.587375] wlp0s20f3: deauthenticating from 18:d6:c7:61:95:c2 by local choice (Reason: 3=DEAUTH_LEAVING)
[ 66.691969] e1000e: EEE TX LPI TIMER: 00000011
[ 69.395364] ACPI: EC: interrupt blocked
[ 79.252679] ACPI: EC: interrupt unblocked
[ 82.053040] iwlwifi 0000:00:14.3: Applying debug destination EXTERNAL_DRAM
[ 82.169363] iwlwifi 0000:00:14.3: Applying debug destination EXTERNAL_DRAM
[ 82.234288] iwlwifi 0000:00:14.3: FW already configured (0) - re-configuring
[ 85.251055] OOM killer enabled.
[ 85.251056] Restarting tasks ... done.
[ 85.260141] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915])
[ 85.293929] PM: suspend exit
[ 85.391753] e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Down
[ 85.627128] iwlwifi 0000:00:14.3: Applying debug destination EXTERNAL_DRAM
...

Second suspend:
[ 140.523471] PM: suspend entry (s2idle)
[ 140.554018] Filesystems sync: 0.030 seconds
[ 140.554655] Freezing user space processes ... (elapsed 0.002 seconds) done.
[ 140.556773] OOM killer disabled.
[ 140.556773] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[ 140.558014] printk: Suspending console(s) (use no_console_suspend to debug)
[ 140.558275] wlp0s20f3: deauthenticating from 18:d6:c7:61:95:c2 by local choice (Reason: 3=DEAUTH_LEAVING)
[ 140.659976] e1000e: EEE TX LPI TIMER: 00000011
[ 143.380606] ACPI: EC: interrupt blocked
[ 158.716081] ACPI: EC: interrupt unblocked
[ 161.516454] iwlwifi 0000:00:14.3: Applying debug destination EXTERNAL_DRAM
[ 161.866014] iwlwifi 0000:00:14.3: Applying debug destination EXTERNAL_DRAM
[ 161.933414] iwlwifi 0000:00:14.3: FW already configured (0) - re-configuring
[ 164.586442] OOM killer enabled.
[ 164.586443] Restarting tasks ...
[ 164.592488] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915])
[ 164.592743] done.
[ 164.637623] PM: suspend exit
[ 164.735062] e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Down
[ 164.999601] iwlwifi 0000:00:14.3: Applying debug destination EXTERNAL_DRAM
...

I've just tested on Eoan with the proposed branch.

It didn't resolve my issue. I'll give it some time today to see if the new kernel
gets released.

This is the kernel I'm running after an apt update/upgrade cycle this morning
Linux MiX 5.3.0-43-generic #36-Ubuntu SMP Mon Mar 16 13:29:01 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Using the kernel package from -proposed makes suspend work normally as well.

Linux journex 5.4.0-20-generic #24-Ubuntu SMP Mon Mar 23 20:55:46 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Though I guess that 5.4.0-20 uses the "workaround", and it may be better to apply the "proper fix" from comment 21?

In any case, I am marking it "verification-done" because it does indeed work.

About comment #23 It's won't work. I'm on Eoan, the proposed kernel is for focal.

It'll be in Eoan kernel when it's "Fix committed".

This bug was fixed in the package linux - 5.4.0-21.25

---------------
linux (5.4.0-21.25) focal; urgency=medium

  * CVE-2020-8835
    - SAUCE: bpf: undo incorrect __reg_bound_offset32 handling

 -- Thadeu Lima de Souza Cascardo <email address hidden> Thu, 26 Mar 2020 17:51:28 -0300

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-eoan' to 'verification-done-eoan'. If the problem still exists, change the tag 'verification-needed-eoan' to 'verification-failed-eoan'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Status changed to 'Confirmed' because the bug affects multiple users.

I've updated the tag to verification-done-eoan.
I was able to suspend three times in a row.

Specs:
ThinkPad X1C7 ubuntu 19.10 eoan

This bug was fixed in the package linux - 5.3.0-51.44

---------------
linux (5.3.0-51.44) eoan; urgency=medium

  * CVE-2020-11884
    - SAUCE: s390/mm: fix page table upgrade vs 2ndary address mode accesses

 -- Thadeu Lima de Souza Cascardo <email address hidden> Wed, 22 Apr 2020 17:35:41 -0300

This bug was fixed in the package linux-oem-osp1 - 5.0.0-1050.55

---------------
linux-oem-osp1 (5.0.0-1050.55) bionic; urgency=medium

  * bionic/linux-oem-osp1: 5.0.0-1050.55 -proposed tracker (LP: #1874611)

  * alsa/sof: external mic can't be deteced on Lenovo and HP laptops
    (LP: #1872569)
    - SAUCE: ASoC: intel/skl/hda - set autosuspend timeout for hda codecs

 -- Chia-Lin Kao (AceLan) <email address hidden> Fri, 24 Apr 2020 18:25:45 +0800

Tried on ThinkPad X13 Yoga with 5.0.0-1050.55 kernel on bionic, this issue is not fixed.

Bin, can you please file a separate bug? I plan to pull Aaron's more generic fix.

This bug was fixed in the package linux-oem-osp1 - 5.0.0-1059.64

---------------
linux-oem-osp1 (5.0.0-1059.64) bionic; urgency=medium

  [ Ubuntu: 5.0.0-52.56 ]

  * CVE-2020-0543
    - SAUCE: x86/cpu: Add a steppings field to struct x86_cpu_id
    - SAUCE: x86/cpu: Add 'table' argument to cpu_matches()
    - SAUCE: x86/speculation: Add Special Register Buffer Data Sampling (SRBDS)
      mitigation
    - SAUCE: x86/speculation: Add SRBDS vulnerability and mitigation documentation
    - SAUCE: x86/speculation: Add Ivy Bridge to affected list

linux-oem-osp1 (5.0.0-1054.59) bionic; urgency=medium

  * bionic/linux-oem-osp1: 5.0.0-1054.59 -proposed tracker (LP: #1881221)

  * Speed up the process of s3 resume (LP: #1876844)
    - PCI/PM: Adjust pcie_wait_for_link_delay() for caller delay
    - PCI/PM: Assume ports without DLL Link Active train links in 100 ms

linux-oem-osp1 (5.0.0-1053.58) bionic; urgency=medium

  * bionic/linux-oem-osp1: 5.0.0-1053.58 -proposed tracker (LP: #1878822)

  * rtl8723bu wifi issue after being turned off (LP: #1878296)
    - rtl8xxxu: Improve TX performance of RTL8723BU on rtl8xxxu driver
    - rtl8xxxu: add bluetooth co-existence support for single antenna
    - rtl8xxxu: remove set but not used variable 'rate_mask'
    - rtl8xxxu: Remove set but not used variable 'vif', 'dev', 'len'

  * Fix Pericom USB controller OHCI/EHCI PME# defect (LP: #1879321)
    - serial: 8250_pci: Move Pericom IDs to pci_ids.h
    - PCI: Avoid Pericom USB controller OHCI/EHCI PME# defect

  * Don't register platform::micmute if the related tokens don't exist
    (LP: #1877275)
    - SAUCE: platform/x86: dell-laptop: don't register platform::micmute if the
      related tokens don't exist

  [ Ubuntu: 5.0.0-49.53 ]

  * disco/linux: 5.0.0-49.53 -proposed tracker (LP: #1878826)
  * Disco update: upstream stable patchset 2020-05-14 (LP: #1878681)
    - remoteproc: Fix wrong rvring index computation
    - mtd: cfi: fix deadloop in cfi_cmdset_0002.c do_write_buffer
    - binder: take read mode of mmap_sem in binder_alloc_free_page()
    - usb: dwc3: gadget: Do link recovery for SS and SSP
    - usb: gadget: udc: bdc: Remove unnecessary NULL checks in bdc_req_complete
    - iio:ad7797: Use correct attribute_group
    - ASoC: q6dsp6: q6afe-dai: add missing channels to MI2S DAIs
    - ASoC: tas571x: disable regulators on failed probe
    - ASoC: wm8960: Fix wrong clock after suspend & resume
    - nfsd: memory corruption in nfsd4_lock()
    - i2c: altera: use proper variable to hold errno
    - rxrpc: Fix DATA Tx to disable nofrag for UDP on AF_INET6 socket
    - net/cxgb4: Check the return from t4_query_params properly
    - xfs: acquire superblock freeze protection on eofblocks scans
    - svcrdma: Fix trace point use-after-free race
    - svcrdma: Fix leak of svc_rdma_recv_ctxt objects
    - PCI: Avoid ASMedia XHCI USB PME# from D0 defect
    - PCI: Move Apex Edge TPU class quirk to fix BAR assignment
    - ARM: dts: bcm283x: Disable dsi0 node
    - cpumap: Avoid warning when CONFIG_DEBUG_PER_CPU_MAPS is enabled
    - net/mlx5: Fix failing fw tracer allocation on s390
    - perf/core: fix parent pid/tid in task exit events
    - bpf, x86_3...

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-groovy' to 'verification-done-groovy'. If the problem still exists, change the tag 'verification-needed-groovy' to 'verification-failed-groovy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

I understand that it's fixed in the mainline quite a while ago, no?
Don't get it why is "verification-needed" now...

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-groovy' to 'verification-done-groovy'. If the problem still exists, change the tag 'verification-needed-groovy' to 'verification-failed-groovy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!