smpboot: don't call topology_sane() when Sub-NUMA-Clustering is enabled
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| linux (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
| Xenial |
Fix Released
|
Medium
|
Matthew Ruffell | ||
| Bionic |
Fix Released
|
Medium
|
Matthew Ruffell | ||
Bug Description
BugLink: https:/
[Impact]
Intel Skylake server processors and onward have a different Last Level Cache (LLC) topology than earlier processors, and such processors have a new feature called Sub-NUMA-Clustering (SNC) which is similar to the existing Cluster-On-Die (CoD) feature earlier server processors has.
Sub-NUMA-Clustering divides the system into two "slices", each of which are allocated half the CPU cores, half the Last Level Cache and one memory controller. Each slice is enumerated as a NUMA node.
The difference between Sub-NUMA-Clustering and Cluster-On-Die is how the Last Level Cache is exposed to each NUMA node. CoD had the same cache line present in each half of the LLC. In SNC, each cache line is only present in its respective slice. Because of this, the semantics around accessing LLC changes, with a process accessing NUMA-local memory only seeing half the LLC capacity.
On systems with Sub-NUMA-Clustering enabled, on the Xenial 4.4 and Bionic 4.15 kernels we see the following oops during NUMA node enumeration:
.... node #0, CPUs: #1 #2 #3 #4 #5 #6
.... node #1, CPUs: #7
sched: CPU #7's llc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency.
WARNING: CPU: 7 PID: 0 at /build/
Modules linked in:
CPU: 7 PID: 0 Comm: swapper/7 Not tainted 4.15.0-47-generic #50~16.04.1-Ubuntu
Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 10/02/2018
RIP: 0010:topology_
Call Trace:
set_cpu_
start_secondary
secondary_
#8 #9 #10 #11 #12 #13
.... node #0, CPUs: #14 #15 #16 #17 #18 #19 #20
.... node #1, CPUs: #21 #22 #23 #24 #25 #26 #27
smp: Brought up 2 nodes, 28 CPUs
This was with a Intel Xeon Gold 5120 CPU on a HP DL360 Gen10.
The oops happens because topology_sane() checks to see if the Last Level Cache line matches across different CPUs, which it no longer does.
[Fix]
The fix comes in the form of the following upstream commit, which landed in Linux 4.17:
commit 1340ccfa9a9afef
Author: Alison Schofield <email address hidden>
Date: Fri Apr 6 17:21:30 2018 -0700
Subject: x86,sched: Allow topologies where NUMA nodes share an LLC
Link: https:/
The commit adds a check for this particular family of Intel processors, and if the CPU family matches, it simply skips the check to topology_sane().
The commit needs minor backports to Xenial 4.4 and Bionic 4.15, with the only remarks being re-arranging #includes and small context fixups.
[Testcase]
Unfortunately, this is hardware specific. To test this, you need a Intel Skylake server processor which supports Sub-NUMA-
We have a customer with a Intel Xeon Gold 5120 CPU on a HP DL360 Gen10 that has successfully tested the below test kernels, with good results.
Xenial 4.4 ppa:
https:/
Xenial 4.15 HWE ppa:
https:/
Running the test kernel, the oops does not reproduce:
smp: Bringing up secondary CPUs ...
x86: Booting SMP configuration:
.... node #0, CPUs: #1
NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
#2 #3 #4 #5 #6
.... node #1, CPUs: #7 #8 #9 #10 #11 #12 #13
.... node #0, CPUs: #14 #15 #16 #17 #18 #19 #20
.... node #1, CPUs: #21 #22 #23 #24 #25 #26 #27
smp: Brought up 2 nodes, 28 CPUs
smpboot: Max logical packages: 1
smpboot: Total of 28 processors activated
[Regression Potential]
The commit modifies a small section of smpboot code, which every machine will execute on boot. The majority of the commit breaks up a large if statement into smaller blocks than it was previously, and adds an extra if statement to check for a specific processor family.
If a regression were to occur, some machines would or would not make their calls to topology_sane(), which in the worst case, would result in a oops message and slightly degraded performance. The system would still function normally.
The commit has been present since 4.17-rc2 and is present in Eoan and Focal. There are no fixup commits, and no additional processor families have been added since.
Because of the small re-arrangement in logic, and the addition of a processor family check, these changes are fairly minor, and I don't think it will cause any regressions.
CVE References
| Changed in linux (Ubuntu): | |
| status: | New → Fix Released |
| Changed in linux (Ubuntu Xenial): | |
| status: | New → In Progress |
| Changed in linux (Ubuntu Bionic): | |
| status: | New → In Progress |
| Changed in linux (Ubuntu Xenial): | |
| importance: | Undecided → Medium |
| Changed in linux (Ubuntu Bionic): | |
| importance: | Undecided → Medium |
| Changed in linux (Ubuntu Xenial): | |
| assignee: | nobody → Matthew Ruffell (mruffell) |
| Changed in linux (Ubuntu Bionic): | |
| assignee: | nobody → Matthew Ruffell (mruffell) |
| tags: | added: sts |
| description: | updated |
| description: | updated |
| description: | updated |
| Changed in linux (Ubuntu Xenial): | |
| status: | In Progress → Fix Committed |
| Changed in linux (Ubuntu Bionic): | |
| status: | In Progress → Fix Committed |
| Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote | #1 |
| tags: | added: verification-needed-xenial |
| Matthew Ruffell (mruffell) wrote | #2 |
| tags: |
added: verification-done-xenial removed: verification-needed-xenial |
| Launchpad Janitor (janitor) wrote | #3 |
| Changed in linux (Ubuntu Xenial): | |
| status: | Fix Committed → Fix Released |
| Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote | #4 |
| tags: | added: verification-needed-bionic |
| Matthew Ruffell (mruffell) wrote | #5 |
| tags: |
added: verification-done-bionic removed: verification-needed-bionic |
| Launchpad Janitor (janitor) wrote | #6 |
| Changed in linux (Ubuntu Bionic): | |
| status: | Fix Committed → Fix Released |
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/