net/mlx5e: EPERM on vlan 0 programming

Bug #1957753 reported by Dmitrii Shcherbakov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Undecided
Po-Hsu Lin
Focal
Fix Released
Medium
Po-Hsu Lin
Hirsute
Won't Fix
Undecided
Po-Hsu Lin
Impish
Fix Released
Medium
Po-Hsu Lin
Jammy
Fix Released
Undecided
Po-Hsu Lin

Bug Description

[Impact]
There is an issue with hypervisor host side mlx5 driver operation on
Bluefield devices in kernels <=5.16

From patch description:
When using libvirt to passthrough VF to VM it will always set the VF vlan
to 0 even if user didn’t request it, this will cause libvirt to fail to
boot in case the PF isn't eswitch owner.

Example of such case is the DPU host PF which isn't eswitch manager, so
any attempt to passthrough VF of it using libvirt will fail.

[Fix]
* 7846665d net/mlx5e: Unblock setting vid 0 for VF in case PF isn't
 eswitch manager

This patch can be cherry-picked into Impish and newer kernels, but it
requires some backport work on F/H due to they're missing commit
b55b3538 that split the legacy code out.

The ESW_ALLOWED marco was later converted into a helper routine in
commit b16f2bb6.

[Test]
Test kernels can be found here:
F: https://people.canonical.com/~phlin/kernel/lp-1957753-mlx5e/F/
H: https://people.canonical.com/~phlin/kernel/lp-1957753-mlx5e/H/
I: https://people.canonical.com/~phlin/kernel/lp-1957753-mlx5e/I/
J: https://people.canonical.com/~phlin/kernel/lp-1957753-mlx5e/J/

Kernels tested on a SUT with this device by Dmitrii Shcherbakov, test
passed with positive feedback.

[Where problems could occur]
If this patch is erroneous, it might cause issue to this mlx5 driver
with certain operations (setting vid 0). And other programs such as
libvirt that depends on this return value might be affected as well.

[Original Bug Description]
There is an issue with hypervisor host side mlx5 driver operation on Bluefield devices in kernels <=5.16 that was recently fixed in master with a one-liner:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7846665d3504812acaebf920d1141851379a7f37

It would be good to have this fix in Focal+ kernels.

This is not needed in https://bugs.launchpad.net/ubuntu/+source/linux-bluefield since it's used at the DPU side whereas the issue in question affects the hypervisor side.

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1957753

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Hello Dmitrii,
can you give these test kernel a try to see if they are working as expected?
F: https://people.canonical.com/~phlin/kernel/lp-1957753-mlx5e/F/
H: https://people.canonical.com/~phlin/kernel/lp-1957753-mlx5e/H/
I: https://people.canonical.com/~phlin/kernel/lp-1957753-mlx5e/I/
J: https://people.canonical.com/~phlin/kernel/lp-1957753-mlx5e/J/

This patch can be cherry-picked into I/J, but required to be backported to F/H

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Note that it seems the current SRU cycle is the last one for Hirsute, so it's very likely that it won't get this fix.

Changed in linux (Ubuntu Impish):
status: New → Incomplete
Changed in linux (Ubuntu Focal):
status: New → Incomplete
Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

Focal:

$ uname -r
5.4.0-97-generic

$ sudo ip link set enp130s0f0 vf 2 vlan 1
RTNETLINK answers: Operation not permitted

$ sudo ip link set enp130s0f0 vf 2 vlan 0 ; echo $?
0

Looks good - vlan 0 programming doesn't result in EPERM as expected contrary to non-zero vlans.

Will try others as well. It would be good to backport it to currently released versions above Focal (for production, we are mostly looking for Focal GA + HWE kernels but it keeping it fixed consistently seems like the right approach).

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

Hirsute kernel looks good as well:

$ uname -r
5.11.0-50-generic

$ sudo ip link set enp130s0f0 vf 2 vlan 1 ; echo $?
RTNETLINK answers: Operation not permitted
2

$ sudo ip link set enp130s0f0 vf 2 vlan 0 ; echo $?
0

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

Impish kernel looks good:

$ uname -r
5.13.0-28-generic

$ sudo ip link set enp130s0f0 vf 2 vlan 1 ; echo $?
RTNETLINK answers: Operation not permitted
2

$ sudo ip link set enp130s0f0 vf 2 vlan 0 ; echo $?
0

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

And, finally, the Jammy kernel:

$ uname -r
5.15.0-17-generic

$ sudo ip link set enp130s0f0 vf 2 vlan 0 ; echo $?
0

$ sudo ip link set enp130s0f0 vf 2 vlan 1 ; echo $?
RTNETLINK answers: Operation not permitted
2

Changed in linux (Ubuntu Focal):
status: Incomplete → Confirmed
Changed in linux (Ubuntu Impish):
status: Incomplete → Confirmed
Changed in linux (Ubuntu Jammy):
status: Incomplete → Confirmed
Changed in linux (Ubuntu Hirsute):
status: New → Confirmed
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Awesome!
I will send the SRU request out, thanks for testing!

Po-Hsu Lin (cypressyew)
description: updated
Po-Hsu Lin (cypressyew)
description: updated
Po-Hsu Lin (cypressyew)
description: updated
Changed in linux (Ubuntu Focal):
assignee: nobody → Po-Hsu Lin (cypressyew)
Changed in linux (Ubuntu Hirsute):
assignee: nobody → Po-Hsu Lin (cypressyew)
Changed in linux (Ubuntu Jammy):
assignee: nobody → Po-Hsu Lin (cypressyew)
Changed in linux (Ubuntu Impish):
assignee: nobody → Po-Hsu Lin (cypressyew)
Changed in linux (Ubuntu Focal):
status: Confirmed → In Progress
Changed in linux (Ubuntu Hirsute):
status: Confirmed → In Progress
Changed in linux (Ubuntu Impish):
status: Confirmed → In Progress
Changed in linux (Ubuntu Jammy):
status: Confirmed → In Progress
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :
description: updated
Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

Thanks!

Stefan Bader (smb)
Changed in linux (Ubuntu Hirsute):
status: In Progress → Won't Fix
Changed in linux (Ubuntu Impish):
importance: Undecided → Medium
status: In Progress → Fix Committed
Changed in linux (Ubuntu Focal):
importance: Undecided → Medium
Stefan Bader (smb)
Changed in linux (Ubuntu Focal):
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.13.0-29.32 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-impish' to 'verification-done-impish'. If the problem still exists, change the tag 'verification-needed-impish' to 'verification-failed-impish'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-impish
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.4.0-98.111 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

Tried the proposed impish kernel - looks good:

$ uname -r
5.13.0-29-generic

$ apt policy linux-image-5.13.0-29-generic
linux-image-5.13.0-29-generic:
  Installed: 5.13.0-29.32
  Candidate: 5.13.0-29.32
  Version table:
 *** 5.13.0-29.32 500
        500 http://ppa.launchpad.net/canonical-kernel-team/proposed/ubuntu impish/main amd64 Packages
        100 /var/lib/dpkg/status

$ readlink /sys/class/net/enp130s0f0v0/device
../../../0000:82:00.3

# VF
$ sudo lspci -s 0000:82:00.3 -v
82:00.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
 Subsystem: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
 Flags: bus master, fast devsel, latency 0, NUMA node 1
 Memory at 3c004000000 (64-bit, prefetchable) [virtual] [size=2M]
 Capabilities: [60] Express Endpoint, MSI 00
 Capabilities: [48] Vital Product Data
 Capabilities: [9c] MSI-X: Enable+ Count=12 Masked-
 Capabilities: [100] Vendor Specific Information: ID=0000 Rev=0 Len=00c <?>
 Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
 Kernel driver in use: mlx5_core
 Kernel modules: mlx5_core

# Parent PF
$ sudo lspci -s 0000:82:00.0 -v
82:00.0 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01)
 Subsystem: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller
 Flags: bus master, fast devsel, latency 0, IRQ 101, NUMA node 1
 Memory at 3c002000000 (64-bit, prefetchable) [size=32M]
 Expansion ROM at <ignored> [disabled]
 Capabilities: [60] Express Endpoint, MSI 00
 Capabilities: [48] Vital Product Data
 Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
 Capabilities: [c0] Vendor Specific Information: Len=18 <?>
 Capabilities: [40] Power Management version 3
 Capabilities: [100] Advanced Error Reporting
 Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
 Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
 Capabilities: [1c0] Secondary PCI Express
 Capabilities: [230] Access Control Services
 Capabilities: [320] Lane Margining at the Receiver <?>
 Capabilities: [370] Physical Layer 16.0 GT/s <?>
 Capabilities: [420] Data Link Feature <?>
 Kernel driver in use: mlx5_core
 Kernel modules: mlx5_core

$ sudo ip link set enp130s0f0 vf 0 vlan 1 ; echo $?
RTNETLINK answers: Operation not permitted
2

$ sudo ip link set enp130s0f0 vf 0 vlan 0 ; echo $?
0

tags: added: verification-done-impish
removed: verification-needed-impish
Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

Tested the proposed focal kernel - looks good as well:

$ uname -r
5.4.0-98-generic

$ apt policy linux-image-5.4.0-98-generic
linux-image-5.4.0-98-generic:
  Installed: 5.4.0-98.111
  Candidate: 5.4.0-98.111
  Version table:
 *** 5.4.0-98.111 500
        500 http://ppa.launchpad.net/canonical-kernel-team/proposed/ubuntu focal/main amd64 Packages
        100 /var/lib/dpkg/status

$ readlink /sys/class/net/enp130s0f0v0/device
../../../0000:82:00.3
$ sudo lspci -s 0000:82:00.3 -v
82:00.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
 Subsystem: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function
 Flags: bus master, fast devsel, latency 0, NUMA node 1
 Memory at 3c004000000 (64-bit, prefetchable) [virtual] [size=2M]
 Capabilities: [60] Express Endpoint, MSI 00
 Capabilities: [48] Vital Product Data
 Capabilities: [9c] MSI-X: Enable+ Count=12 Masked-
 Capabilities: [100] Vendor Specific Information: ID=0000 Rev=0 Len=00c <?>
 Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
 Kernel driver in use: mlx5_core
 Kernel modules: mlx5_core

$ sudo lspci -s 0000:82:00.0 -v
82:00.0 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01)
 Subsystem: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller
 Flags: bus master, fast devsel, latency 0, IRQ 99, NUMA node 1
 Memory at 3c002000000 (64-bit, prefetchable) [size=32M]
 Expansion ROM at <ignored> [disabled]
 Capabilities: [60] Express Endpoint, MSI 00
 Capabilities: [48] Vital Product Data
 Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
 Capabilities: [c0] Vendor Specific Information: Len=18 <?>
 Capabilities: [40] Power Management version 3
 Capabilities: [100] Advanced Error Reporting
 Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
 Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
 Capabilities: [1c0] Secondary PCI Express
 Capabilities: [230] Access Control Services
 Capabilities: [320] Lane Margining at the Receiver <?>
 Capabilities: [370] Physical Layer 16.0 GT/s <?>
 Capabilities: [420] Data Link Feature <?>
 Kernel driver in use: mlx5_core
 Kernel modules: mlx5_core

$ sudo ip link set enp130s0f0 vf 0 vlan 1 ; echo $?
RTNETLINK answers: Operation not permitted
2

$ sudo ip link set enp130s0f0 vf 0 vlan 0 ; echo $?
0

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for linux-hwe-5.13 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (16.6 KiB)

This bug was fixed in the package linux - 5.4.0-100.113

---------------
linux (5.4.0-100.113) focal; urgency=medium

  * focal/linux: 5.4.0-100.113 -proposed tracker (LP: #1959900)

  * CVE-2022-22942
    - SAUCE: drm/vmwgfx: Fix stale file descriptors on failed usercopy

  * CVE-2022-0330
    - drm/i915: Flush TLBs before releasing backing store

  * Focal update: v5.4.166 upstream stable release (LP: #1957008)
    - netfilter: selftest: conntrack_vrf.sh: fix file permission
    - Linux 5.4.166
    - net/packet: rx_owner_map depends on pg_vec
    - USB: gadget: bRequestType is a bitfield, not a enum
    - HID: holtek: fix mouse probing
    - udp: using datalen to cap ipv6 udp max gso segments
    - selftests: Calculate udpgso segment count without header adjustment

  * Focal update: v5.4.165 upstream stable release (LP: #1957007)
    - serial: tegra: Change lower tolerance baud rate limit for tegra20 and
      tegra30
    - ntfs: fix ntfs_test_inode and ntfs_init_locked_inode function type
    - HID: quirks: Add quirk for the Microsoft Surface 3 type-cover
    - HID: google: add eel USB id
    - HID: add hid_is_usb() function to make it simpler for USB detection
    - HID: add USB_HID dependancy to hid-prodikeys
    - HID: add USB_HID dependancy to hid-chicony
    - HID: add USB_HID dependancy on some USB HID drivers
    - HID: bigbenff: prevent null pointer dereference
    - HID: wacom: fix problems when device is not a valid USB device
    - HID: check for valid USB device for many HID drivers
    - can: kvaser_usb: get CAN clock frequency from device
    - can: kvaser_pciefd: kvaser_pciefd_rx_error_frame(): increase correct
      stats->{rx,tx}_errors counter
    - can: sja1000: fix use after free in ems_pcmcia_add_card()
    - nfc: fix potential NULL pointer deref in nfc_genl_dump_ses_done
    - selftests: netfilter: add a vrf+conntrack testcase
    - vrf: don't run conntrack on vrf with !dflt qdisc
    - bpf: Fix the off-by-two error in range markings
    - ice: ignore dropped packets during init
    - bonding: make tx_rebalance_counter an atomic
    - nfp: Fix memory leak in nfp_cpp_area_cache_add()
    - seg6: fix the iif in the IPv6 socket control block
    - udp: using datalen to cap max gso segments
    - iavf: restore MSI state on reset
    - iavf: Fix reporting when setting descriptor count
    - IB/hfi1: Correct guard on eager buffer deallocation
    - mm: bdi: initialize bdi_min_ratio when bdi is unregistered
    - ALSA: ctl: Fix copy of updated id with element read/write
    - ALSA: hda/realtek - Add headset Mic support for Lenovo ALC897 platform
    - ALSA: pcm: oss: Fix negative period/buffer sizes
    - ALSA: pcm: oss: Limit the period size to 16MB
    - ALSA: pcm: oss: Handle missing errors in snd_pcm_oss_change_params*()
    - btrfs: clear extent buffer uptodate when we fail to write it
    - btrfs: replace the BUG_ON in btrfs_del_root_ref with proper error handling
    - nfsd: Fix nsfd startup race (again)
    - tracefs: Have new files inherit the ownership of their parent
    - clk: qcom: regmap-mux: fix parent clock lookup
    - drm/syncobj: Deal with signalled fences in drm_syncobj_find_fence.
    - can: pch_can: pc...

Changed in linux (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (12.3 KiB)

This bug was fixed in the package linux - 5.13.0-30.33

---------------
linux (5.13.0-30.33) impish; urgency=medium

  * impish/linux: 5.13.0-30.33 -proposed tracker (LP: #1960055)

  * systemd/248.3-1ubuntu8.2 ADT test failure with linux/5.13.0-29.32
    (LP: #1960034)
    - Revert "block: avoid to quiesce queue in elevator_init_mq"
    - Revert "blk-mq: cancel blk-mq dispatch work in both blk_cleanup_queue and
      disk_release()"

linux (5.13.0-29.32) impish; urgency=medium

  * impish/linux: 5.13.0-29.32 -proposed tracker (LP: #1959238)

  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/2022.01.31)

  * CVE-2022-22942
    - SAUCE: drm/vmwgfx: Fix stale file descriptors on failed usercopy

  * CVE-2022-0330
    - drm/i915: Flush TLBs before releasing backing store

  * Impish update: upstream stable patchset 2022-01-05 (LP: #1956508)
    - ACPI: Get acpi_device's parent from the parent field
    - USB: serial: option: add Telit LE910S1 0x9200 composition
    - USB: serial: option: add Fibocom FM101-GL variants
    - usb: dwc2: gadget: Fix ISOC flow for elapsed frames
    - usb: dwc2: hcd_queue: Fix use of floating point literal
    - usb: dwc3: gadget: Ignore NoStream after End Transfer
    - usb: dwc3: gadget: Check for L1/L2/U3 for Start Transfer
    - usb: dwc3: gadget: Fix null pointer exception
    - net: nexthop: fix null pointer dereference when IPv6 is not enabled
    - usb: chipidea: ci_hdrc_imx: fix potential error pointer dereference in probe
    - usb: typec: fusb302: Fix masking of comparator and bc_lvl interrupts
    - usb: hub: Fix usb enumeration issue due to address0 race
    - usb: hub: Fix locking issues with address0_mutex
    - binder: fix test regression due to sender_euid change
    - ALSA: ctxfi: Fix out-of-range access
    - ALSA: hda/realtek: Add quirk for ASRock NUC Box 1100
    - ALSA: hda/realtek: Fix LED on HP ProBook 435 G7
    - media: cec: copy sequence field for the reply
    - Revert "parisc: Fix backtrace to always include init funtion names"
    - HID: wacom: Use "Confidence" flag to prevent reporting invalid contacts
    - staging/fbtft: Fix backlight
    - staging: greybus: Add missing rwsem around snd_ctl_remove() calls
    - staging: rtl8192e: Fix use after free in _rtl92e_pci_disconnect()
    - fuse: release pipe buf after last use
    - xen: don't continue xenstore initialization in case of errors
    - xen: detect uninitialized xenbus in xenbus_init
    - KVM: PPC: Book3S HV: Prevent POWER7/8 TLB flush flushing SLB
    - tracing/uprobe: Fix uprobe_perf_open probes iteration
    - tracing: Fix pid filtering when triggers are attached
    - mmc: sdhci-esdhc-imx: disable CMDQ support
    - mmc: sdhci: Fix ADMA for PAGE_SIZE >= 64KiB
    - mdio: aspeed: Fix "Link is Down" issue
    - powerpc/32: Fix hardlockup on vmap stack overflow
    - PCI: aardvark: Deduplicate code in advk_pcie_rd_conf()
    - PCI: aardvark: Implement re-issuing config requests on CRS response
    - PCI: aardvark: Simplify initialization of rootcap on virtual bridge
    - PCI: aardvark: Fix link training
    - proc/vmcore: fix clearing user buffer by properly using clear_user()
...

Changed in linux (Ubuntu Impish):
status: Fix Committed → Fix Released
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

This has been applied in Jammy via stable update process: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1959376

Changed in linux (Ubuntu):
status: In Progress → Fix Released
Changed in linux (Ubuntu Jammy):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.