ceph: fix inode number handling on arches with 32-bit ino_t

Bug #1899582 reported by bugproxy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Fix Released
High
Skipper Bug Screeners
linux (Ubuntu)
Fix Released
Undecided
Canonical Kernel Team
Focal
Fix Released
Medium
Frank Heimes
Groovy
Fix Released
Undecided
Canonical Kernel Team

Bug Description

SRU Justification:
==================

[Impact]

* A problem occurs with 32-bit ino_t values on 64-bit architectures (like s390x).

* Without this patch, cephfs kernel client does not work properly on s390x: the initial symtom is missing the first created file/directory on cephfs mount point as reported in https://tracker.ceph.com/issues/46828.

[Fix]

* cherry-pick for groovy: ebce3eb2f7ef9f6ef01a60874ebd232450107c9a ebce3eb2f7ef "ceph: fix inode number handling on arches with 32-bit ino_t"

* backport for focal: https://launchpadlibrarian.net/501822049/0001-ceph-fix-inode-number-handling-on-arches-with-32-bit.patch

[Test Case]

* IBM Development team and Test team have carried out simple testing and stress testing by automatically creating many Ceph clusters on s390x (with different configuration each time), doing the cephfs kernel mounting, checking inode numbers, running stress tests. The same procedures were done on x86_64 arch by Test team.

[Regression Potential]

* There is a certain regression risk with this patch, since:

* common ceph filesystem code in fs/ceph/ is modified

* the inode handling is touched

* ceph is used for supplemental volumes

* and as the original author of this patch (Jeff Layton) wrote in the commit message, the potential issue might happen on 32-bit arches only and he also provided workaround for those arches.

[Other]

* The above patch/commit was upstream accepted with kernel 5.9.

* Cherry picking the patch/commit to groovy works fine, hence this SRU.

* For focal a cherry pick does not apply cleanly, hence a backport got added to LP 1899582.

__________

Backport provided for git-commit to 20.04 kernel.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ebce3eb2f7ef9f6ef01a60874ebd232450107c9a

Backport patch attached for commit ebce3eb upstream.

It was applied and tested on top of Ubuntu commit :

linux (5.4.0-48.52) focal; urgency=medium

Revision history for this message
bugproxy (bugproxy) wrote : backport of commit ebce3eb to Ubuntu 20.04

Default Comment by Bridge

tags: added: architecture-s39064 bugnameltc-187973 severity-high targetmilestone-inin2004
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → ceph (Ubuntu)
Changed in ubuntu-z-systems:
importance: Undecided → High
affects: ceph (Ubuntu) → linux (Ubuntu)
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Has this been forwarded to the stable trees?

I do not see it applied on 5.8.x-stable or lower trees.

Currently only available in linux-5.9 which we are not shipping yet.

Changed in linux (Ubuntu):
status: New → Incomplete
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
Changed in linux (Ubuntu Groovy):
assignee: Skipper Bug Screeners (skipper-screen-team) → nobody
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2020-10-13 11:40 EDT-------
Hi,

This patch is not going to be merged to stable 5.8 tree, more details at : https://lkml.org/lkml/2020/9/5/106.

Thus we have to do this backport with minor change for 5.4 tree for Ubuntu 20.04 LTS. The most significant change/difference from the original commit is in the hunk in fs/ceph/inode.c.

Revision history for this message
Frank Heimes (fheimes) wrote :

If it's not possible (due user-visible changes) to get it marked for stable, I'm wondering if it can be added to groovy -- could become difficult.
On the other hand side if the backport should be applied to focal, it needs to be in groovy first, to avoid regressions on updates.
I verified that the commit cherry-picks cleanly on groovy master-next, hence I'll create a kernel SRU, but this will land (if at all) after 20.10 GA, since we already reached the 20.10 kernel freeze.

Revision history for this message
Frank Heimes (fheimes) wrote :

I just noticed that it is a user-visible change on 32-bit arches only.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-10-14 04:43 EDT-------
This is the original bug report at Ceph project : https://tracker.ceph.com/issues/46828. Might help to clarify a bit ...

Frank Heimes (fheimes)
Changed in linux (Ubuntu Groovy):
status: Incomplete → New
Changed in ubuntu-z-systems:
status: New → Triaged
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

So

>> 1/ inode numbers will be seen to have changed between kernel versions.
>> 32-bit arches will see large inode numbers now instead of the hashed
>> ones they saw before.
>>

That seems fine to me. This is all during runtime only, and doesn't change ondisk format at all right?

>> 2/ any really old software not built with LFS support may start failing
>> stat() calls with -EOVERFLOW on inode numbers >2^32. Nothing much we
>> can do about these, but hopefully the intersection of people running
>> such code on ceph will be very small.

I think we universally support LFS even on armhf & i386 arches. Also note the https://lintian.debian.org/tags/binary-file-built-without-LFS-support.html

It may have been better to scope the patch for backport to s390x only, however, given that hwe kernels will have this eventually, imho it is fine to submit this.

Btw, you can also submit kernel patch to the kernel-team mailing list, if it is formatted as per Ubuntu Kernel team guidelines with like BugLink: https://bugs.launchpad.net/bugs/1899582.

Note we are past kernel freeze now, and there is no SRU cycle between now and 8th of November, due to datacenter move as announced in https://lists.ubuntu.com/archives/kernel-sru-announce/2020-September/000170.html Hence the earlier this can land is probably December.

Frank Heimes (fheimes)
description: updated
Changed in linux (Ubuntu Focal):
assignee: nobody → Frank Heimes (fheimes)
Changed in linux (Ubuntu Groovy):
assignee: nobody → Frank Heimes (fheimes)
Frank Heimes (fheimes)
description: updated
description: updated
Revision history for this message
Frank Heimes (fheimes) wrote :

Kernel SRU request submitted for groovy:
https://lists.ubuntu.com/archives/kernel-team/2020-October/thread.html#114060
changing status to 'In Progress' for groovy.

Changed in linux (Ubuntu Groovy):
status: New → In Progress
assignee: Frank Heimes (fheimes) → Canonical Kernel Team (canonical-kernel-team)
Changed in ubuntu-z-systems:
status: Triaged → In Progress
Revision history for this message
Francis Ginther (fginther) wrote :

@xnox, @fheimes, we are preparing a groovy respin which will pick this up.

Revision history for this message
Frank Heimes (fheimes) wrote :

@fginther: glad to hear that - thanks a lot!

Revision history for this message
Frank Heimes (fheimes) wrote :

Kernel test builds based on groovy master-next (cherry-pick) and focal master-next (backport) are available here for further testing: https://people.canonical.com/~fheimes/lp1899582/

Revision history for this message
Frank Heimes (fheimes) wrote :

Kernel SRU request submitted for focal:
https://lists.ubuntu.com/archives/kernel-team/2020-October/thread.html#114069
changing status to 'In Progress' for focal.

Changed in linux (Ubuntu Focal):
status: New → In Progress
Stefan Bader (smb)
Changed in linux (Ubuntu Focal):
importance: Undecided → Medium
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-10-19 03:28 EDT-------
(In reply to comment #25)
> Kernel test builds based on groovy master-next (cherry-pick) and focal
> master-next (backport) are available here for further testing:
> https://people.canonical.com/~fheimes/lp1899582/

The kernels seem both fine on groovy and focal, respectively.

Revision history for this message
Frank Heimes (fheimes) wrote :

That's great - thx for testing, Tuan!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 5.8.0-25.26

---------------
linux (5.8.0-25.26) groovy; urgency=medium

  * groovy/linux: 5.8.0-25.26 -proposed tracker (LP: #1899940)

  * CVE-2020-12351
    - Bluetooth: L2CAP: Fix calling sk_filter on non-socket based channel

  * CVE-2020-12352
    - Bluetooth: A2MP: Fix not initializing all members

  * CVE-2020-12351 // CVE-2020-12352 // CVE-2020-24490
    - Bluetooth: Disable High Speed by default
    - Bluetooth: MGMT: Fix not checking if BT_HS is enabled
    - [Config] Disable BlueZ highspeed support

  * ec2-hibinit-agent needs to properly initialize swap file (LP: #1892728)
    - ext4: implement swap_activate aops using iomap

 -- Andrea Righi <email address hidden> Thu, 15 Oct 2020 12:09:24 +0200

Changed in linux (Ubuntu Groovy):
status: In Progress → Fix Released
Ian May (ian-may)
Changed in linux (Ubuntu Focal):
status: In Progress → Fix Committed
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-11-23 06:18 EDT-------
Hello,

I tested the latest kernel 5.4.0.55.58 (#61) in 'proposed-focal' and it seems to work just fine.

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (78.9 KiB)

This bug was fixed in the package linux - 5.4.0-56.62

---------------
linux (5.4.0-56.62) focal; urgency=medium

  * focal/linux: 5.4.0-56.62 -proposed tracker (LP: #1905300)

  * CVE-2020-4788
    - selftests/powerpc: rfi_flush: disable entry flush if present
    - powerpc/64s: flush L1D on kernel entry
    - powerpc/64s: flush L1D after user accesses
    - selftests/powerpc: entry flush test

linux (5.4.0-55.61) focal; urgency=medium

  * focal/linux: 5.4.0-55.61 -proposed tracker (LP: #1903175)

  * Update kernel packaging to support forward porting kernels (LP: #1902957)
    - [Debian] Update for leader included in BACKPORT_SUFFIX

  * Avoid double newline when running insertchanges (LP: #1903293)
    - [Packaging] insertchanges: avoid double newline

  * EFI: Fails when BootCurrent entry does not exist (LP: #1899993)
    - efivarfs: Replace invalid slashes with exclamation marks in dentries.

  * CVE-2020-14351
    - perf/core: Fix race in the perf_mmap_close() function

  * raid10: Block discard is very slow, causing severe delays for mkfs and
    fstrim operations (LP: #1896578)
    - md: add md_submit_discard_bio() for submitting discard bio
    - md/raid10: extend r10bio devs to raid disks
    - md/raid10: pull codes that wait for blocked dev into one function
    - md/raid10: improve raid10 discard request
    - md/raid10: improve discard request for far layout
    - dm raid: fix discard limits for raid1 and raid10
    - dm raid: remove unnecessary discard limits for raid10

  * Bionic: btrfs: kernel BUG at /build/linux-
    eTBZpZ/linux-4.15.0/fs/btrfs/ctree.c:3233! (LP: #1902254)
    - btrfs: drop unnecessary offset_in_page in extent buffer helpers
    - btrfs: extent_io: do extra check for extent buffer read write functions
    - btrfs: extent-tree: kill BUG_ON() in __btrfs_free_extent()
    - btrfs: extent-tree: kill the BUG_ON() in insert_inline_extent_backref()
    - btrfs: ctree: check key order before merging tree blocks

  * Ethernet no link lights after reboot (Intel i225-v 2.5G) (LP: #1902578)
    - igc: Add PHY power management control

  * Undetected Data corruption in MPI workloads that use VSX for reductions on
    POWER9 DD2.1 systems (LP: #1902694)
    - powerpc: Fix undetected data corruption with P9N DD2.1 VSX CI load emulation
    - selftests/powerpc: Make alignment handler test P9N DD2.1 vector CI load
      workaround

  * [20.04 FEAT] Support/enhancement of NVMe IPL (LP: #1902179)
    - s390: nvme ipl
    - s390: nvme reipl
    - s390/ipl: support NVMe IPL kernel parameters

  * uvcvideo: add mapping for HEVC payloads (LP: #1895803)
    - media: uvcvideo: Add mapping for HEVC payloads

  * Focal update: v5.4.73 upstream stable release (LP: #1902115)
    - ibmveth: Switch order of ibmveth_helper calls.
    - ibmveth: Identify ingress large send packets.
    - ipv4: Restore flowi4_oif update before call to xfrm_lookup_route
    - mlx4: handle non-napi callers to napi_poll
    - net: fec: Fix phy_device lookup for phy_reset_after_clk_enable()
    - net: fec: Fix PHY init after phy_reset_after_clk_enable()
    - net: fix pos incrementment in ipv6_route_seq_next
    - net/smc: fix valid DMBE buffer sizes
    - net...

Changed in linux (Ubuntu Focal):
status: Fix Committed → Fix Released
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-12-01 04:31 EDT-------
IBM Buzgilla status->closed, Fix Released by all requested distros

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.