[UBUNTU 22.04] dbginfo.sh: stabilization of data collection

Bug #1971959 reported by bugproxy
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Fix Released
High
Skipper Bug Screeners
s390-tools (Ubuntu)
Fix Released
High
Unassigned
Jammy
Fix Released
High
Unassigned
Kinetic
Fix Released
High
Unassigned

Bug Description

SRU Justification:
==================

[Impact]

 * The s390x-specific dbginfo.sh script, used to collect runtime,
   configuration and trace information for debugging Linux on IBM Z,
   has stabilization issues while collecting data.

 * Script hangups are possible on single commands,

 * potentially missing data collection on error

 * and overwrites of buffers by diag commands.

[Test Plan]

 * Prepare an IBM zSystems or LinuxONE system
   (ideally an LPAR as well as a z/VM guest)
   with an Ubuntu 22.04 system.

 * The s390-tools package is mandatory and always installed,
   but needs to be replaced with an updated/fixed version -
   pick the version from the PPA that is mentioned below.

 * Execute dbginfo.sh (as root or with sudo).

 * Monitor the output and verify if all 18 steps are completed,
   without any crashes or hangs.

 * Inspect the size and content of the result file:
   /tmp/DBGINFO-*.tgz

[Where problems could occur]

 * The script might no run on older generations anymore.

 * Some of the things that are gathered might not be
   properly picked (like z/VM data on LPAR).

 * If a step fails (like in the past), no summarized
   output might be generated.

__________

Description: dbginfo.sh: stabilization of data collection
Symptom: - script hangup possible on single commands
                - missing data collection on error
                - overwrite of buffers by diag commands
Problem: hangup and loss of data collection
Solution: rework code regarding
                - add timeout function
                - remove problematic commands
                - include handling improvements
Component: s390-tools

bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-198039 severity-high targetmilestone-inin2204
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
Frank Heimes (fheimes)
affects: linux (Ubuntu) → s390-tools (Ubuntu)
Changed in ubuntu-z-systems:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
importance: Undecided → High
Changed in s390-tools (Ubuntu):
assignee: Skipper Bug Screeners (skipper-screen-team) → Frank Heimes (fheimes)
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2022-05-06 05:00 EDT-------
With Ubuntu 22.04 being based on s390-tools v2.20, the following patches need to be applied for jammy on top of s390-tools v2.20 for the stabilization of data collection with dbginfo.sh.

Commits based on s390-tools v2.20:

02a0d12988c43886943a26d3617b8f9422e6171a
be47b51890433693ca0e92316ce6e1492c046e73
2677a4182db0303ae6c90b727b09fafb52f8a46d
812df795c346c39002fb2cd60740866b2960c8c6
a0d6edf03c255ca2f906c4eb98398e8fcf03f0b8
81920f7cfd964f8133406848c5cee2b1d40cf7c8
a8579a072757ddd3d973dcd90ac73921a8dc55ce
2ab27bdaf2f00868ea54847d2dec360508c427f5
50a4740443673737df7c2b4e81ad2f2db448d43f
164d4817ecca5deadaff2b268876bbf330a0f985
abec41f514eea8b26ade6ffac88168f90a686ba5
fcb503ea3c9bd8612726aaea9d46e98af86fc049
2eea614bbff9e4e1755ca933d849c1f4964f4827
3a13cb43f211d997e3c5ea83cfc7ff71b964dca8

Frank Heimes (fheimes)
Changed in s390-tools (Ubuntu):
status: New → In Progress
Changed in ubuntu-z-systems:
status: New → In Progress
Frank Heimes (fheimes)
Changed in s390-tools (Ubuntu Kinetic):
assignee: Frank Heimes (fheimes) → nobody
Changed in s390-tools (Ubuntu Jammy):
status: New → In Progress
Changed in s390-tools (Ubuntu Kinetic):
status: In Progress → Fix Committed
Changed in ubuntu-z-systems:
status: In Progress → Fix Committed
Changed in s390-tools (Ubuntu Jammy):
importance: Undecided → High
Changed in s390-tools (Ubuntu Kinetic):
importance: Undecided → High
Revision history for this message
Simon Chopin (schopin) wrote :

The fix for this bug has been uploaded to Kinetic and the SRU queue for Jammy.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package s390-tools - 2.20.0-0ubuntu4

---------------
s390-tools (2.20.0-0ubuntu4) kinetic; urgency=medium

  * Fix chreipl-fcp-mpath (LP: #1971993)
    - Move chreipl-fcp-mpath* from /lib/udev/rules.d to /lib/udev.
    - d/control:
      + Build-Depend on bsdextrautils for hexdump
      + Add multiple explicit Depends on udev
      + s390-tools-chreipl-fcp-mpath: drop unnecessary Depends on lvm2
    - No longer change attributes of chreipl-fcp-mpath-common.sh to 755,
      since only the input script '.in' has a she-bang, but not the '.sh'
      anymore (was done with commit c2f8988).
    - Add d/p/0d15a07-chreipl-fcp-mpath-bundle-a-pre-cooked-man-page.patch
      to bundle a pre-cooked version of the man page for chreipl-fcp-mpath
      Required minor context adjustment for CHANGELOG.md hunk
      and changes in d/rules.
    - Add missing README.md to s390-tools-chreipl-fcp-mpath.doc
      (and with that also the README.md for genprotimg to s390-tools.docs).
  * Add new CPU-MF Counters for new IBM Z hardware (LP: #1960119) by:
    - d/p/2515832-util_arch-Add-IBM-z16-as-known-machine.patch and
    - d/p/cce5f51-cpumf-lscpumf-Add-IBM-z16-extended-counter-set-def.patch
  * Add exploitation support of new IBM Z crypto hardware (LP: #1959548) with:
    - d/p/b16a6d4f-lszcrypt-add-CEX8S-support.patch
    - d/p/bcbb6fca-zcryptstats-add-CEX8-support.patch
    - d/p/4382901d-lszcrypt-show-AP-bus-msg-size-limit-capability.patch
    - d/p/27dce331-lszcrypt-add-support-for-checkstop-state.patch
    - d/p/a29b3c89-lszcrypt-new-options-to-show-only-accel-cca-or-ep11-.patch
    - d/p/a8b0d7ac-lszcrypt-new-options-to-filter-cards-queues-only.patch
    - d/p/46fd42af-lszcrypt-new-option-to-show-the-serial-numbers-of-CC.patch
  * Stabilization of data collection in dbginfo.sh script (LP: #1971959)
    by adding several upstream patches:
    - d/p/*-dbginfo.sh-*.patch
    - whereas one needed minor context adjutment for the CHANGELOG.md hunk:
      d/p/50a4740-dbginfo.sh-replace-which-by-builtin-command-type-for.patch
  * Fix cmsfs-fuse mount failure due to unknown option '-o hard_remove'
    (LP: #1978323) with:
    d/p/0981df6-cmsfs-fuse-fix-enabling-of-hard_remove-option.patch

 -- Frank Heimes <email address hidden> Fri, 20 May 2022 13:48:34 +0200

Changed in s390-tools (Ubuntu Kinetic):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

What is the test case for this bug? How would one go about verifying, if the fix is working?

Revision history for this message
Frank Heimes (fheimes) wrote :

Sorry, I totally missed to copy over the SRU Justification from my notes to the Bug description.
Did it now - test plan is included.

description: updated
description: updated
description: updated
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello bugproxy, or anyone else affected,

Accepted s390-tools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools/2.20.0-0ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in s390-tools (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-jammy
Revision history for this message
Frank Heimes (fheimes) wrote :
Download full text (3.8 KiB)

Test on z/VM:
"
$ dpkg -l s390-tools
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=================-============-=============================>
ii s390-tools 2.20.0-0ubuntu3.1 s390x fundamental utilities for Lin>

$ sudo dbginfo.sh
dbginfo.sh: Debug information script version 2.20.0-build-20220623
Copyright IBM Corp. 2002, 2022

Hardware platform = s390x
Runtime environment = z/VM
Kernel version = 5.15.0 (5.15.0-39-generic)
OS version / distro = Ubuntu 22.04 LTS
Date and time of info = 2022-06-23-18-20-06

1 of 18: Collecting sysfs
 Warning: falling back to slow path
2 of 18: Collecting procfs
3 of 18: Collecting config files
4 of 18: Collecting command output
5 of 18: Collecting hyptop for z/VM - 5s output
6 of 18: Collecting z/VM output
dbginfo.sh: Warning: No program to communicate to z/VM CP
6 of 18: Skip z/VM: vmcp not available
7 of 18: Collecting network output
8 of 18: Skip osa oat: qethqoat not available
9 of 18: Skip ethtool: not available
10 of 18: Skip Trafic Control: tc not available
11 of 18: Skip bridge: not available
12 of 18: Skip OpenVSwitch: ovs-vsctl not available
13 of 18: Skip KVM: no virsh command
14 of 18: Skip docker: not available
15 of 18: Skip nvme: not available
16 of 18: Collecting log files
 0 logfiles over 50 MB
17 of 18: Postprocessing
18 of 18: Finalizing: Creating archive with collected data

Collected data was saved to:
 >> /tmp/DBGINFO-2022-06-23-18-20-06-hwe0003-118F67.tgz <<

Please review all collected data before sending to your service organization.

$ ls -l /tmp/DBGINFO-2022-06-23-18-20-06-hwe0003-118F67.tgz
-rw------- 1 root root 5907470 Jun 23 18:20 /tmp/DBGINFO-2022-06-23-18-20-06-hwe0003-118F67.tgz
"

and test on LPAR:

"
$ dpkg -l s390-tools
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=================-============-=============================>
ii s390-tools 2.20.0-0ubuntu3.1 s390x fundamental utilities for Lin>

$ sudo dbginfo.sh
dbginfo.sh: Debug information script version 2.20.0-build-20220623
Copyright IBM Corp. 2002, 2022

Hardware platform = s390x
Runtime environment = LPAR
Kernel version = 5.15.0 (5.15.0-23-generic)
OS version / distro = Ubuntu 22.04 LTS
Date and time of info = 2022-06-23-18-20-08

1 of 18: Collecting sysfs
 Warning: falling back to slow path
2 of 18: Collecting procfs
3 of 18: Collecting config files
4 of 18: Collecting command output
5 of 18: Collecting hyptop for LPAR - 5s output
6 of 18: Skip z/VM: no z/VM environment
7 of 18: Collecting network output
8 of 18: Skip osa oat: qethqoat not available
9 of 18: Skip ethtool: not available
10 of 18: Skip Trafic Control: tc not available
11 of 18: Skip bridge: not available
12 of 18: Skip OpenVSwitch: ovs-vsctl not available
13 of 18: Skip ...

Read more...

tags: added: verification-done verification-done-jammy
removed: verification-needed verification-needed-jammy
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package s390-tools - 2.20.0-0ubuntu3.1

---------------
s390-tools (2.20.0-0ubuntu3.1) jammy; urgency=medium

  * Fix chreipl-fcp-mpath (LP: #1971993)
    - Move chreipl-fcp-mpath* from /lib/udev/rules.d to /lib/udev.
    - d/control:
      + Build-Depend on bsdextrautils for hexdump
      + Add multiple explicit Depends on udev
      + s390-tools-chreipl-fcp-mpath: drop unnecessary Depends on lvm2
    - No longer change attributes of chreipl-fcp-mpath-common.sh to 755,
      since only the input script '.in' has a she-bang, but not the '.sh'
      anymore (was done with commit c2f8988).
    - Add d/p/0d15a07-chreipl-fcp-mpath-bundle-a-pre-cooked-man-page.patch
      to bundle a pre-cooked version of the man page for chreipl-fcp-mpath
      Required minor context adjustment for CHANGELOG.md hunk
      and changes in d/rules.
    - Add missing README.md to s390-tools-chreipl-fcp-mpath.doc
      (and with that also the README.md for genprotimg to s390-tools.docs).
  * Add new CPU-MF Counters for new IBM Z hardware (LP: #1960119) by:
    - d/p/2515832-util_arch-Add-IBM-z16-as-known-machine.patch and
    - d/p/cce5f51-cpumf-lscpumf-Add-IBM-z16-extended-counter-set-def.patch
  * Add exploitation support of new IBM Z crypto hardware (LP: #1959548) with:
    - d/p/b16a6d4f-lszcrypt-add-CEX8S-support.patch
    - d/p/bcbb6fca-zcryptstats-add-CEX8-support.patch
    - d/p/4382901d-lszcrypt-show-AP-bus-msg-size-limit-capability.patch
    - d/p/27dce331-lszcrypt-add-support-for-checkstop-state.patch
    - d/p/a29b3c89-lszcrypt-new-options-to-show-only-accel-cca-or-ep11-.patch
    - d/p/a8b0d7ac-lszcrypt-new-options-to-filter-cards-queues-only.patch
    - d/p/46fd42af-lszcrypt-new-option-to-show-the-serial-numbers-of-CC.patch
  * Stabilization of data collection in dbginfo.sh script (LP: #1971959)
    by adding several upstream patches:
    - d/p/*-dbginfo.sh-*.patch
    - whereas one needed minor context adjutment for the CHANGELOG.md hunk:
      d/p/50a4740-dbginfo.sh-replace-which-by-builtin-command-type-for.patch
  * Fix cmsfs-fuse mount failure due to unknown option '-o hard_remove'
    (LP: #1978323) with:
    d/p/0981df6-cmsfs-fuse-fix-enabling-of-hard_remove-option.patch

 -- Frank Heimes <email address hidden> Fri, 20 May 2022 13:48:34 +0200

Changed in s390-tools (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for s390-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2022-07-06 19:34 EDT-------
Fix verified and released to -updates, therefore we can close this bug.
Thanks everybody for your work.

Changing status to: CLOSED

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.