Ubuntu
s390-tools package

[UBUNTU 20.04] zgetdump can not handle multivolume dumps

Bug #1987387 reported by bugproxy
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Fix Released
Medium
Skipper Bug Screeners
s390-tools (Ubuntu)
Fix Released
High
Unassigned
Focal
Fix Released
High
Unassigned
s390-tools-signed (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Released
High
Unassigned

Bug Description

SRU Justification:
==================

[Impact]

 * The zgetdump tool (as part of the current s390-tools version in focal)
   is not able to handle multi-volume dumps (DASD disk) dumps.

 * While this is rarely needed, it is extremely annoying if one is
   in usually urgent need to use it and it does not work.

 * On s390x systems multi-volume (DASD disk) dumps are pretty common,
   due to usually costly and therefore limited disk resources,
   DASD disks are usually relatively small.

[Fix]

 * d55b787 d55b787d05eb9bd70f93c36cf859b66b2ad02038 "zgetdump: Fix device node determination via sysfs"

[Test Plan]

 * Have an IBM zSystems LPAR with Ubuntu 20.04 / focal (latest).

 * Have two (or more) additional DASD disks reserved for storing dumps,
   assigned to the system and enabled:
   sudo chzdev -e 0002, 0003
   (Assuming 0001 has Ubuntu 20.04 installed - all 3390 disks).
   Get the block device names using 'lsdasd' - assuming here dasdb, dasdc

 * If needed low-level format these disks:
   sudo dasdfmt -y -b 4096 /dev/dasdb
   sudo dasdfmt -y -b 4096 /dev/dasdc

 * Create one single partition per disk:
   sudo fdasd -a /dev/dasdb
   sudo fdasd -a /dev/dasdc

 * Create an mvdump.conf file that points to the above disks
   sudo vi /etc/mvdump.conf
   ...
   cat /etc/mvdump.conf
   /dev/dasdb1
   /dev/dasdc1

 * Re-write the zipl boot record like this:
   sudo zipl -n -M /etc/mvdump.conf

 * Now ipl the system from the first dump DASD: 0002
   and initiate the DASD dump by:
   1. Stop all CPUs.
   2. Store status on the IPL CPU.
   3. IPL the dump tool on the IPL CPU.

 * Wait until the dump is completed and re-ipl the Ubuntu 20.04 again (0001).

 * Without this fix one will see a message like this:
   sudo zgetdump -i /dev/dasdb
   zgetdump: Could not open "/sys/bus/ccw/devices/0.0.0002/dasdb/dev" (No such file or directory)

 * With the fix one will see a message like this:
   zgetdump -i /dev/dasdb
   General dump info:
     Dump format........: s390mv_ext
     Version............: 1
     Dump created.......:
     ...
     Dump device info:
       Volume 0: 0.0.0002 (online/active)
       Volume 1: 0.0.0003 (online/valid)

 * For more in-depth details see the
   'Linux on System z. Using the Dump Tools.' documentation:
   https://www.ibm.com/docs/en/linux-on-systems?topic=dump-hmc-se-example

[Where problems could occur]

 * A new function got introduced to 'check whether a device with a given
   busid is online'.
   Issues could occur in case this function is broken and
   checks for a wrong busid, has a wrong path
   or handled the status wrongly.

 * The kind of 'little' refactoring of that patch may lead to
   further unexpected issues (that can largely identified by a test build).

 * The additional use of libutil functions may cause issues
   in case of an outdated libutil that does not offer all needed functions.
   (Testable with a test build.)

 * Erroneous code may even break single volume dumps

[Other Info]

 * This code is known to work with hirsute and newer Ubuntu releases,
   esp. jammy (respectively their s390-tools versions).

 * The upstream code can be cleanly cherry-picked,
   hence is applied as-is.

 * An updated s390-tools version 2.12.0-0ubuntu3.7 with a
   patched zgetdump tool was build and made available via this PPA:
   https://launchpad.net/~fheimes/+archive/ubuntu/lp1987387
   and was already successfully tested!
__________

== Comment: #0 - Thorsten Diehl <email address hidden> - 2022-08-16 12:40:46 ==
I installed Ubuntu 20.04.4 LTS on IBM z14, enabled two DASDs, created one partition on each DASD and created an mvdump.conf file like this:
/dev/dasdc1
/dev/dasdd1

I wrote the boot record by zipl -n -M mvdump.conf and IPLed the system from dasdc devno.
The dump completed succesfully.
Then I tried to get this dump via zgetdump on the restarted Ubuntu 20.04.4 system (zgetdump -v reports version 2.12.0-build-20220506), I got the following error:
root@m8330032:~# zgetdump -i /dev/dasdc
zgetdump: Could not open "/sys/bus/ccw/devices/0.0.9405/dasdc/dev" (No such file or directory)
root@m8330032:~# zgetdump -i /dev/dasdc1
zgetdump: Could not open "/sys/bus/ccw/devices/0.0.9405/dasdc/dev" (No such file or directory)
root@m8330032:~# zgetdump -i /dev/dasdd
zgetdump: Could not open "/sys/bus/ccw/devices/0.0.9405/dasdc/dev" (No such file or directory)
root@m8330032:~# zgetdump -i /dev/dasdd1
zgetdump: No valid dump found on "/dev/dasdd1"
root@m8330032:~#

However, If I'm doing the same zgetdump on another system (e.g. with newer s390-tools version), I get the expected result
m83lp32:~ # zgetdump -i /dev/dasdc
General dump info:
  Dump format........: s390mv_ext
  Version............: 1
  Dump created.......: Tue, 16 Aug 2022 18:31:57 +0200
  Dump ended.........: Tue, 16 Aug 2022 18:32:02 +0200
  Dump CPU ID........: ff1fa1e739068000
  UTS node name......: m8330032.lnxne.boe
  UTS kernel release.: 5.4.0-124-generic
  UTS kernel version.: #140-Ubuntu SMP Thu Aug 4 02:23:07 UTC 2022
  Build arch.........: s390x (64 bit)
  System arch........: s390x (64 bit)
  CPU count (online).: 16
  CPU count (real)...: 16
  Dump memory range..: 4096 MB
  Real memory range..: 4096 MB
  Dump file size.....: 849 MB

Memory map:
  0000000000000000 - 00000000ffffffff (4096 MB)

Dump device info:
  Volume 0: 0.0.9405 (online/active)
  Volume 1: 0.0.9406 (online/valid)
m83lp32:~ #

The error is easily reproducible.
Please update zgetdump to a newer version to solve this RAS problem.

With Jammy (22.04.1; s390-tools version 2.20.0-build-20220623) this problem does not occur.

== Comment: #3 - Jan Hoeppner <email address hidden> - 2022-08-22 01:43:45 ==
There were several issues fixed in s390-tools v2.15.1 in regards to multivolume dumps: https://github.com/ibm-s390-linux/s390-tools/releases/tag/v2.15.1

Especially the following upstream commit for zgetdump:
https://github.com/ibm-s390-linux/s390-tools/commit/d55b787d05eb9bd70f93c36cf859b66b2ad02038

---
External link: https://warthogs.atlassian.net/browse/PEI-30

See original description
bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-199411 severity-medium targetmilestone-inin---
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
Frank Heimes (fheimes)
affects: linux (Ubuntu) → s390-tools (Ubuntu)
Changed in ubuntu-z-systems:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
Changed in s390-tools (Ubuntu):
importance: Undecided → Medium
Changed in ubuntu-z-systems:
importance: Undecided → Medium
Revision history for this message
Frank Heimes (fheimes) wrote :

A test package was build in PPA and is available here:
https://launchpad.net/~fheimes/+archive/ubuntu/lp1987387
Would you mind giving it a try and testing if the mentioned commit properly fixes this issue?"

(Btw. in addition to the s390x-tools package from the PPA,
the s390-tools-signed package must be installed to, to satisfy dependencies.")

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2022-08-25 08:51 EDT-------
Hi Frank,
yes, with the s390-tools packages from your repo the problem is fixed; zgetdump provides the expected results. Please go ahead with this. Thank you!

Revision history for this message
Frank Heimes (fheimes) wrote :

Many thx Thorsten for the quick test! Will go ahead now with it ...

bugproxy (bugproxy)
tags: added: targetmilestone-inin2004
removed: targetmilestone-inin---
Revision history for this message
Frank Heimes (fheimes) wrote :
description: updated
Changed in s390-tools (Ubuntu):
status: New → In Progress
Changed in ubuntu-z-systems:
status: New → In Progress
Revision history for this message
Frank Heimes (fheimes) wrote :
Changed in s390-tools (Ubuntu):
assignee: Skipper Bug Screeners (skipper-screen-team) → nobody
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "debdiff_s390-tools_focal_from_2.12.0-0ubuntu3.6_to_2.12.0-0ubuntu3.7.diff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Frank Heimes (fheimes)
Changed in s390-tools (Ubuntu):
importance: Medium → High
description: updated
tags: added: pei-30
Simon Chopin (schopin)
Changed in s390-tools (Ubuntu Focal):
status: New → Triaged
Changed in s390-tools (Ubuntu):
status: In Progress → Fix Released
Changed in s390-tools (Ubuntu Focal):
importance: Undecided → High
Changed in s390-tools-signed (Ubuntu Focal):
status: New → Triaged
Changed in s390-tools-signed (Ubuntu):
status: New → Fix Released
Changed in s390-tools-signed (Ubuntu Focal):
importance: Undecided → High
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: In Progress → Fix Released
status: Fix Released → In Progress
Revision history for this message
Simon Chopin (schopin) wrote :

Packaging side, LGTM.

However, I'm a bit skeptical at the patch itself, as it technically removes a feature. Do we have an analysis of why it's OK to remove the `block:` entries? I'm not too familiar with the ecosystem, so wouldn't it possible to have a focal userspace running on an older kernel that still presents the old way in its sysfs?

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2022-11-03 04:35 EDT-------
(In reply to comment #16)
> Packaging side, LGTM.
>
> However, I'm a bit skeptical at the patch itself, as it technically removes
> a feature. Do we have an analysis of why it's OK to remove the `block:`
> entries? I'm not too familiar with the ecosystem, so wouldn't it possible to
> have a focal userspace running on an older kernel that still presents the
> old way in its sysfs?

According to what i found,
this was deprecated in 2006 and can be re-enabled again in kernel with
CONFIG_SYSFS_DEPRECATED but then a lot of udev tools will break.

https://elixir.bootlin.com/linux/latest/source/init/Kconfig#L1287
https://cateee.net/lkddb/web-lkddb/SYSFS_DEPRECATED.html

Does s390x support such old kernels ?

Revision history for this message
Frank Heimes (fheimes) wrote :

Ok, since first Ubuntu support for s390x was introduced with 16.04 (in 2016) with kernel 4.4 (so ~10 years after it got flagged as deprecated), I think it's fair to say that it will not hurt us (Ubuntu).

Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello bugproxy, or anyone else affected,

Accepted s390-tools into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools/2.12.0-0ubuntu3.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in s390-tools (Ubuntu Focal):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-focal
Changed in s390-tools-signed (Ubuntu Focal):
status: Triaged → Fix Committed
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello bugproxy, or anyone else affected,

Accepted s390-tools-signed into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools-signed/2.12.0-0ubuntu3.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: In Progress → Fix Committed
Frank Heimes (fheimes)
description: updated
Revision history for this message
Frank Heimes (fheimes) wrote :

I was able to successfully verify a multivolume dump using two DASD ECKD disks with zgetdump on a z/VM guest running focal - see attachment.

Hence I'm adjusting the tags accordingly...

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla
Download full text (3.3 KiB)

------- Comment From <email address hidden> 2022-12-02 10:59 EDT-------
I installed Ubuntu 20.04.5 with latest updates.
s390-tools version was 2.12.0-0ubuntu3.6

I enabled -proposed repo (unsigned) and updated to 2.12.0-0ubuntu3.7:

root@a3515006:~# apt-get install s390-tools/focal-proposed
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '2.12.0-0ubuntu3.7' (Ubuntu:20.04/focal-proposed [s390x]) for 's390-tools'
The following packages were automatically installed and are no longer required:
apport-symptoms libfwupdplugin1 libxmlb1 python3-apport
python3-problem-report python3-systemd
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
s390-tools-signed
Suggested packages:
s390-tools-cpuplugd s390-tools-osasnmpd s390-tools-statd lsscsi
The following packages will be upgraded:
s390-tools s390-tools-signed
2 upgraded, 0 newly installed, 0 to remove and 34 not upgraded.
Need to get 1255 kB of archives.
After this operation, 4096 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://ports.ubuntu.com/ubuntu-ports focal-proposed/main s390x s390-tools s390x 2.12.0-0ubuntu3.7 [1248 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports focal-proposed/main s390x s390-tools-signed s390x 2.12.0-0ubuntu3.7 [6916 B]
Fetched 1255 kB in 0s (5456 kB/s)
(Reading database ... 85300 files and directories currently installed.)
Preparing to unpack .../s390-tools_2.12.0-0ubuntu3.7_s390x.deb ...
Unpacking s390-tools (2.12.0-0ubuntu3.7) over (2.12.0-0ubuntu3.6) ...
Preparing to unpack .../s390-tools-signed_2.12.0-0ubuntu3.7_s390x.deb ...
Unpacking s390-tools-signed (2.12.0-0ubuntu3.7) over (2.12.0-0ubuntu3.6) ...
Setting up s390-tools-signed (2.12.0-0ubuntu3.7) ...
Setting up s390-tools (2.12.0-0ubuntu3.7) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for systemd (245.4-4ubuntu3.19) ...

Multivolume dump information is now shown correctly:

root@a3515006:~# lsdasd
Bus-ID Status Name Device Type BlkSz Size Blocks
================================================================================
0.0.6b14 active dasda 94:0 ECKD 4096 42259MB 10818360
0.0.6b17 active dasdb 94:4 ECKD 4096 21129MB 5409180
0.0.6b15 active dasdc 94:8 ECKD 4096 42259MB 10818360
0.0.6b16 active dasdd 94:12 ECKD 4096 21129MB 5409180
root@a3515006:~# zgetdump -i /dev/dasdc
General dump info:
Dump format........: s390mv_ext
Version............: 1
Dump created.......: Fri, 02 Dec 2022 15:46:49 +0000
Dump ended.........: Fri, 02 Dec 2022 15:47:00 +0000
Dump CPU ID........: ff0f5dc839318000
UTS node name......: a3515006.lnxne.boe
UTS kernel release.: 5.4.0-135-generic
UTS kernel version.: #152-Ubuntu SMP Wed Nov 23 21:05:01 UTC 2022
Build arch.........: s390x (64 bit)
System arch........: s390x (64 bit)
CPU count (online).: 16
CPU count (real)...: 16
Dump memory range..: 16384 MB
Real memory range..: 16384 MB
Dump file size.....: 1037 MB

Memory map:
0000000000000000 - 00000003ffffffff (16384 MB)

Dump device info:
Volume 0: 0.0.6b15 (online/ac...

Read more...

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2023-01-23 05:02 EDT-------
Hmm, the s390-tools version 2.12.0-0ubuntu3.7 (as of "proposed") is still not avilable. I'm stuck on the broken 2.12.0-0ubuntu3.6 version.

Revision history for this message
Frank Heimes (fheimes) wrote :

Hello Thorsten,
the Ubuntu SRU policy is that we have to patch latest releases first and than go down to the oldest release that is affected, and that just took some time (esp. during year end break).

But it's almost there - and you can even use it today - from proposed.
s390-tools version 2.12.0-0ubuntu3.7:
https://launchpad.net/ubuntu/+source/s390-tools/2.12.0-0ubuntu3.7
is the package that incl. the zgetdump fix (as well as support for the secure boot trailer).

You can easily use it as follows:
sudo add-apt-repository -y "deb http://us.ports.ubuntu.com/ubuntu-ports/ $(lsb_release -sc)-proposed main universe"
sudo apt update
sudo apt install --yes s390-tools

(It will soon be promoted from -proposed to -updates.)

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package s390-tools - 2.12.0-0ubuntu3.7

---------------
s390-tools (2.12.0-0ubuntu3.7) focal; urgency=medium

  * d/p/lp1987387-zgetdump-Fix-device-node-determination-via-sysfs.patch
    Fix zgetdump can not handle multivolume dumps. (LP: #1987387)
  * d/p/lp1996069-zipl-boot-add-secure-boot-trailer.patch
    Add secure boot trailer in zipl stage 3 to keep compatibility with
    upcoming IBM zSystems firmware updates. (LP: #1996069)

 -- Frank Heimes <email address hidden> Fri, 11 Nov 2022 09:07:52 +0200

Changed in s390-tools (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package s390-tools-signed - 2.12.0-0ubuntu3.7

---------------
s390-tools-signed (2.12.0-0ubuntu3.7) focal; urgency=medium

  * Rebuild against 2.12.0-0ubuntu3.7 (LP: #1987387, LP: #1996069)

 -- Frank Heimes <email address hidden> Thu, 25 Aug 2022 09:18:36 +0200

Changed in s390-tools-signed (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for s390-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Frank Heimes (fheimes) wrote :

Now that we (Thorsten) just talked (better wrote) about it,
it got promoted to -updates and will land there in a few hours.
So I would say that in a few hours (latest tomorrow) you can also get it via the normal update process, means:
sudo apt update
sudo apt full-upgrade # or selectively # sudo apt install s390-tools

Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2023-01-23 13:04 EDT-------
Updated version 2.12.0-0ubuntu3.7 is now in -updates repository.
Successfully verified and closed.
Thank you!

tags: added: targetmilestone-inin20045
removed: targetmilestone-inin2004
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.