[20.04 FEAT] Enhanced tooling for guest images

s390-tools v2.9.0 is the current one, hence setting to Incomplete until 2.10.0 got released.

------- Comment From <email address hidden> 2019-07-30 07:26 EDT-------
Moved target to 20.04, will not make it in time for 19.10

This got moved to >= s390-tools version 2.13,
hence quite risky to get it into 20.04 ...

------- Comment From <email address hidden> 2020-03-03 03:06 EDT-------
The code can be found at https://github.com/mhartmay/s390-tools, it is based on upstream s390-tools. The new tool is genprotimg, the source files are in the same-named subdirectory.

Can we please have a ppa with the new s390-tools, including genprotimg?

The list of commits needed on top of s390-tools master:

538377b genprotimg: introduce new tool for the creation of PV images
f2d0980 include/boot/ipl.h: add constants needed by the tooling
02be680 include/boot/s390.h: add guard for `struct __vector128`
218b444 genprotimg: add relocator for stage3b
ec8b6de genprotimg: boot: use C pre-processor for linker script generation
2de4f02 genprotimg: boot: initial bootloader support
7243aeb zipl/libc: printf: print on linemode and ASCII console
a2d8a0f zipl/sclp: add `sclp_print_ascii`
3fe5ec8 zipl/sclp: add macros for the control-program masks
363107c zipl: refactor all EBCDIC code into separate files

Hi,

Anybody can activate and create a PPA for themselves, tick s390x as architecture and self build packages. And every PPA self-generates sipl signing key, such that one can have signed builds too automatically.

Do you need help & training on how to do that?

Are you interested in that for master branch development too?

You can also push/mirror git repositories to launchpad, and setup daily/on-change builds that would be published in a PPA for any Ubuntu releases you wish for.

I'd rather train IBM engineers to use launchpad in a self-service way to quickly turn around test packages / repositories by themselves, without blocking on manual assistance from Canonical.

Regards,

Dimitri.

------- Comment From <email address hidden> 2020-03-05 09:48 EDT-------
Hi Dimitri, yes I agree that it would be best to have a common session to run through that, but since this is unfortunately pretty urgent for us, would you mind creating a patched s390-tool for us in this case? We don't have the resources to do that by ourselves on short notice? Your support would really appreciated here. Many thanks in advance.....

------- Comment From <email address hidden> 2020-03-16 09:21 EDT-------
The changes for the guest tooling have been committed to the upstream s390-tools repository, https://github.com/ibm-s390-tools/s390-tools/commits/master.

------- Comment From <email address hidden> 2020-03-16 10:28 EDT-------
The following upstream commits need to be applied on top of the current s390-tools package in focal:

971970989bbb1de8887d11b0ab8e4f19adbd484f (zipl/boot: fix comment in stage3.lds)
bac3f93772bdf8618c2c9677c59569d70e4a39c0 (lib/zt_common: add STATIC_ASSERT macro)
cc16e41595d6dcb942f84443f27a1b52d06d17da (zipl: use STATIC_ASSERT macro for no padding verification)
6fe9e6c55c69c14971dca55551009f5060418aae (zipl/libc: Introduce vsnprintf)
8874b908254c47c8a6fd7a1aca2c7371c11035c4 (zipl/libc: Fix potential buffer overflow in printf)
f7430027b41d5ad6220e962a179c2a5213330a44 (zipl/libc: Replace sprintf with snprintf)
36fed0e6c6590631c4ce1707c8fe3c3397bcce4d (zipl/libc: Indicate truncated lines in printf with '...')
400167f5128a14ba48b0d05b7b777b42c450c73f (Support `lib/zt_common.h` to be used in assembler and add `_AC` macro)
9d39a4bd47008b15bbf4ebe672b91d6d63888536 (zipl: move IPL related definitions into separate header)
675c854fa3239882c59a9419c776eb13bc70cf76 (zipl: move SIGP related functions and definitions into separate header)
0e385a81caf7c266c0784613e0264c03271eb99a (zipl: add SIGP_SET_ARCHITECTURE to sigp.h and use it)
d884fb8db4c4f383780d6fc8087abd8f80e1c8b8 (zipl/stage3: make IPL_DEVICE definition consistent with tape0.S)
7e37a1d4e0605ea120db18f82d039c055fd5d737 (zipl: move Linux layout definitions into separate header)
c871050097ecb2ec83cf3018ea36e01cd22cbe7d (zipl: tape0: use constants defined in linux_layout.h)
c07104dbc734ec6e55accf1bd2091b251f312ed8 (zipl: use STAGE3_ENTRY for STAGE3_LOAD_ADDRESS)
97ab8fb4e98c84a89d421c08b392db665125a3c0 (zipl: move loaders layout definitions into separate header)
67e76b8ebd8acb4aef1d22309287776892b7267e (zipl/s390.h: rename `inline` macro into `__always_inline`)
24fe8c1d1b75185f341ec2d0efc6c34f0b9263f1 (zipl: move __always_inline/barrier/__pa32/pa to zt_common.h)
2e28291c75d73b92921f7769eaa803fe3222f383 (zipl: make BLK_PWRT unsigned int)
c55ceabc6726a7806922d288149003661f673a2f (Consolidate MIN and MAX macros)
f454c6825f5087cf671d0dfbe96f7f3d148569d6 (zipl: remove libc.h include in s390.h)
b83c8944f195117609a36f383f32377014e34c31 (zipl: move s390.h to include/boot/s390.h)
b0f82d22f9f60a0a8db1976751aa5a875e7c5f80 (zipl/libc: include 's390.h')
2568863f581cff9bf3b1e27c2d2917b5ae3b5177 (include/boot/s390.h: move panic and panic_notify to libc.h)
305235a7bce814f71ec113a612b6117c96894e23 (include/boot/s390.h: fixes for -Werror=sign-conversion)
a37170b8bec07a0ffc3270a4c78124e1117f0337 (zipl: refactor all EBCDIC code into separate files)
303a3707e2e59e0ad581876db426a52fffa606b0 (zipl/sclp: add macros for the control-program masks)
f99560f734e8101a0e8195d73e3350d9211335b8 (zipl/sclp: add `sclp_print_ascii`)
e51663bbca8770c1f7986dac47a59193dbf96010 (zipl/libc: printf: print on linemode and ASCII console)
67aef9bbf3b5d18c70e8c4a45734bcb6d6744a8c (Consolidate `ALIGN, __ALIGN_MASK, ARRAY_SIZE` macros)
3356d6f4facd748f8f5cf24ffc5056db3e915f2c (genprotimg: boot: initial bootloader support)
2d600570df98a1d26a6f3947ae8c39bcde00b464 (genprotimg: boot: use C pre-processor for linker script generation)
d2f8f972cff7aacbef8e72577af70dbf59ba3ead (genprot...

The codedrop of commits since 2018 is not an acceptable volume of code to review and land post Feature Freeze.

They intermix desired features, with refactoring, and white space changes, unrelated changes to dbginfo, etc.

Why have you not been continuously releasing these commits publicly?
Why is there such a wide mix of development merged?

We are past Feature Freeze, and Feature Freeze Exception for s390-tools was not granted yet, as risk of regressions of non-PVM systems needs to be done. Especially of code changes around zdev & zipl. Please see https://bugs.launchpad.net/ubuntu/+source/s390-tools/+bug/1866866 for further questions from the release team.

 CHANGELOG.md | 22 +-
 Makefile | 4 +-
 README.md | 7 +-
 cmsfs-fuse/dasd.c | 1 +
 cmsfs-fuse/helper.h | 2 -
 cpumf/Makefile | 3 +-
 cpumf/bin/cpumf_helper.in | 155 +++----
 cpumf/data/cpum-cf-extended-z15.ctr | 376 +++++++++++++++
 cpumf/data/cpum-cf-hw-counter.map | 5 +-
 genprotimg/.gitignore | 5 +
 genprotimg/Makefile | 26 ++
 genprotimg/README.md | 101 ++++
 genprotimg/boot/.gitignore | 4 +
 genprotimg/boot/Makefile | 97 ++++
 genprotimg/boot/common_memory_layout.h | 25 +
 genprotimg/boot/head.S | 29 ++
 genprotimg/boot/stage3a.c | 62 +++
 genprotimg/boot/stage3a.h | 34 ++
 genprotimg/boot/stage3a.lds.S | 103 +++++
 genprotimg/boot/stage3a_init.S | 26 ++
 genprotimg/boot/stage3b.c | 77 ++++
 genprotimg/boot/stage3b.h | 42 ++
 genprotimg/boot/stage3b.lds.S | 87 ++++
 genprotimg/boot/stage3b_reloc.S | 53 +++
 genprotimg/man/Makefile | 12 +
 genprotimg/man/genprotimg.8 | 97 ++++
 genprotimg/src/Makefile | 101 ++++
 genprotimg/src/common.h | 39 ++
 genprotimg/src/genprotimg.c | 181 ++++++++
 genprotimg/src/include/pv_crypto_def.h | 25 +
 genprotimg/src/include/pv_hdr_def.h | 84 ++++
 genprotimg/src/pv/pv_args.c | 405 ++++++++++++++++
 genprotimg/src/pv/pv_args.h | 53 +++
 genprotimg/src/pv/pv_comp.c | 446 ++++++++++++++++++
 genprotimg/src/pv/pv_comp.h | 78 ++++
 genprotimg/src/pv/pv_comps.c | 252 ++++++++++
 genprotimg/src/pv/pv_comps.h | 42 ++
 genprotimg/src/pv/pv_error.c | 37 ++
 genprotimg/src/pv/pv_error.h | 62 +++
 genprotimg/src/pv/pv_hdr.c | 293 ++++++++++++
 genprotimg/src/pv/pv_hdr.h | 36 ++
 genprotimg/src/pv/pv_image.c | 820 +++++++++++++++++++++++++++++++++
 genprotimg/src/pv/pv_image.h | 68 +++
 genprotimg/src/pv/pv_ipib.c | 128 +++++
 genprotimg/src/pv/pv_ipib.h | 27 ++
 genprotimg/src/pv/pv_opt_item.c | 26 ++
 genprotimg/src/pv/pv_opt_item.h | 20 +
 genprotimg/src/pv/pv_stage3.c | 164 +++++++
 genprotimg/src/pv/pv_stage3.h | 30 ++
 genprotimg/src/utils/align.h | 24 +
 genprotimg/src/utils/buffer.c | 69 +++
 genprotimg/src/utils/buffer.h | 31 ++
 genprotimg/src/utils/crypto.c | 798 ++++++++++++++++++++++++++++++++
 genprotimg/src/utils/crypto.h | 104 +++++
 genprotimg/src/utils/file_utils.c | 234 ++++++++++
 genprotimg/src/utils/file_utils.h | 34 ++
 include/boot/ipl.h | 190 ++++++++
 include/boot/linux_layout.h | 34 ++
 include/boot/loaders_layout.h | 32 ++
 {zipl => include}/boot/s390.h | 131 ++----
 include/boot/sigp.h | 56 +++
 include/lib/util_base.h ...

------- Comment From <email address hidden> 2020-03-17 10:47 EDT-------
Did you use the command `git cherry-pick 971970989bbb1de..65b9fc442c1a4ff2458` to apply the commits? If so, please pick them individually.

The reason I ask for this is that the listed commits should not have any changes to zdev, cpumf, zdump, ...

On Tue, 17 Mar 2020 at 14:56, bugproxy <email address hidden> wrote:
>
> ------- Comment From <email address hidden> 2020-03-17 10:47 EDT-------
> Did you use the command `git cherry-pick 971970989bbb1de..65b9fc442c1a4ff2458` to apply the commits? If so, please pick them individually.
>
> The reason I ask for this is that the listed commits should not have any
> changes to zdev, cpumf, zdump, ...

In anticipation of Ubuntu Feature Freeze and outstanding feature work,
prior to Ubuntu feature freeze I have uploaded s390-tools snapshot
into Focal based on 12th of February code drop
33031241c320f0479b39f55a0f86c5786f4c1822.

The expectation was that any further updates to the s390-tools master
repository, will be in line with Ubuntu Feature Freeze policy and
would only contain bug fixes and features requested to ship in 20.04
GA.

Since 33031241c320f0479b39f55a0f86c5786f4c1822 master branch has had
49 commits pushed, out of which 37 commits are now requested to be
cherrypicked
out of order. Majority of commits are in this requested, one commit is
another request, and a third request duplicates 4 commits from this
one.

Over 20.04 lifetime, we will be having s390-tools SRUs, and if I now
take 37/49 commits from master, any future cherrypicks will have
conflicts.

I am attaching a list of commits that are in master branch, but not in
Ubuntu. Those prefixed with Y were requested in this request, those
prefixed with O were requested in other tickets. Note that commit list
that is requested to be applied by Ubuntu looks like this at the
moment:
skip 1, take 1, skip 1, take 2, skip 5, take 4, skip 2, take 22, skip
2, take 8, skip 1.

Also I question the requirement of some of these commits. For example:
Y b06af60 README.md: remove useless empty line
is not really necessary for this feature, yet is requested to be cherrypicked.

Overall the order of commits on master branch, and their priority look
very messy.

This is starting to look a lot like the OpenSSL cherrypick requests,
which didn't take into account previous backports that were requested
and uploaded into Ubuntu thus all the patches provided, conflicted
with each other.

So, shall I cherry pick 37/49 commits and have cherrypick conflicts
for the next 10 years?
Or will you clean up master branch to drop commits you don't want to
request for 20.04 GA?
Or should I take the whole of master branch?
Do you want to cut an s390-tools release from the master branch for us
to ship that?

--
Regards,

Dimitri.

Here is a list of commits that are in master, not in ubuntu, and have not yet been requested to be backported acorss any launchpad tickets:

f742ed7 dbginfo: gather ethtool output for per-queue coalescing
2c10642 cpumf_helper: Avoid perl warning from pod2usage function
1086548 cpumf: Add IBM z15 extended counter defintion file
fbf8513 zfcpdbf: print HBA FC Endpoint Security trace records
67496af zdev: Report FC Endpoint Security of zfcp devices
16b2799 zdev: Handle special case in if-case
c063273 zdev: Introduce read-only attributes
5d2871d cpumf/data: Add new deflate counters for IBM z15
6fcf64e lib/util_file.h: fix typo in the macro guard
4fa9656 dbginfo: collect softnet_stat
87b54fc CHANGELOG: Fix formatting
d415b8e dbginfo: Removed collection of /var/log/opencryptoki/

Please either drop them from master branch or the focal upstream maintenance branch, or let me know if you want them in 20.04 GA.

------- Comment From <email address hidden> 2020-03-17 14:47 EDT-------
(In reply to comment #23)
> Here is a list of commits that are in master, not in ubuntu, and have not
> yet been requested to be backported acorss any launchpad tickets:
>
> f742ed7 dbginfo: gather ethtool output for per-queue coalescing
> 2c10642 cpumf_helper: Avoid perl warning from pod2usage function
> 1086548 cpumf: Add IBM z15 extended counter defintion file
> fbf8513 zfcpdbf: print HBA FC Endpoint Security trace records
> 67496af zdev: Report FC Endpoint Security of zfcp devices
> 16b2799 zdev: Handle special case in if-case
> c063273 zdev: Introduce read-only attributes
> 5d2871d cpumf/data: Add new deflate counters for IBM z15
> 6fcf64e lib/util_file.h: fix typo in the macro guard
> 4fa9656 dbginfo: collect softnet_stat
> 87b54fc CHANGELOG: Fix formatting
> d415b8e dbginfo: Removed collection of /var/log/opencryptoki/
>
> Please either drop them from master branch or the focal upstream maintenance
> branch, or let me know if you want them in 20.04 GA.

Chief Product Owner for Linux on Z hat on: If picking all of the above commits is ok for Canonical (those are mostly bugfixes or hardware enablement) then this seems like the most straightforward solution to me.

This bug was fixed in the package s390-tools - 2.12.0-0ubuntu3

---------------
s390-tools (2.12.0-0ubuntu3) focal; urgency=medium

  * Update patch series to master tip:
    - PVM / genprotimg LP: #1834534, FFe LP: #1866866
    - zipl/libc: Fix potential buffer overflow LP: #1865032
    - zipl: Fix secureboot documentation LP: #1864654
    - Many other smaller bugfixes

 -- Dimitri John Ledkov <email address hidden> Fri, 20 Mar 2020 12:08:13 +0000

------- Comment From <email address hidden> 2020-03-23 04:31 EDT-------
IBM Bugzilla status-> closed, Fix Released with focal