Ubuntu on IBM z Systems

[FFe][20.10 FEAT] zlib/gzip hardware compression enablement

Bug #1884514 reported by bugproxy
16
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Fix Released
High
Skipper Bug Screeners
gzip (Ubuntu)
Fix Released
High
Canonical Foundations Team
zlib (Ubuntu)
Fix Released
Undecided
Canonical Foundations Team

Bug Description

[Feature Freeze Exception]

1. Feature:

The latest s390x hardware comes with a new concept for hardware assisted compression/decompression, based on a 'Next Accelerator Function unit' (NXU). This is an on-chip concept, a co-processor for compression available to all cores on the same chip. It provides functions as normal 'problem state' instructions (in other words user space instructions that are directly consumable by libraries and applications) and supports DEFLATE compliant compression/decompression + GZIP CRC/ZLIB Adler.

To enable this hardware support for zlib and gzip, zlib needs to be compiled with CFLAGS="-O2 -DDFLTCC -DDFLTCC_LEVEL_MASK=0x7e".
The value of 0x7e enables hardware compression for the compression levels 1-6 (out of 0-9).
There is a significant business value of having the enablement active by default, since tests on a system with 4 IFLs and hardware compression enabled this way offered (especially for DEFLATE as best case) a speed up of more than 40x.

2. Changes

Adding CFLAGS="-O2 -DDFLTCC -DDFLTCC_LEVEL_MASK=0x7e" attributes/macro during build of zlib and gzip.

3. Regression risk

The concept is new and comes with IBM z15 and LinuxONE III only and is with that specific and limited to s390x.
Hence a potential regressions of this late addition would be limited to s390x and there again with the above changes to zlib (and gzip).
In case of unforeseen issues with the NXU hardware component, a fallback to software is done.
On top a modified zlib package was build and made available in a PPA for further testing.
This change should have been included earlier, but was blocked by other zlib tickets/bugs that needed to be fixed and integrated first (like LP 1893170), hence this request for late addition.
__________

HW Compression need to be enabled with Default-Compression-Level 6.

Adding following CFLAGS attributes during build of zlib and gzip
"-DDFLTCC_LEVEL_MASK=0x7e"

CFLAGS="-O2 -DDFLTCC -DDFLTCC_LEVEL_MASK=0x7e" ./configure

See original description
bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-186315 severity-high targetmilestone-inin2010
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
Andrew Cloke (andrew-cloke)
Changed in ubuntu-z-systems:
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
importance: Undecided → High
Frank Heimes (fheimes)
no longer affects: linux (Ubuntu)
Changed in zlib (Ubuntu):
assignee: nobody → Canonical Foundations Team (canonical-foundations)
Changed in gzip (Ubuntu):
assignee: nobody → Canonical Foundations Team (canonical-foundations)
Changed in ubuntu-z-systems:
assignee: Canonical Kernel Team (canonical-kernel-team) → Skipper Bug Screeners (skipper-screen-team)
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: New → Triaged
Francis Ginther (fginther)
tags: added: id-5ef4ccb4eee44577120e7abb
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

https://launchpad.net/~mwhudson/+archive/ubuntu/scratch/+packages?field.name_filter=zlib will shortly contain a package with this change.

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2020-08-25 05:55 EDT-------
zlib1g_1.2.11.dfsg-2ubuntu2~ppa2_s390x.deb works fine.

It does not contain fixes for LP1889059 though.

Revision history for this message
Frank Heimes (fheimes) wrote : Re: [20.10 FEAT] zlib/gzip hardware compression enablement

I'm wondering a bit about the statement that it does not contain the fixes for LP 1889059,
since I saw this comment https://bugs.launchpad.net/ubuntu/+source/zlib/+bug/1889059/comments/10
So the package made available by Michael (comment #1) should fix both tickets - this (LP 1884514) and LP 1889059. Wondering what went wrong ...

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Yes as Frank says, it's certainly meant to have the fix for the other bug. Does it fail the tests for that bug?

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2020-08-26 06:50 EDT-------
Hi! I double-checked, and the test for LP1889059 actually passes. So zlib1g_1.2.11.dfsg-2ubuntu2~ppa2_s390x.deb is fully okay.

It's a number of tests for a not-yet-opened bug that fail, and I confused them with LP1889059 test. I already released the fix "upstream" (https://github.com/madler/zlib/pull/410#issuecomment-679384711) and I'm currently notifying the distros. So stay tuned..

Sorry about the confusion!

Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: Triaged → In Progress
Revision history for this message
Frank Heimes (fheimes) wrote : Re: [20.10 FEAT] zlib/gzip hardware compression enablement

Let me try to bring all the zlib related tickets together and summarize the situation,
since I've just noticed that this new one just came in:
LP 1893170 - 'zlib: DFLTCC compression level switching issues'.

So Michael's package zlib1g_1.2.11.dfsg-2ubuntu2~ppa2_s390x.deb solves LP 1884514 (according to LP 1884514, comment #2) as well as LP 1889059 (according to LP 1884514, comment #5).

And I think it should also solve LP 1882494, right? (But didn't saw a +1 for this)

But during the test of the package it was identified that another fix is needed (LP 1884514, comment #5 'stay tuned') that is now addressed with:
LP 1893170 - '[Ubuntu 20.10] zlib: DFLTCC compression level switching issues'.
Based on that commit 992a7afc3edfa511dff0650d1c545b11bf64e655 (mentioned in LP 1893170 is now needed on top (of the test package).

Did I got it right?

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2020-08-27 08:20 EDT-------
Right. The new LP 1893170 - 'zlib: DFLTCC compression level switching issues' actually improves the code that was introduced in LP 1882494 - 'zlib not working on all s390x systems configurations'.

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote : Re: [20.10 FEAT] zlib/gzip hardware compression enablement

Ah hah OK, I'll prepare packages with that patch instead then!

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Right sorry for the delay, I've built a test package for this bug at https://launchpad.net/~mwhudson/+archive/ubuntu/devirt/+packages?field.name_filter=zlib, now that the bug fix has migrated to groovy release.

Is there a need for this bug to be private? Uploading this to the archive now will require a FFe (https://wiki.ubuntu.com/FreezeExceptionProcess) and in any case it makes sense to reference the bug in the changelog.

Frank Heimes (fheimes)
information type: Private → Public
Frank Heimes (fheimes)
description: updated
Frank Heimes (fheimes)
summary: - [20.10 FEAT] zlib/gzip hardware compression enablement
+ [FFe][20.10 FEAT] zlib/gzip hardware compression enablement
Changed in ubuntu-z-systems:
status: In Progress → Triaged
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2020-10-01 05:08 EDT-------
This functionallity will only be exploited on z15. On older HW it will be changed to the software implementation.

Revision history for this message
Steve Langasek (vorlon) wrote :

FFe approved.

Changed in zlib (Ubuntu):
status: New → Triaged
Changed in gzip (Ubuntu):
status: New → Triaged
Revision history for this message
Frank Heimes (fheimes) wrote :

Many thx for the approval.

Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: Triaged → In Progress
Revision history for this message
Frank Heimes (fheimes) wrote :

The changes landed in groovy proposed:
zlib1g | 1:1.2.11.dfsg-2ubuntu3 | groovy-proposed | s390x
hence updating status to Fix Committed.

Changed in zlib (Ubuntu):
status: Triaged → Fix Committed
Changed in ubuntu-z-systems:
status: In Progress → Fix Committed
no longer affects: gzip (Ubuntu)
Revision history for this message
Frank Heimes (fheimes) wrote :

Successfully tested on z13, no regressions identified.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zlib - 1:1.2.11.dfsg-2ubuntu3

---------------
zlib (1:1.2.11.dfsg-2ubuntu3) groovy; urgency=medium

  * Enable hardware compression on s390x at level 6. LP: #1884514

 -- Michael Hudson-Doyle <email address hidden> Thu, 24 Sep 2020 08:44:35 +1200

Changed in zlib (Ubuntu):
status: Fix Committed → Fix Released
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-10-12 06:44 EDT-------
IBM Bugzilla status-> closed , Fix Released with groovy

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-10-13 06:13 EDT-------
The released zlib1g:s390x 1:1.2.11.dfsg-2ubuntu3 s390x passes all my tests.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-10-26 08:33 EDT-------
We did the zlib part here - but the bugzilla is called "zlib/gzip" - will there be gzip debs?

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

so gzip has the accel patches, and it is built

+ifeq (${DEB_TARGET_ARCH},s390x)
+CONFIGURE_ARGS+= --enable-dfltcc
+endif
+

However, I don't see that there is -DDFLTCC_LEVEL_MASK=0x7e let me open gzip task.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-11-18 08:36 EDT-------
@xnox, sorry, I didn't quite get your comment - do you need some info about gzip from us?

Frank Heimes (fheimes)
Changed in gzip (Ubuntu):
assignee: nobody → Canonical Foundations Team (canonical-foundations)
Steve Langasek (vorlon)
tags: added: fr-994
Dimitri John Ledkov (xnox)
Changed in gzip (Ubuntu):
status: New → In Progress
importance: Undecided → High
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package gzip - 1.10-2ubuntu3

---------------
gzip (1.10-2ubuntu3) hirsute; urgency=medium

  * Enable DFLTCC optimisations on s390x by default. LP: #1884514

 -- Dimitri John Ledkov <email address hidden> Thu, 28 Jan 2021 15:56:12 +0000

Changed in gzip (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2021-01-29 07:13 EDT-------
gzip (1.10-2ubuntu3) works fine, thanks!

Revision history for this message
Frank Heimes (fheimes) wrote :

Thx for the confirmation!

With that and all components of this bug on status Fix Released, this is closed now.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.