Please backport libzstd 1.3.1+dfsg-1 (universe) from artful

Bug #1717040 reported by Yann Collet
38
This bug affects 3 people
Affects Status Importance Assigned to Milestone
libzstd (Ubuntu)
Fix Released
Undecided
Andreas Hasenack
Xenial
Fix Released
Undecided
Andreas Hasenack
Zesty
Won't Fix
Undecided
Andreas Hasenack
Artful
Fix Released
Undecided
Andreas Hasenack
Bionic
Fix Released
Undecided
Andreas Hasenack

Bug Description

[Special SRU Notes]

The Artful SRU must be published before the Xenial SRU in order to preserve the upgrade path.

[Impact]

 * libzstd v0.5.1 is an experimental version,
   which generates and read an experimental format
   incompatible with official libzstd v1+ format.

 * Backporting a newer version >= v1.0, such as artful's v1.3.1,
   fixes this issue

 * This backport/SRU is being requested by upstream.

[Test Case]

There are three major test cases: compression tests, package upgrade tests, and release upgrade tests.

This SRU has so many tests because it's introducing a new package and a carefully thought out upgrade plan.

a) compression tests

 * with the pristine package from each release, compress a big file. For example, /usr/bin/snap. Name the resulting compressed file according to the version that was used to compress it, like this:

Xenial:
$ zstd /usr/bin/snap -o snap.0.5.1.zst
Compressed 15528016 bytes into 4134889 bytes ==> 26.63%

Artful:
$ zstd /usr/bin/snap -o snap.1.3.1.zst
/usr/bin/snap : 31.03% (11318760 => 3512707 bytes, snap.1.3.1.zst)

 * Upgrade to the package made available through this SRU and compress it again:

Xenial:
$ zstd /usr/bin/snap -o snap.1.3.1.zst
/usr/bin/snap : 25.11% (15528016 => 3899137 bytes, snap.1.3.1.zst)

Artful:
$ zstd /usr/bin/snap -o snap.1.3.1-updated.zst
/usr/bin/snap : 31.03% (11318760 => 3512707 bytes, snap.1.3.1-updated.zst)

* Uncompress all the generated files using the new version. This proves it can handle files compressed by the old one:

Xenial:
$ zstd -d snap.0.5.1.zst -o snap.0.5.1
snap.0.5.1.zst : 15528016 bytes
$ zstd -d snap.1.3.1.zst -o snap.1.3.1
snap.1.3.1.zst : 15528016 bytes

Artful:
$ zstd -d snap.1.3.1.zst -o snap.1.3.1
snap.1.3.1.zst : 11318760 bytes
$ zstd -d snap.1.3.1-updated.zst -o snap.1.3.1-updated
snap.1.3.1-updated.zst: 11318760 bytes

 * Take md5sums of all uncompressed files and the original. They must of course match:

Xenial:
$ md5sum /usr/bin/snap snap.0.5.1 snap.1.3.1
0fa4fa69d79ef4685aaa93be5b3aa33f /usr/bin/snap
0fa4fa69d79ef4685aaa93be5b3aa33f snap.0.5.1
0fa4fa69d79ef4685aaa93be5b3aa33f snap.1.3.1

Artful:
$ md5sum /usr/bin/snap snap.1.3.1 snap.1.3.1-updated
0b65998bc775d444b2ee7d00cd18634a /usr/bin/snap
0b65998bc775d444b2ee7d00cd18634a snap.1.3.1
0b65998bc775d444b2ee7d00cd18634a snap.1.3.1-updated

* Downgrade zstd back to the original version of the ubuntu release you are testing:

Xenial:
$ sudo apt install zstd=0.5.1-1

Artful:
$ sudo apt install zstd=1.3.1+dfsg-1 libzstd1=1.3.1+dfsg-1

* Try to decompress the zst file created with the 1.3.1 version from the previous test. In xenial only, it should fail to recognize the format:
$ zstd -d snap.1.3.1.zst -o /dev/null
zstd: /dev/null already exists; do you wish to overwrite (y/N) ? y
zstd: snap.1.3.1.zst: not in zstd format

In artful it should work:
$ zstd -d snap.1.3.1.zst -o snap.1.3.1-new
snap.1.3.1.zst : 11318760 bytes

* Still in artful, compare the md5, which should match:
$ md5sum snap.1.3.1-new /usr/bin/snap
0b65998bc775d444b2ee7d00cd18634a snap.1.3.1-new
0b65998bc775d444b2ee7d00cd18634a /usr/bin/snap

b) package upgrade tests
b.1)XENIAL

* Install the xenial original packages:
$ sudo apt install zstd=0.5.1-1 libzstd0=0.5.1-1 libzstd-dev=0.5.1-1

* Enable xenial-proposed and verify that a dist-upgrade will upgrade zstd, pull in the new libzstd1, and leave libzstd-dev alone:
$ sudo apt dist-upgrade
...
The following NEW packages will be installed:
  libzstd1
The following packages will be upgraded:
  zstd
1 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.

* In the end you will have these installed, at these versions:
ii libzstd-dev 0.5.1-1
ii libzstd0 0.5.1-1
ii libzstd1 1.3.1+dfsg-1~ubuntu0.16.04.1
ii zstd 1.3.1+dfsg-1~ubuntu0.16.04.1

* Installing the new versioned dev package should remove the unversioned one:
$ sudo apt install libzstd1-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  libzstd-dev
The following NEW packages will be installed:
  libzstd1-dev
0 upgraded, 1 newly installed, 1 to remove and 0 not upgraded.

* The end result being:
ii libzstd0 0.5.1-1
ii libzstd1 1.3.1+dfsg-1~ubuntu0.16.04.1
ii libzstd1-dev 1.3.1+dfsg-1~ubuntu0.16.04.1
ii zstd 1.3.1+dfsg-1~ubuntu0.16.04.1

* Conversely, installing the unversioned dev package removes the versioned one::
Installing libzstd-dev removes libzstd1-dev:
$ sudo apt install libzstd-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  libzstd1-dev
The following NEW packages will be installed:
  libzstd-dev
0 upgraded, 1 newly installed, 1 to remove and 0 not upgraded.

* Start from scratch and reinstall the original xenial zstd packages:
$ sudo apt install zstd=0.5.1-1 libzstd0=0.5.1-1 libzstd-dev=0.5.1-1

* Verify that a simple apt upgrade (not dist-upgrade) upgrades zstd and installs the new libzstd1:
$ sudo apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  libzstd1
The following packages will be upgraded:
  zstd
1 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 392 kB of archives.
After this operation, 904 kB of additional disk space will be used.
Do you want to continue? [Y/n]

* After the upgrade, you should have:
ii libzstd-dev 0.5.1-1
ii libzstd0 0.5.1-1
ii libzstd1 1.3.1+dfsg-1~ubuntu0.16.04.1
ii zstd 1.3.1+dfsg-1~ubuntu0.16.04.1

b.2) ARTFUL
* starting with all original artful zstd packages installed:
ii libzstd-dev 1.3.1+dfsg-1
ii libzstd1 1.3.1+dfsg-1
ii zstd 1.3.1+dfsg-1

* a dist-upgrade upgrades them just fine, with no new dependencies introduced:
The following packages will be upgraded:
  libzstd-dev libzstd1 zstd
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

* which results in:
ii libzstd-dev 1.3.1+dfsg-1ubuntu0.1
ii libzstd1 1.3.1+dfsg-1ubuntu0.1
ii zstd 1.3.1+dfsg-1ubuntu0.1

* libzstd1-dev, the transitional package, can also be installed just fine:
$ sudo apt install libzstd1-dev
The following NEW packages will be installed:
  libzstd1-dev
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.

* removing libzstd-dev also removes libzstd1-dev:
The following packages will be REMOVED:
  libzstd-dev* libzstd1-dev*
0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.

* installing just the new libzstd1-dev pull in the unversioned dev package:
$ sudo apt install libzstd1-dev
The following additional packages will be installed:
  libzstd-dev libzstd1
The following NEW packages will be installed:
  libzstd-dev libzstd1 libzstd1-dev
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.

c) release upgrade tests
RELEASE UPGRADE FROM XENIAL TO ARTFUL
1) starting with the 0.5.1 dev package in xenial:
ii libzstd-dev 0.5.1-1
ii libzstd0 0.5.1-1
ii zstd 0.5.1-1

dist-upgrade to proposed version:
ii libzstd-dev 0.5.1-1
ii libzstd0 0.5.1-1
ii libzstd1 1.3.1+dfsg-1~ubuntu0.16.04.1
ii zstd 1.3.1+dfsg-1~ubuntu0.16.04.1

in the do-release-upgrade upgrade list:
libzstd-dev libzstd1 zstd

deemed obsolete and removed in the end: libzstd0

end result in artful:
ii libzstd-dev 1.3.1+dfsg-1ubuntu0.1
ii libzstd1 1.3.1+dfsg-1ubuntu0.1
ii zstd 1.3.1+dfsg-1ubuntu0.1

2) starting with the 1.3.1 dev package in xenial:
ii libzstd0 0.5.1-1
ii libzstd1 1.3.1+dfsg-1~ubuntu0.16.04.1
ii libzstd1-dev 1.3.1+dfsg-1~ubuntu0.16.04.1
ii zstd 1.3.1+dfsg-1~ubuntu0.16.04.1

do-release-upgrade stats:
install list: libzstd-dev
upgrade list: libzstd1 libzstd1-dev zstd
remove (auto installed): libzstd0

end result:
ii libzstd-dev 1.3.1+dfsg-1ubuntu0.1
ii libzstd1 1.3.1+dfsg-1ubuntu0.1
ii libzstd1-dev 1.3.1+dfsg-1ubuntu0.1
ii zstd 1.3.1+dfsg-1ubuntu0.1

RELEASE UPGRADE FROM ARTFUL TO BIONIC

- with all zstd packages installed:
ii libzstd-dev 1.3.1+dfsg-1ubuntu0.1
ii libzstd1 1.3.1+dfsg-1ubuntu0.1
ii libzstd1-dev 1.3.1+dfsg-1ubuntu0.1
ii zstd 1.3.1+dfsg-1ubuntu0.1

do-release-upgrade stats:
upgrade list: libzstd-dev libzstd1 libzstd1-dev zstd
nothing else

end result in bionic:
ii libzstd-dev 1.3.3+dfsg-1ubuntu1
ii libzstd1 1.3.3+dfsg-1ubuntu1
ii libzstd1-dev 1.3.3+dfsg-1ubuntu1
ii zstd 1.3.3+dfsg-1ubuntu1

- without the transitional package installed:
ii libzstd-dev 1.3.1+dfsg-1ubuntu0.1
ii libzstd1 1.3.1+dfsg-1ubuntu0.1
ii zstd 1.3.1+dfsg-1ubuntu0.1

do-release-upgrade stats:
upgrade list: libzstd-dev libzstd1 zstd
nothing else

end result in bionic:
ii libzstd-dev 1.3.3+dfsg-1ubuntu1
ii libzstd1 1.3.3+dfsg-1ubuntu1
ii zstd 1.3.3+dfsg-1ubuntu1

[Regression Potential]

 * During transition, if some user switches to newer zstd version, and then send compressed data to another user has not yet updated zstd, the receiver will not be able to decompress as long as he does not update.

 * Note though that newer version contains a compatibility module which makes it able to read data generated by older versions, such as v0.5.1. Any existing document compressed with v0.5.1 will still be readable after update to v1.3.1.

[Other Info]

 * Linux kernel 4.14 now integrates zstd compression / decompression, primarily for BtrFS and SquashFS, with patches available for reiser4, zram, and initrd. Kernel version does not support compatibility module, hence cannot read data from v0.5.1. Userland tools associated with these services will fail if they link to Ubuntu LTS libzstd, as generating incompatible data.

 * On top of the incompatible format issue, libzstd v0.5.1 API is old and missing several features that applications relying on libzstd need, such as streaming interface, or bulk processing for dictionary compression.

--- Original description ---

Please backport libzstd 1.3.1+dfsg-1 (universe) from artful to xenial.

Reason for the backport:
========================
Current version in Xenial is v0.5.1,
it's an experimental version, using its own, incompatible format.
As a consequence, zstd on Ubuntu Xenial is not compatible with the rest of the world.

This is of important concern for products using libzstd as a shared library :
on Ubuntu Xenial, produced data is different, not compatible with v1+.

This issue has been made more pressing with the integration of zstd in Linux Kernel,
as userland tools must also be updated to read and generate zstd.

Note : this request was first improperly filled at : https://bugs.launchpad.net/zesty-backports/+bug/1717037

Testing:
========
Mark off items in the checklist [X] as you test them, but please leave the checklist so that backporters can quickly evaluate the state of testing.

Backport available from PPA:
https://launchpad.net/~ginggs/+archive/ubuntu/backports

* xenial:
[X] Package builds without modification (needs debhelper 10)
[X] zstd installs cleanly and runs
[ ] zstd-dbgsym installs cleanly and runs
[ ] libzstd1-dbgsym installs cleanly and runs
[X] libzstd-dev installs cleanly and runs
[X] libzstd1 installs cleanly and runs

No reverse dependencies

Graham Inggs (ginggs)
summary: - Please backport libzstd 1.3.1 from debian
+ Please backport libzstd 1.3.1+dfsg-1 (universe) from artful
description: updated
Revision history for this message
Yann Collet (cyan4973) wrote :

Successfully tested 2 packages : zsdt, libzstd-dev, libzstd1

Revision history for this message
Yann Collet (cyan4973) wrote :

I meant "3" packages ...

description: updated
Graham Inggs (ginggs)
Changed in xenial-backports:
status: New → Confirmed
Revision history for this message
Yann Collet (cyan4973) wrote :

Any new on this backport ?

Revision history for this message
Yann Collet (cyan4973) wrote :

ping

Revision history for this message
David Britton (dpb) wrote :

Hi Yann -- Why is -dbgsym untested?

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Given your arguments, we are tempted to make this an official SRU instead of a backport.

To help argue the case, could you come up with more detailed examples showing how the current version in xenial is incompatible with the world?

Some ideas:
- you said the kernel has incorporated zstd, do you have an example? Perhaps even better, an example that shows how zstd 0.5.1 in xenial is incompatible with what the xenial kernel (or hwe kernel for xenial) supports?
- can we demonstrate that zstd in xenial cannot decompress some zstd file compressed with the current upstream version? Perhaps a public file or URL somewhere even?

Here is a short discussion about the SRU possibilities for this package in #ubuntu-devel on freenode: https://irclogs.ubuntu.com/2017/12/06/%23ubuntu-devel.html#t19:53

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Oh, I must have done something wrong before, because zstd 0.5 did decompress a file compressed by 1.3 in my test. Now I can see that's not the case:

$ zstd -d snap.v1.3.zst -o decompressed #using zstd 0.5.1
zstd: snap.v1.3.zst: not in zstd format

So that shows files compressed with 1.3 cannot be decompressed by 0.5. That could be used as an argument *against* the SRU, though :) Do you have pointers showing that 0.5 was experimental, that things changed and that it should be dropped?

Checking the release notes, looks like v0.8 is the first release with the final compression format? https://github.com/facebook/zstd/releases/tag/v0.8.0

Revision history for this message
Yann Collet (cyan4973) wrote :

> Why is -dbgsym untested?

The PPA I tested did not contain these packages

Revision history for this message
Yann Collet (cyan4973) wrote :

Regarding format version :

- All versions < v1.0 are considered experimental. This used to be clearly labelled on the project's homepage, though it has been updated since then. zstd is only branded and supported by Facebook since v1.0.
- Technically, the compression format has been settled at v0.8.0, so all versions >= v0.8 are compatible with each other.
- zstd contains an optional backward compatibility module, which is enabled by default in both CLI and library. This module makes it possible for newer versions to read older versions, but *not the other way round*. Therefore, v1.0+ will be able to read data produced by v0.5.1, but v0.5.1 will not be able to read data from v1.0+.
- Note that the compatibility module is optional, and a number of projects have selected to not include it. Such projects are therefore unable to read data produced with v0.5.1. This includes, for example, the Linux Kernel codec (integrated in 4.14).

Hope it helps

Revision history for this message
Yann Collet (cyan4973) wrote :

Note that the format is an (important) part of the problem,
but there is also the topic of the API, which has changed much since v0.5.1.

The simplest prototypes are still the same (ZSTD_compress(), ZSTD_decompress()),
but advanced ones have evolved.
The streaming API is very different, and the older one, present in v0.5.1, is now considered deprecated (it generates compilation warnings, and in a future version, symbols will be removed).
The Dictionary API for Bulk processing doesn't exist.
Even minor details matter : all objects in zstd share a common pattern starting with ZSTD_createObject() and ending with ZSTD_freeObject(). For convenience, ZSTD_freeObject() accepts NULL pointers, like free(). But that's only true since >= v0.7.0, hence not in v0.5.1.

The API has stabilised in v0.8.1, adding only a couple of capabilities since, and guaranteeing continued support for existing interfaces labelled "stable" (all symbols published by default).

A number of applications expecting an API >= 0.8.1 will fail with libzstd v0.5.1, either at link stage, because a symbol is missing, or even sometimes at run stage, because a behaviour is different.

Changed in libzstd (Ubuntu):
status: New → Confirmed
no longer affects: libzstd (Ubuntu)
affects: xenial-backports → libzstd (Ubuntu)
description: updated
Revision history for this message
Yann Collet (cyan4973) wrote :

New template updated

description: updated
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Is the zesty version ok? 1.1.2?

Revision history for this message
Yann Collet (cyan4973) wrote :

Yes, it would be a pretty strong step forward compared to v0.5.1.

v1.1.2 is from December 2016 : https://github.com/facebook/zstd/releases/tag/v1.1.2

The main feature developed since then is multithreaded compression.
I guess a few users will miss it, but it's a minor setback compared to current situation.

PS : any reason why zesty's 1.1.2 would be preferable to artful's 1.3.1 ?

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I ask because otherwise we have to justify why we would be updating the zesty version (1.1.2) to artful's (1.3.1).

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

It's a justification issue, and also trying to keep the general spirit of *Stable* Release Updates (emphasis mine). What problems would we be fixing by updating zesty's zstd from 1.1.2 to 1.3.1, and if they are worth such an update. If it's just multithreading capability, then I think we can skip zesty, given that artful has it and zesty will be end of life in about a month (Jan 2018).

Revision history for this message
Yann Collet (cyan4973) wrote :

Multithreading is the main new capability I can think of,
and that's probably the one users will miss the most,
but there are quite a few more ones, listed on the release page, and in the NEWS document :
https://github.com/facebook/zstd/releases
https://github.com/facebook/zstd/blob/dev/NEWS#L24

The list is pretty long though, and just copy/pasting it wouldn't give a sense of priority.
Let's say there are a lot of small improvements and little bug fixing contributing to an overall better experience. Given a choice, I would recommend v1.3.1 over v1.1.2.

Changed in libzstd (Ubuntu Xenial):
assignee: nobody → Andreas Hasenack (ahasenack)
Changed in libzstd (Ubuntu Zesty):
assignee: nobody → Andreas Hasenack (ahasenack)
Changed in libzstd (Ubuntu Xenial):
status: New → In Progress
Changed in libzstd (Ubuntu Zesty):
status: New → In Progress
Changed in libzstd (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Yann Collet (cyan4973) wrote :

Any news on this update ?

Revision history for this message
David Britton (dpb) wrote :

Hey Yann -- no news. Still in progress.

Revision history for this message
Andreas Hasenack (ahasenack) wrote : Re: [Bug 1717040] Re: Please backport libzstd 1.3.1+dfsg-1 (universe) from artful

It's my next thing to do

On Dec 15, 2017 20:35, "David Britton" <email address hidden> wrote:

> Hey Yann -- no news. Still in progress.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1717040
>
> Title:
> Please backport libzstd 1.3.1+dfsg-1 (universe) from artful
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/libzstd/+bug/
> 1717040/+subscriptions
>

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hmm, why did it have to require a newer debhelper...

I don't think an SRU can depend on something that's in backports. I will have to read up on "Standards-Version" and debhelper versions to see what I will have to change in the package to build properly with xenial's debhelper (version 9.x).

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Ok, after an analysis done by a core dev (thanks @paelzer!), I have a way forward with this.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :
description: updated
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

@cyan4973, may I change your first test case into something that won't involve a download/build/install cycle? I was thinking about just using the 0.5.1 version in xenial to try to decompress the 1.3.1 compressed file I created in the test I just added to the bug description. Something like this:

"""
* Downgrade zstd back to xenial's original version:

$ sudo apt install zstd=0.5.1-1

* Try to decompress the zst file created with the 1.3.1 version from the previous test. It should fail to recognize the format:
$ zstd -d snap.1.3.1.zst -o /dev/null
zstd: /dev/null already exists; do you wish to overwrite (y/N) ? y
zstd: snap.1.3.1.zst: not in zstd format
"""

Would that be ok, and still be in the spirit of this update?

Revision history for this message
Yann Collet (cyan4973) wrote :

@ahasenack : yes, completely

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Review on the MPs is complete and it is now sponsored for SRU review in [xenial/zesty]-unapproved.

description: updated
description: updated
description: updated
Revision history for this message
Robie Basak (racb) wrote :

Yann, thank you for the detailed analysis. This is very helpful in giving us confidence in making changes to the stable releases.

The addition of a libzstd1, and the update of zstd itself to use it, sounds reasonable to me.

An additional thing for Regression Potential though I think?

We're deliberately breaking the API provided by libzstd-dev in Xenial then, according to Yann's comment 10. If a user is relying on some local build against libzstd-dev then this proposed SRU will break it. We should call this out and make a decision on it. I presume Andreas and Yann are both in favor? I think this is what Adam was referring to in https://irclogs.ubuntu.com/2017/12/06/%23ubuntu-devel.html#t20:12 ? It doesn't sound counter to the intent to me. Any opinions on using libzstd1-dev in the SRU instead to mitigate this? Or are you asking to specifically not do that and change the API of libzstd-dev itself?

I'm not particularly worried about justifications for Zesty. It's about to go EOL anyway. Anything that is acceptable for Xenial should also be acceptable for Zesty unless I'm missing some particular case where the circumstances are different.

Revision history for this message
Yann Collet (cyan4973) wrote :

Hi Robie

As far as I can tell, it is much more likely that an application expecting zstd >=v1.0.0 will break, most likely at link stage, and maybe at run stage, when provided with v0.5.1 instead.
The other way round should be less problematic : applications expecting to work with v0.5.1 should still work with v.1.3.1.

The reason is :
All symbols defined in v0.5.1 are also present in the "stable" section of v1.3.1.

For reference, here is the API for v0.5.1 : https://github.com/facebook/zstd/blob/v0.5.1/lib/zstd.h
and here is the API for v1.3.1 : https://github.com/facebook/zstd/blob/v1.3.1/lib/zstd.h

The major counter example is the streaming API.
This one used to be available in a separate package :
https://github.com/facebook/zstd/blob/v0.5.1/lib/zbuff.h
It is clearly labelled experimental, hence does not provide any API stability guarantee.

Nonetheless, in v1.3.1, this API is still available here :
https://github.com/facebook/zstd/blob/v1.3.1/lib/deprecated/zbuff.h
It is the exact same API, defining the same symbols.
What has changed is that this API is now branded "deprecated", which means it will generate deprecation warnings during compilation (except if these warnings are explicitly disabled, using typically -Wno-deprecated-declarations for gcc, or the custom macro ZBUFF_DISABLE_DEPRECATE_WARNINGS).

The deprecation warnings include comments to invite users to migrate towards the new "stable" streaming API, defined here : https://github.com/facebook/zstd/blob/v1.3.1/lib/zstd.h#L248
Of course, this is only useful for developers.

Revision history for this message
Robie Basak (racb) wrote :
Download full text (3.4 KiB)

How about this:

1. Add transitional packages libzstd1-dev -> libzstd-dev in Artful and Bionic (no Conflicts).

2. Backport what is in Artful, but drop the transitional libzstd1-dev package, rename libzstd-dev to libzstd1-dev and have it Conflicts with libzstd-dev.

This will land after Zesty EOLs, so don't worry about Zesty.

This will result in:

The zstd CLI from the zstd package will be updated, linked against libzstd1 and using new wire protocol.

libzstd.so.1 will become available in Xenial. libzstd.so.0 will remain available.

libzstd-dev will continue to exist and will not change, with the latest version remaining in the release pocket.

libzstd1-dev will be added. Users will only be able to have either libzstd-dev or libzstd1-dev installed at once. The one installed will determine whether the old API or the new API is used, and whether the link will be against the old or new soname.

Users already building using libzstd-dev locally in Xenial will not break and will continue to be able to rebuild (ie. "stay green") without any required intervention. If they wish, they can choose to use libzstd1-dev in Xenial instead. Or they can switch to use a newer release of Ubuntu or build directly against upstream sources, which I think is more likely for this use case anyway. Switching to libzstd1-dev will require code changes due to the API change, but this way this won't be forced upon them.

Users building against libzstd-dev locally who switch to libzstd1-dev will be automatically transitioned to libzstd-dev again upon upgrade to Artful or Bionic. Users who hardcode the libzstd1-dev name in any automated builds will break after upgrading beyond Bionic, since we'll drop the transitional package at that point, so should really do something like "if release == Xenial: install libzstd1-dev; else install libzstd-dev" if they want to use a distribution provided library on Xenial *and* build against the newer wire protocol and library.

If a security or bugfix updates to libzstd.so.0 becomes required, we'll need to adjust the release pocket source package to build only the libzstd0 binary package at that time.

Yann: would you be happy with this?

Rationale:

There is an SRU policy exception "to adjust to changes in the environment, server protocols, web services, and similar". I think, under this exception, we want an SRU to cause the zstd CLI to produce new format output with backwards compatibility, and for libzstd.so.1 to be made available so that other third parties who link against the library we ship can do that also (we'd update other packages in the Ubuntu release too, but we don't have any in this particular case).

> It is clearly labelled experimental, hence does not provide any API stability guarantee.

Separate from an upstream guarantee though, Ubuntu _does_ try to ensure that stable releases do not change behaviour, except when it is deliberate and the project chooses to make an exception. This guarantee is separate from any upstream guarantees. A blanket policy by a distribution is much easier for users to rely upon versus individual upstream guarantees that may not all match. Part of the point of a distribution that does stable re...

Read more...

Revision history for this message
Yann Collet (cyan4973) wrote :

I have a feeling that I don't fully grasp all the implied side effects here,
so if this process feels safer based on your experience, please go ahead.

I don't think the API will be an issue in this upgrade : as stated, all published symbols in v0.5.1 are still published in v1.3.1, so I would expect such application to continue linking properly.

The change of format is another thing, and an application hardwired to deal with v0.5.x experimental format, if such a thing exist, would be surprised to deal with data using v1.x format. So some caution in the transition is indeed welcomed.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Ok, I'll handle it with a new set of packages.

Revision history for this message
Graham Inggs (ginggs) wrote :

"...an application hardwired to deal with v0.5.x experimental format, if such a thing exist..."

@ahasenack: Isn't this precisely we have Ubuntu Backports?
We are saying:
here is a new version of this software
we have checked that it doesn't break anything in the archive
you need to opt-in to install it
and if you've built some local packages with it, you may have some breakage

Revision history for this message
Robie Basak (racb) wrote :

On Tue, Jan 16, 2018 at 07:37:15PM -0000, Graham Inggs wrote:
> "...an application hardwired to deal with v0.5.x experimental format, if
> such a thing exist..."
>
> @ahasenack: Isn't this precisely we have Ubuntu Backports?

I think we're somewhere in the middle.

AIUI, virtually nothing else in the wider ecosystem can consume output
produced by Xenial's zstd CLI tool. By publishing an update into
xenial-updates, we'd be recommending that users update to it
automatically (in practice the majority will get it without taking a
separate step) because the version released with Xenial is effectively
broken in terms of interoperability.

> you need to opt-in to install it

The point of considering an SRU for this fix, rather than backports, is
precisely to remove this requirement. Users expect broken stuff to be
fixed without having to individually opt-in to each fix, but instead by
installing general "updates".

I can see how this particular proposed update could go either way. But
right now, we seem to be swinging in the direction of releasing it as an
SRU, especially because we have a way to do that with no known downside.
Our existing SRU policy does permit an update as a result of a change in
the wider Internet environment. That we have an interop problem because
the zstd CLI produces output that can't practically be handled anywhere
else does, IMHO, qualify it for an SRU under this existing policy
permission. The question is if and how we do it in the face of
regression risk to existing users.

Comments welcome, but please frame the discussion in terms of what we're
trying to do and in terms of our existing SRU policy documented at
https://wiki.ubuntu.com/StableReleaseUpdates.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Yeah, but the backports process is kind of broken and seeing no traction, that's why we are trying an SRU.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I have some good results. Xenial first, and artful in a subsequent but comment.

XENIAL (details in the attached zstd-xenial.txt file):
starting with:
libzstd-dev 0.5.1-1
libzstd0 0.5.1-1
zstd 0.5.1-1

results in:
libzstd-dev 0.5.1-1
libzstd0 0.5.1-1
libzstd1 1.3.1+dfsg-1~ubuntu0.16.04.1~ppa3
zstd 1.3.1+dfsg-1~ubuntu0.16.04.1~ppa3

Installing the new 1.3.1 dev package results in the removal of the old one:
The following packages will be REMOVED:
  libzstd-dev
The following NEW packages will be installed:
  libzstd1-dev

which gives you:
libzstd0 0.5.1-1
libzstd1 1.3.1+dfsg-1~ubuntu0.16.04.1~ppa3
libzstd1-dev 1.3.1+dfsg-1~ubuntu0.16.04.1~ppa3
zstd 1.3.1+dfsg-1~ubuntu0.16.04.1~ppa3

RELEASE UPGRADE TO ARTFUL
a) starting with old 0.5.x dev package:
libzstd-dev 0.5.1-1
libzstd0 0.5.1-1
libzstd1 1.3.1+dfsg-1~ubuntu0.16.04.1~ppa3
zstd 1.3.1+dfsg-1~ubuntu0.16.04.1~ppa3

You end up with:
libzstd-dev 1.3.1+dfsg-1ubuntu0.1~ppa2
libzstd1 1.3.1+dfsg-1ubuntu0.1~ppa2
zstd 1.3.1+dfsg-1ubuntu0.1~ppa2

b) starting with new dev package:
libzstd0 0.5.1-1
libzstd1 1.3.1+dfsg-1~ubuntu0.16.04.1~ppa3
libzstd1-dev 1.3.1+dfsg-1~ubuntu0.16.04.1~ppa3
zstd 1.3.1+dfsg-1~ubuntu0.16.04.1~ppa3

You end up with:
libzstd-dev 1.3.1+dfsg-1ubuntu0.1~ppa2
libzstd1 1.3.1+dfsg-1ubuntu0.1~ppa2
libzstd1-dev 1.3.1+dfsg-1ubuntu0.1~ppa2
zstd 1.3.1+dfsg-1ubuntu0.1~ppa2

Revision history for this message
Andreas Hasenack (ahasenack) wrote :
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

ARTFUL (details in the attached zstd-artful.txt file):

starting with:
libzstd-dev 1.3.1+dfsg-1
libzstd1 1.3.1+dfsg-1
zstd 1.3.1+dfsg-1

results in (no big deal):
libzstd-dev 1.3.1+dfsg-1ubuntu0.1~ppa2
libzstd1 1.3.1+dfsg-1ubuntu0.1~ppa2
zstd 1.3.1+dfsg-1ubuntu0.1~ppa2

installing the new libzstd1-dev transitional package changes nothing, as expected:
The following NEW packages will be installed:
  libzstd1-dev
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.

but removing libzstd-dev also removes libzstd1-dev:
The following packages will be REMOVED:
  libzstd-dev* libzstd1-dev*
0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.

and if you start from nothing and install libzstd1-dev (the transitional package);
The following additional packages will be installed:
  libzstd-dev libzstd1
The following NEW packages will be installed:
  libzstd-dev libzstd1 libzstd1-dev
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.

you also get the "real" dev package, and the actual library.

I'll do the same layout for bionic.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :
Changed in libzstd (Ubuntu Zesty):
status: In Progress → Won't Fix
Changed in libzstd (Ubuntu Artful):
status: New → Confirmed
status: Confirmed → In Progress
assignee: nobody → Andreas Hasenack (ahasenack)
Changed in libzstd (Ubuntu):
status: Fix Released → In Progress
assignee: nobody → Andreas Hasenack (ahasenack)
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I have to retest this. d-shlibmove (why is that needed btw? Never heard of this yet another helper) is complaining the libzstd1-dev has no provides for libzstd-dev.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I repeated the tests with the xenial package having the extra "Provides: libzstd-dev" and saw no ill effects. Attaching the new txt files with the details.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

This artful test now includes a release-upgrade run to bionic

Revision history for this message
Robie Basak (racb) wrote :

The testing looks good, thanks. Could you please check one more thing? Rather than "dist-upgrade" on Xenial, could you please check that the new zstd is pulled in on a plain "apt-get upgrade" and via the update manager? If apt declines because it involves installing libzstd1 then the SRU will be moot. I think it should work, but it's probably worth explicitly checking.

For SRU verification, I'd be quite happy if you performed the Test Case as in the bug description, the testing you described in comments 42 and 43, and my request above. Please could you update the bug description to make it clear that this is the SRU verification plan? Feel free to just say "see comment X" up there. I just want to ensure that it isn't missed.

I've left a couple of questions in the MP against Xenial I'd like cleared up. Apart from that, looks good from an ~ubuntu-sru review perspective. Thanks!

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

xenial apt upgrade test:
a) zstd, libzstd0, libzstd-dev installed
ubuntu@xenial-zstd-sru-1717040:~$ dpkg-query -W zstd libzstd0 libzstd-dev
libzstd-dev 0.5.1-1
libzstd0 0.5.1-1
zstd 0.5.1-1

ubuntu@xenial-zstd-sru-1717040:~$ sudo apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'sudo apt autoremove' to remove it.
The following NEW packages will be installed:
  libzstd1
The following packages will be upgraded:
  zstd
1 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 392 kB of archives.
After this operation, 904 kB of additional disk space will be used.
Do you want to continue? [Y/n]

after:
ubuntu@xenial-zstd-sru-1717040:~$ dpkg -l|grep zstd
ii libzstd-dev 0.5.1-1 amd64 fast lossless compression algorithm -- development files
ii libzstd0 0.5.1-1 amd64 fast lossless compression algorithm
ii libzstd1 1.3.1+dfsg-1~ubuntu0.16.04.1~ppa5 amd64 fast lossless compression algorithm
ii zstd 1.3.1+dfsg-1~ubuntu0.16.04.1~ppa5 amd64 fast lossless compression algorithm -- CLI tool

ubuntu@xenial-zstd-sru-1717040:~$ sudo apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  libfreetype6 libzstd0
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
  libzstd1
The following packages will be upgraded:
  zstd
1 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 392 kB of archives.
After this operation, 904 kB of additional disk space will be used.
Do you want to continue? [Y/n]

end result:
ubuntu@xenial-zstd-sru-1717040:~$ dpkg -l|grep zstd
ii libzstd0 0.5.1-1 amd64 fast lossless compression algorithm
ii libzstd1 1.3.1+dfsg-1~ubuntu0.16.04.1~ppa5 amd64 fast lossless compression algorithm
ii zstd 1.3.1+dfsg-1~ubuntu0.16.04.1~ppa5 amd64 fast lossless compression algorithm -- CLI tool

c) motd

motd acks the available upgrades:
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-112-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

2 packages can be updated.
0 updates are security updates.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

xenial desktop upgrade via notification:
Starting with these installed:
libzstd-dev 0.5.1-1
libzstd0 0.5.1-1
zstd 0.5.1-1

End result after the gui updated it (see screenshot for the prompt):
ii libzstd-dev 0.5.1-1 amd64 fast lossless compression algorithm -- development files
ii libzstd0 0.5.1-1 amd64 fast lossless compression algorithm
ii libzstd1 1.3.1+dfsg-1~ubuntu0.16.04.1~ppa5 amd64 fast lossless compression algorithm
ii zstd 1.3.1+dfsg-1~ubuntu0.16.04.1~ppa5 amd64 fast lossless compression algorithm -- CLI tool

Robie Basak (racb)
description: updated
description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libzstd - 1.3.3+dfsg-1ubuntu1

---------------
libzstd (1.3.3+dfsg-1ubuntu1) bionic; urgency=medium

  * d/control: New transitional package libzstd1-dev to facilitate the release
    upgrade from xenial (LP: #1717040)
  * d/rules: also run the clean Makefile target in the pzstd directory.

 -- Andreas Hasenack <email address hidden> Fri, 02 Feb 2018 11:29:34 -0200

Changed in libzstd (Ubuntu Bionic):
status: In Progress → Fix Released
Revision history for this message
Yann Collet (cyan4973) wrote :

Is the Xenial patch good to go ?
Or is there any blocker remaining at this stage ?

Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello Yann, or anyone else affected,

Accepted libzstd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libzstd/1.3.1+dfsg-1~ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in libzstd (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-xenial
Revision history for this message
Andreas Hasenack (ahasenack) wrote : Re: [Bug 1717040] Re: Please backport libzstd 1.3.1+dfsg-1 (universe) from artful

I pinged the SRU team again today. The package was uploaded to proposed but
needs SRU approval to actually migrate to it and become available.

On Wed, Feb 14, 2018 at 8:41 PM, Yann Collet <email address hidden>
wrote:

> Is the Xenial patch good to go ?
> Or is there any blocker remaining at this stage ?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1717040
>
> Title:
> Please backport libzstd 1.3.1+dfsg-1 (universe) from artful
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/libzstd/+bug/
> 1717040/+subscriptions
>

Revision history for this message
Robie Basak (racb) wrote :

Hello Yann, or anyone else affected,

Accepted libzstd into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libzstd/1.3.1+dfsg-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in libzstd (Ubuntu Artful):
status: In Progress → Fix Committed
tags: added: verification-needed-artful
Revision history for this message
Robie Basak (racb) wrote :

@Yann

It was in the SRU queue. I've processed that now, so it will build shortly in xenial-proposed and artful-proposed and be published as a proposed update visible to all users. Then there's a seven day testing and aging period. After that, I'll look at the test reports and if all is well we can publish it into xenial-updates (and artful-updates) which will make it available as a general update to all users.

Revision history for this message
Robie Basak (racb) wrote :

Looks like ubuntustudio uses libzstd via blender -> libopenvdb3.2 -> libblosc1. Seems pretty obscure though - far from the common use case of blender AFAIK.

Revision history for this message
Yann Collet (cyan4973) wrote :

I tried to follow the procedure presented on https://wiki.ubuntu.com/Testing/EnableProposed,
but got blocked at this stage :

```
sudo apt-get install zstd/xenial-proposed
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Release 'xenial-proposed' for 'zstd' was not found
```

and indeed, a further call to `apt list --upgradable`
does not show `zstd/xenial-proposed` in the list of packages.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Yeah, it's not published yet for some reason:

https://launchpad.net/ubuntu/+source/libzstd

shows this for xenial:

" Note: Some binary packages for this source are not yet published in the repository."

Same for artful

@racb do you know if something is wrong, or is it just the nowadays normal delay we have in builders and publishers and what else?

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

It was stuck in the NEW queue, and just approved. Should arrive in the proposed pocket soon.

Revision history for this message
Yann Collet (cyan4973) wrote :

I could test `apt-get install zstd/xenial-proposed` today.

And it works as intended : library is there and usable, cli works fine too.

Just as a little detail, here is the message I'm receiving during installation :

```
The following additional packages will be installed:
  libzstd1
The following packages will be REMOVED:
  libzstd-dev
The following NEW packages will be installed:
  zstd
The following packages will be DOWNGRADED:
  libzstd1
0 upgraded, 1 newly installed, 1 downgraded, 1 to remove and 157 not upgraded.
Need to get 391 kB of archives.
After this operation, 258 kB of additional disk space will be used.

```

I was surprised by the DOWNGRADED statement
because libzstd1 wasn't installed on this VM, so there should be nothing to downgrade.

Anyway, this is mostly a cosmetic issue,
final statement is : it works all fine.

Revision history for this message
Yann Collet (cyan4973) wrote :

I've got an explanation for the "DOWNGRADED" statement :
I had ppa:ginggs/backports in my list of ppa.
This ppa contains an updated version of zstd.
So that's why when applying `apt-get install zstd/xenial-proposed`, it features the "DOWNGRADED" statement.

Removing the PPA, and re-installing the default zstd v0.5.1 from Xenial, I now get :
```
sudo apt install zstd/xenial-proposed
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '1.3.1+dfsg-1~ubuntu0.16.04.1' (Ubuntu:16.04/xenial-proposed [amd64]) for 'zstd'
Selected version '5.4.0-6ubuntu1~16.04.9' (Ubuntu:16.04/xenial-proposed [amd64]) for 'libstdc++6' because of 'zstd'
The following additional packages will be installed:
  libzstd1
The following NEW packages will be installed:
  libzstd1
The following packages will be upgraded:
  zstd
1 upgraded, 1 newly installed, 0 to remove and 157 not upgraded.
Need to get 0 B/391 kB of archives.
After this operation, 905 kB of additional disk space will be used.
```

which is the expected outcome.

then :
```
zstd -V
*** zstd command line interface 64-bits v1.3.1, by Yann Collet ***
```

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I'll post my verification in steps, since the list of checks is long.

a) compression tests - xenial - PASS
ubuntu@xenial-zstd:~$ zstd /usr/bin/snap -o snap.0.5.1.zst
Compressed 17183248 bytes into 4541079 bytes ==> 26.43%
ubuntu@xenial-zstd:~$ ll snap.0.5.1.zst
-rw-rw-r-- 1 ubuntu ubuntu 4541079 Feb 19 17:52 snap.0.5.1.zst
ubuntu@xenial-zstd:~$

Package from proposed:
 *** 1.3.1+dfsg-1~ubuntu0.16.04.1 400
        400 http://br.archive.ubuntu.com/ubuntu xenial-proposed/universe amd64 Packages

ubuntu@xenial-zstd:~$ zstd /usr/bin/snap -o snap.1.3.1.zst
/usr/bin/snap : 24.91% (17183248 => 4281096 bytes, snap.1.3.1.zst)
ubuntu@xenial-zstd:~$ ll snap.1.3.1.zst
-rwxr-xr-x 1 ubuntu ubuntu 4281096 Nov 30 19:48 snap.1.3.1.zst*

Uncompressing:
ubuntu@xenial-zstd:~$ zstd -d snap.0.5.1.zst -o snap.0.5.1
snap.0.5.1.zst : 17183248 bytes
ubuntu@xenial-zstd:~$ zstd -d snap.1.3.1.zst -o snap.1.3.1
snap.1.3.1.zst : 17183248 bytes

md5 verification is OK:
ubuntu@xenial-zstd:~$ md5sum /usr/bin/snap snap.0.5.1 snap.1.3.1
50ff3c166c443c32f043294416a5b5d9 /usr/bin/snap
50ff3c166c443c32f043294416a5b5d9 snap.0.5.1
50ff3c166c443c32f043294416a5b5d9 snap.1.3.1

xenial downgrade fails to recognize the new format, as expected:
ubuntu@xenial-zstd:~$ zstd -d snap.1.3.1.zst -o /dev/null
zstd: /dev/null already exists; do you wish to overwrite (y/N) ? y
zstd: snap.1.3.1.zst: not in zstd format

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

The artful package is stuck in the "new" queue as well, I'll skip the artful tests for now.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :
Download full text (3.9 KiB)

b) xenial package upgrade tests - PASS
starting with:
$ dpkg -l|grep zstd
ii libzstd-dev 0.5.1-1 amd64 fast lossless compression algorithm -- development files
ii libzstd0 0.5.1-1 amd64 fast lossless compression algorithm
ii zstd 0.5.1-1 amd64 fast lossless compression algorithm -- CLI tool

dist-upgrade:
The following NEW packages will be installed:
  libzstd1
The following packages will be upgraded:
  zstd
1 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.

results in:
ubuntu@xenial-zstd:~$ dpkg -l|grep zstd
ii libzstd-dev 0.5.1-1 amd64 fast lossless compression algorithm -- development files
ii libzstd0 0.5.1-1 amd64 fast lossless compression algorithm
ii libzstd1 1.3.1+dfsg-1~ubuntu0.16.04.1 amd64 fast lossless compression algorithm
ii zstd 1.3.1+dfsg-1~ubuntu0.16.04.1 amd64 fast lossless compression algorithm -- CLI tool

Installing the new versioned dev package should remove the unversioned one:
ubuntu@xenial-zstd:~$ sudo apt install libzstd1-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  libzstd-dev
The following NEW packages will be installed:
  libzstd1-dev

end result:
ubuntu@xenial-zstd:~$ dpkg -l|grep zstd
ii libzstd0 0.5.1-1 amd64 fast lossless compression algorithm
ii libzstd1 1.3.1+dfsg-1~ubuntu0.16.04.1 amd64 fast lossless compression algorithm
ii libzstd1-dev 1.3.1+dfsg-1~ubuntu0.16.04.1 amd64 fast lossless compression algorithm -- development files
ii zstd 1.3.1+dfsg-1~ubuntu0.16.04.1 amd64 fast lossless compression algorithm -- CLI tool

* Conversely, installing the unversioned dev package removes the versioned one:
ubuntu@xenial-zstd:~$ sudo apt install libzstd-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  libzstd1-dev
The following NEW packages will be installed:
  libzstd-dev

* Verify that a simple apt upgrade (not dist-upgrade) upgrades zstd and installs the new libzstd1:
Starting with:
ubuntu@xenial-zstd:~$ dpkg -l|grep zstd
ii libzstd-dev 0.5.1-1 amd64 fast lossless compression algorithm -- development files
ii libzstd0 0.5.1-1 amd64 fast lossless compression algorithm
ii zstd 0.5.1-1 amd64 fast lossless compression algorithm -- CLI tool

ubuntu@xenial-zstd:~$ sudo apt upgrade
Reading package lists... Done
Build...

Read more...

description: updated
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

a) compression tests - artful - PASS
starting with the original artful packages:
ubuntu@artful-zstd:~$ dpkg -l|grep zstd
ii libzstd1 1.3.1+dfsg-1 amd64 fast lossless compression algorithm
ii zstd 1.3.1+dfsg-1 amd64 fast lossless compression algorithm -- CLI tool

ubuntu@artful-zstd:~$ zstd /usr/bin/snap -o snap.1.3.1.zst
/usr/bin/snap : 31.03% (11318760 => 3512707 bytes, snap.1.3.1.zst)

gives:
-rwxr-xr-x 1 ubuntu ubuntu 3.4M Nov 30 16:42 snap.1.3.1.zst

Now upgrading to the package in proposed:
The following additional packages will be installed:
  libzstd1
The following packages will be upgraded:
  libzstd1 zstd
...
Get:1 http://br.archive.ubuntu.com/ubuntu artful-proposed/universe amd64 zstd amd64 1.3.1+dfsg-1ubuntu0.1 [238 kB]
Get:2 http://br.archive.ubuntu.com/ubuntu artful-proposed/universe amd64 libzstd1 amd64 1.3.1+dfsg-1ubuntu0.1 [152 kB]
...

Compressing with the new package:
ubuntu@artful-zstd:~$ zstd /usr/bin/snap -o snap.1.3.1-updated.zst
/usr/bin/snap : 31.03% (11318760 => 3512707 bytes, snap.1.3.1-updated.zst)

Decompressing the previous one:
ubuntu@artful-zstd:~$ zstd -d snap.1.3.1.zst -o snap.1.3.1
snap.1.3.1.zst : 11318760 bytes

Decompressing the updated one:
ubuntu@artful-zstd:~$ zstd -d snap.1.3.1-updated.zst -o snap.1.3.1-updated
snap.1.3.1-updated.zst: 11318760 bytes

MD5 of everything:
ubuntu@artful-zstd:~$ md5sum /usr/bin/snap snap.1.3.1 snap.1.3.1-updated
0b65998bc775d444b2ee7d00cd18634a /usr/bin/snap
0b65998bc775d444b2ee7d00cd18634a snap.1.3.1
0b65998bc775d444b2ee7d00cd18634a snap.1.3.1-updated

Downgrading back to the original artful version:
ubuntu@artful-zstd:~$ sudo apt install zstd=1.3.1+dfsg-1 libzstd1=1.3.1+dfsg-1
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be DOWNGRADED:
  libzstd1 zstd

Decompressing and another md5 snapshot:
ubuntu@artful-zstd:~$ zstd -d snap.1.3.1.zst -o snap.1.3.1-new
snap.1.3.1.zst : 11318760 bytes
ubuntu@artful-zstd:~$ md5sum snap.1.3.1-new /usr/bin/snap
0b65998bc775d444b2ee7d00cd18634a snap.1.3.1-new
0b65998bc775d444b2ee7d00cd18634a /usr/bin/snap

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

b) artful package upgrade tests - PASS
Starting with the original artful packages:
dpubuntu@artful-zstd:~$ dpkg -l|grep zstd
ii libzstd-dev 1.3.1+dfsg-1 amd64 fast lossless compression algorithm -- development files
ii libzstd1 1.3.1+dfsg-1 amd64 fast lossless compression algorithm
ii zstd 1.3.1+dfsg-1 amd64 fast lossless compression algorithm -- CLI tool

dist-upgrade pulls in no new packages and just upgrades the ones I have already:
The following packages will be upgraded:
  libzstd-dev libzstd1 zstd
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

resulting in:
ubuntu@artful-zstd:~$ dpkg -l|grep zstd
ii libzstd-dev 1.3.1+dfsg-1ubuntu0.1 amd64 fast lossless compression algorithm -- development files
ii libzstd1 1.3.1+dfsg-1ubuntu0.1 amd64 fast lossless compression algorithm
ii zstd 1.3.1+dfsg-1ubuntu0.1 amd64 fast lossless compression algorithm -- CLI tool

installing the libzstd1-dev transitional package:
ubuntu@artful-zstd:~$ sudo apt install libzstd1-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  libzstd1-dev
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.

removing libzstd-dev removes the transitional package:
ubuntu@artful-zstd:~$ sudo apt purge libzstd-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  libzstd-dev* libzstd1-dev*
0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.

installing the transitional package pulls in the unversioned dev package:
ubuntu@artful-zstd:~$ sudo apt install libzstd1-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libzstd-dev
The following NEW packages will be installed:
  libzstd-dev libzstd1-dev

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

RELEASE UPGRADE FROM XENIAL TO ARTFUL - with libzstd-dev 0.5.1, others at 1.3.1

PASS

Starting with the original xenial packages:
ubuntu@xenial2artful:~$ dpkg -l|grep zstd
ii libzstd-dev 0.5.1-1 amd64 fast lossless compression algorithm -- development files
ii libzstd0 0.5.1-1 amd64 fast lossless compression algorithm
ii zstd 0.5.1-1 amd64 fast lossless compression algorithm -- CLI tool

dist-upgrading to proposed:
ubuntu@xenial2artful:~$ dpkg -l|grep zstd
ii libzstd-dev 0.5.1-1 amd64 fast lossless compression algorithm -- development files
ii libzstd0 0.5.1-1 amd64 fast lossless compression algorithm
ii libzstd1 1.3.1+dfsg-1~ubuntu0.16.04.1 amd64 fast lossless compression algorithm
ii zstd 1.3.1+dfsg-1~ubuntu0.16.04.1 amd64 fast lossless compression algorithm -- CLI tool

Configuring update-manager to allow third-party repositories (not sure if -proposed falls into that category):
# echo -e "[Sources]\nAllowThirdParty=yes" > /etc/update-manager/release-upgrades.d/allow.cfg

Changing prompt to "normal" in /etc/update-manager/release-upgrades

do-release-upgrade to artful

In details about upgraded packages, we see these mentioned:
libzstd-dev libzstd1 zstd

At the end, libzstd0 is removed:
Remove (was auto installed) libpython3.5 libzstd0

Final result in the now artful system:
root@xenial2artful:~# dpkg -l|grep zstd
ii libzstd-dev 1.3.1+dfsg-1ubuntu0.1 amd64 fast lossless compression algorithm -- development files
ii libzstd1 1.3.1+dfsg-1ubuntu0.1 amd64 fast lossless compression algorithm
ii zstd 1.3.1+dfsg-1ubuntu0.1 amd64 fast lossless compression algorithm -- CLI tool

Showing that proposed was used:
root@xenial2artful:~# apt-cache policy zstd
zstd:
  Installed: 1.3.1+dfsg-1ubuntu0.1
  Candidate: 1.3.1+dfsg-1ubuntu0.1
  Version table:
 *** 1.3.1+dfsg-1ubuntu0.1 500
        500 http://br.archive.ubuntu.com/ubuntu artful-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     1.3.1+dfsg-1 500
        500 http://br.archive.ubuntu.com/ubuntu artful/universe amd64 Packages

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

RELEASE UPGRADE FROM XENIAL TO ARTFUL
- starting with the 1.3.1 dev package in xenial

PASS

root@xenial2artful:~# dpkg -l|grep zstd
ii libzstd0 0.5.1-1 amd64 fast lossless compression algorithm
ii libzstd1 1.3.1+dfsg-1~ubuntu0.16.04.1 amd64 fast lossless compression algorithm
ii libzstd1-dev 1.3.1+dfsg-1~ubuntu0.16.04.1 amd64 fast lossless compression algorithm -- development files
ii zstd 1.3.1+dfsg-1~ubuntu0.16.04.1 amd64 fast lossless compression algorithm -- CLI tool

Details of upgrade:
Install: libzstd-dev
Upgrade: libzstd1 libzstd1-dev zstd

At the end:
Remove (was auto installed) libpython3.5 libzstd0 libzstd1-dev

end result (transitional package was removed):
root@xenial2artful:~# dpkg -l|grep zstd
ii libzstd-dev 1.3.1+dfsg-1ubuntu0.1 amd64 fast lossless compression algorithm -- development files
ii libzstd1 1.3.1+dfsg-1ubuntu0.1 amd64 fast lossless compression algorithm
ii zstd 1.3.1+dfsg-1ubuntu0.1 amd64 fast lossless compression algorithm -- CLI tool

Confirming proposed was used in the release upgrade:
root@xenial2artful:~# apt-cache policy zstd
zstd:
  Installed: 1.3.1+dfsg-1ubuntu0.1
  Candidate: 1.3.1+dfsg-1ubuntu0.1
  Version table:
 *** 1.3.1+dfsg-1ubuntu0.1 500
        500 http://br.archive.ubuntu.com/ubuntu artful-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     1.3.1+dfsg-1 500
        500 http://br.archive.ubuntu.com/ubuntu artful/universe amd64 Packages

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

RELEASE UPGRADE FROM ARTFUL TO BIONIC - with all zstd packages installed

PASS

root@artful2bionic:~# dpkg -l|grep zstd
ii libzstd-dev 1.3.1+dfsg-1ubuntu0.1 amd64 fast lossless compression algorithm -- development files
ii libzstd1 1.3.1+dfsg-1ubuntu0.1 amd64 fast lossless compression algorithm
ii libzstd1-dev 1.3.1+dfsg-1ubuntu0.1 amd64 transitional package for libzstd-dev
ii zstd 1.3.1+dfsg-1ubuntu0.1 amd64 fast lossless compression algorithm -- CLI tool

do-release-upgrade -d details:
upgrade: libzstd-dev libzstd1 libzstd1-dev zstd

At the end, nothing about zstd
Final state of zstd packages in bionic:
root@artful2bionic:~# dpkg -l|grep zstd
ii libzstd-dev 1.3.3+dfsg-1ubuntu1 amd64 fast lossless compression algorithm -- development files
ii libzstd1 1.3.3+dfsg-1ubuntu1 amd64 fast lossless compression algorithm
ii libzstd1-dev 1.3.3+dfsg-1ubuntu1 amd64 transitional package for libzstd-dev
ii zstd 1.3.3+dfsg-1ubuntu1 amd64 fast lossless compression algorithm -- CLI tool

And they come from bionic universe:
root@artful2bionic:~# apt-cache policy zstd
zstd:
  Installed: 1.3.3+dfsg-1ubuntu1
  Candidate: 1.3.3+dfsg-1ubuntu1
  Version table:
 *** 1.3.3+dfsg-1ubuntu1 500
        500 http://br.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
        100 /var/lib/dpkg/status

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

RELEASE UPGRADE FROM ARTFUL TO BIONIC - without the transitional package installed:

PASS

root@artful2bionic:~# dpkg -l|grep zstd
ii libzstd-dev 1.3.1+dfsg-1ubuntu0.1 amd64 fast lossless compression algorithm -- development files
ii libzstd1 1.3.1+dfsg-1ubuntu0.1 amd64 fast lossless compression algorithm
ii zstd 1.3.1+dfsg-1ubuntu0.1 amd64 fast lossless compression algorithm -- CLI tool

do-release-upgrade -d details:
Upgrade: libzstd-dev libzstd1 zstd

At the end, nothing about zstd

Final state in bionic:
root@artful2bionic:~# dpkg -l|grep zstd
ii libzstd-dev 1.3.3+dfsg-1ubuntu1 amd64 fast lossless compression algorithm -- development files
ii libzstd1 1.3.3+dfsg-1ubuntu1 amd64 fast lossless compression algorithm
ii zstd 1.3.3+dfsg-1ubuntu1 amd64 fast lossless compression algorithm -- CLI tool

root@artful2bionic:~# apt-cache policy zstd
zstd:
  Installed: 1.3.3+dfsg-1ubuntu1
  Candidate: 1.3.3+dfsg-1ubuntu1
  Version table:
 *** 1.3.3+dfsg-1ubuntu1 500
        500 http://br.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
        100 /var/lib/dpkg/status

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I believe this completes all testing from the SRU test case. Adjusting tags to reflect that state.

tags: added: verification-done-artful verification-done-xenial
removed: verification-needed-artful verification-needed-xenial
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for libzstd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libzstd - 1.3.1+dfsg-1ubuntu0.1

---------------
libzstd (1.3.1+dfsg-1ubuntu0.1) artful; urgency=medium

  * d/control: New transitional package libzstd1-dev to facilitate the release
    upgrade from xenial (LP: #1717040)
  * d/rules: also run the clean Makefile target in the pzstd directory.

libzstd (1.3.1+dfsg-1) unstable; urgency=medium

  * Team upload
  * New upstream version
  * Exclude examples from upstream tarball
    Closes: #869581
  * debhelper 10
  * cme fix dpkg-control
  * Standards-Version: 4.1.0 (no changes needed)
  * Fix copyright

libzstd (1.2.0-1) unstable; urgency=medium

  * New upstream release (Closes: #863159).

libzstd (1.1.2-1) unstable; urgency=medium

  * Team upload.
  * New upstream version 1.1.2

libzstd (1.1.1-1) unstable; urgency=medium

  * New upstream version 1.1.1 (Closes: #844248)
  * Whitespace fix to d/control
  * Ensure hardening flags pass through to compiler by appending to CPPFLAGS

libzstd (1.1.0-1) unstable; urgency=medium

  * New upstream version 1.1.0 (Closes: #839960)
  * Change uploader email
  * Build new pzstd binary, including manpage
  * Fix miscellaneous issues with d/rules and d/*.install
  * Move docs from libzstd-dev to the zstd binary

libzstd (1.0.0-1) unstable; urgency=medium

  * Imported Upstream version 1.0.0 (Closes: #836574)
  * Bump library package name to libzstd1
  * Changed project URLs, copyrights after move to facebook.
  * d/rules: hardening=+all

libzstd (0.8.0-1) unstable; urgency=medium

  [ Kevin Murray ]
  * New upstream version (Closes: #834114)

  [ Andreas Tille ]
  * hardening=+bindnow

libzstd (0.5.1-1) unstable; urgency=medium

  [ Kevin Murray ]
  * New upstream version (fixes a FTBFS)
  * Package pkgconfig file (Closes: #813854)
  * Fix path to tagged archives in d/watch
  * Add vcs URLs
  * Bump to standards version 3.9.7

  [ Mattia Rizzolo ]
  * debian/rules: Remove a lot of uneeded comments and lines

libzstd (0.4.7-1) unstable; urgency=low

  * New upstream version
  * Remove build date encoding to enable reproducible build

libzstd (0.4.5-1) unstable; urgency=low

  * Initial release (Closes: #806767)

 -- Andreas Hasenack <email address hidden> Tue, 16 Jan 2018 19:28:18 +0000

Changed in libzstd (Ubuntu Artful):
status: Fix Committed → Fix Released
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

andreas@nsnx:~$ rmadison -s xenial libzstd
 libzstd | 0.5.1-1 | xenial/universe | source
andreas@nsnx:~$ rmadison -s xenial-updates libzstd
andreas@nsnx:~$ rmadison -s xenial-proposed libzstd
 libzstd | 1.3.1+dfsg-1~ubuntu0.16.04.1 | xenial-proposed/universe | source

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I'm told xenial will be handled after 16.04.4 is released. The task is still open, and listed in https://people.canonical.com/~ubuntu-archive/pending-sru.html, and an SRU team member shall pick it up then following the normal SRU workflow.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libzstd - 1.3.1+dfsg-1~ubuntu0.16.04.1

---------------
libzstd (1.3.1+dfsg-1~ubuntu0.16.04.1) xenial; urgency=medium

  * Backport to xenial (LP: #1717040):
    - d/compat, d/control: use debhelper 9 since version 10 is only in
      xenial-backports
    - d/control, d/rules: new libzstd1-dev package. Added a Provides for
      libzstd-dev to keep d-shlibmove happy.
  * d/rules: also run the clean Makefile target in the pzstd directory.

libzstd (1.3.1+dfsg-1) unstable; urgency=medium

  * Team upload
  * New upstream version
  * Exclude examples from upstream tarball
    Closes: #869581
  * debhelper 10
  * cme fix dpkg-control
  * Standards-Version: 4.1.0 (no changes needed)
  * Fix copyright

libzstd (1.2.0-1) unstable; urgency=medium

  * New upstream release (Closes: #863159).

libzstd (1.1.2-1) unstable; urgency=medium

  * Team upload.
  * New upstream version 1.1.2

libzstd (1.1.1-1) unstable; urgency=medium

  * New upstream version 1.1.1 (Closes: #844248)
  * Whitespace fix to d/control
  * Ensure hardening flags pass through to compiler by appending to CPPFLAGS

libzstd (1.1.0-1) unstable; urgency=medium

  * New upstream version 1.1.0 (Closes: #839960)
  * Change uploader email
  * Build new pzstd binary, including manpage
  * Fix miscellaneous issues with d/rules and d/*.install
  * Move docs from libzstd-dev to the zstd binary

libzstd (1.0.0-1) unstable; urgency=medium

  * Imported Upstream version 1.0.0 (Closes: #836574)
  * Bump library package name to libzstd1
  * Changed project URLs, copyrights after move to facebook.
  * d/rules: hardening=+all

libzstd (0.8.0-1) unstable; urgency=medium

  [ Kevin Murray ]
  * New upstream version (Closes: #834114)

  [ Andreas Tille ]
  * hardening=+bindnow

libzstd (0.5.1-1) unstable; urgency=medium

  [ Kevin Murray ]
  * New upstream version (fixes a FTBFS)
  * Package pkgconfig file (Closes: #813854)
  * Fix path to tagged archives in d/watch
  * Add vcs URLs
  * Bump to standards version 3.9.7

  [ Mattia Rizzolo ]
  * debian/rules: Remove a lot of uneeded comments and lines

libzstd (0.4.7-1) unstable; urgency=low

  * New upstream version
  * Remove build date encoding to enable reproducible build

libzstd (0.4.5-1) unstable; urgency=low

  * Initial release (Closes: #806767)

 -- Andreas Hasenack <email address hidden> Tue, 16 Jan 2018 21:50:37 +0000

Changed in libzstd (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Yann Collet (cyan4973) wrote :

Update confirmed !
Thanks !

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.