package chrony 3.2-4ubuntu4.2 failed to install/upgrade: installed chrony package post-removal script subprocess returned error exit status 1

Upgrade from bionic to focal broke for this system which has chrony installed.

After do-release-upgrade exited (warning that the system was in an indeterminate state) I was unable to fix the system initially. It seemed that systemd-timesyncd was somehow interacting badly with chrony:

$ sudo apt --fix-broken install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  libboost-filesystem1.65.1 libboost-iostreams1.65.1 libboost-system1.65.1 libcwidget3v5 libip4tc2 libnspr4 libnss3 python3.6 python3.6-minimal
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  libboost-iostreams1.71.0 libcwidget4 libxapian30
Suggested packages:
  libcwidget-dev xapian-tools
The following packages will be REMOVED:
  chrony
The following NEW packages will be installed:
  libboost-iostreams1.71.0 libcwidget4
The following packages will be upgraded:
  libxapian30
1 upgraded, 2 newly installed, 1 to remove and 439 not upgraded.
2 not fully installed or removed.
Need to get 0 B/1204 kB of archives.
After this operation, 3022 kB of additional disk space will be used.
Do you want to continue? [Y/n]
(Reading database ... 171920 files and directories currently installed.)
Removing chrony (3.2-4ubuntu4.2) ...
Job for systemd-timesyncd.service failed because the control process exited with error code.
See "systemctl status systemd-timesyncd.service" and "journalctl -xe" for details.
dpkg: error processing package chrony (--remove):
 installed chrony package post-removal script subprocess returned error exit status 1
Errors were encountered while processing:
 chrony
E: Sub-process /usr/bin/dpkg returned an error code (1)

I turned off the systemd-timesync ntp and that then allowed chrony to be removed (not quite what I wanted but that is another story):

mark@bigben:/var/log/chrony$ sudo timedatectl status
                      Local time: Sat 2020-04-11 10:55:31 UTC
                  Universal time: Sat 2020-04-11 10:55:31 UTC
                        RTC time: Sat 2020-04-11 10:55:31
                       Time zone: Etc/UTC (UTC, +0000)
       System clock synchronized: yes
systemd-timesyncd.service active: yes
                 RTC in local TZ: no
mark@bigben:/var/log/chrony$ sudo timedatectl set-ntp false
mark@bigben:/var/log/chrony$ sudo timedatectl status
                      Local time: Sat 2020-04-11 10:55:50 UTC
                  Universal time: Sat 2020-04-11 10:55:50 UTC
                        RTC time: Sat 2020-04-11 10:55:50
                       Time zone: Etc/UTC (UTC, +0000)
       System clock synchronized: yes
systemd-timesyncd.service active: no
                 RTC in local TZ: no

$ sudo apt --fix-broken install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  libboost-filesystem1.65.1 libboost-iostreams1.65.1 libboost-system1.65.1 libcwidget3v5 libip4tc2 libnspr4 libnss3 python3.6 python3.6...

Hello Mark,

Debian chrony maintainer here.

So, this error occurs when attempting to rehabilitate systemd-timesyncd when
uninstalling/purging chrony (see restore_timesyncd() in d/postrm). This used
to be possible when systemd-timesyncd and chrony were co-installable but since
systemd 245.4-2{,ubuntu1}, this is no longer the case because systemd-timesyncd
is now a standalone package conflicting with chrony, hence the exit status 1.

To fix this, the restore_timesyncd() function found in the chrony
3.2-4ubuntu4.2 postrm script should be updated as follows:

restore_timesyncd() {
    # on next reboot it would start, but that would leave time
    # unsynchronized until then. So as the Conflicts in the service file
    # kill systemd-timesyncd re-establish it if it is enabled
    if [ -d /run/systemd/system ]; then
        deb-systemd-invoke start systemd-timesyncd || true
    fi

From what I can see, the chrony version found in Disco Dingo is also be affected by this issue.

Cheers,
Vincent

Hi Mark,
thanks for the report.

To give everyone arriving here some background:
I was recently fixing a bunch of chrony issues in bug 1867036 which also let me identify bug 1870144. After discussions the latter was supposed to be fixed in a systemd upload for bug 1849156.
It seems that systemd upload now (at the start of the Easter weekend to be more specific) has landed and is causing this :-/

For the root cause analysis I agree to Vincent (thanks for chiming in, glad to see your LP account now works).

We already changed that line which potentially was an issue in Eoan:

$ git show pkg/ubuntu/eoan-devel:debian/postrm | grep -A 8 '^restore_timesyncd' | grep "start "
        deb-systemd-invoke start systemd-timesyncd || true

$ git show pkg/ubuntu/disco-devel:debian/postrm | grep -A 8 '^restore_timesyncd' | grep "start "
        deb-systemd-invoke start systemd-timesyncd

$ git show pkg/ubuntu/bionic-devel:debian/postrm | grep -A 8 '^restore_timesyncd' | grep "start "
        systemctl start systemd-timesyncd

So Focal and Eoan are already ok. Upgraders "through" Eoan won't see this issue.
But due to the latest systemd upload upgraders from Bionic will see it.
There is nothing to fix "in Focal" anymore as we already have it there (we might silence the warning, but that is far less important than the broken upgrade).

Instead what we will need to SRU the same fix to Bionic to be there before people will upgrade to Focal.

This will fix that the upgrade "fails" as chrony will become more tolerant to the systemd change.
do-release-upgrade will still remove chrony on upgrade.

I filed an extra bug for that to keep the discussions separate.
=> see bug 1872902

Tested against suggested fixes in PPA:
=> https://launchpad.net/~paelzer/+archive/ubuntu/bug-1872183-chrony-starts-timesyncd/

Working with these changes in place (in bionic). I updated the SRU Template about the Test details.

MPs:
- B SRU https://code.launchpad.net/~paelzer/ubuntu/+source/chrony/+git/chrony/+merge/382270
- F drop of no more needed https://code.launchpad.net/~paelzer/ubuntu/+source/chrony/+git/chrony/+merge/382271

MPs got a review, uploading to Focal and to Bionic-unapproved

I see the fix in Focal unapproved. I think that's sufficient to accept the Bionic SRU into bionic-proposed. Thanks!

Hello Mark, or anyone else affected,

Accepted chrony into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/chrony/3.2-4ubuntu4.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

For Focal we actually we also need to fix the tests.
I found:
time-sources-from-dhcp-servers FAIL stderr: Failed to try-restart systemd-timesyncd.service: Unit systemd-timesyncd.service is masked.

This should have stopped systemd from migrating TBH, but anyway lets fix it.

All autopkgtests for the newly accepted chrony (3.2-4ubuntu4.3) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

chrony/3.2-4ubuntu4.3 (armhf, s390x, amd64, i386, ppc64el, arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#chrony

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

I ran two Bionic VMs one as-is and one with proposed enabled to get the new chrony.
One upgrade starts at 3.2-4ubuntu4.2 the other one at 3.2-4ubuntu4.3 from proposed.

Both run `do-release-upgrade -d`
Both run into bug 1872902
  Calculating the changes
  MarkInstall systemd-timesyncd:amd64 < none -> 245.4-4ubuntu1 @un uN Ib > FU=1
   Removing: chrony
    MarkDelete chrony:amd64 < 3.2-4ubuntu4.3 -> 3.5-6ubuntu3 @ii umU Ib > FU=0

Unfixed one ran into:
Removing chrony (3.2-4ubuntu4.2) ...
Job for systemd-timesyncd.service failed because the control process exited with error code.
See "systemctl status systemd-timesyncd.service" and "journalctl -xe" for details.
dpkg: error processing package chrony (--remove):
 installed chrony package post-removal script subprocess returned error exit status 1
ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/chrony.0.crash'
Exception during pm.DoInstall(): E:Sub-process /usr/bin/dpkg returned an error code (1)

The fixed version worked though the upgrade just fine.

Marking verified

P.S. Rbasak mentioned that in this case we might want to shorten the aging period for the SRU to get more upgrade-tests of the new code on prior to Focal release. @SRU team do you think that is reasonable and if so what would be a good time then?

Since this SRU is for the purpose of fixing the upgrade path from Bionic to Focal and Focal's release is imminent, and taking into account the trivial nature of the diff and that it only applies to chrony's postrm, I think it makes sense to skip the usual ageing period, since otherwise we'll miss out on getting testing for this from all the people testing the upgrade from Bionic to Focal currently.

The autopkgtests fail with:

client.c:644:5: error: conflicting types for ‘gettimeofday’
 int gettimeofday(struct timeval *tv,
     ^~~~~~~~~~~~
In file included from client.c:20:0:
/usr/include/powerpc64le-linux-gnu/sys/time.h:68:12: note: previous declaration of ‘gettimeofday’ was here
 extern int gettimeofday (struct timeval *__restrict __tv,
            ^~~~~~~~~~~~
<builtin>: recipe for target 'client.o' failed
make: Leaving directory '/tmp/autopkgtest.gfnV6L/build.CJP/src/test/simulation/clknetsim'
make: *** [client.o] Error 1

That is a known issue in newer releases and fixed there, it is a kernel/kernel-header change that broke users of gettimeofday - the tests still run at 4.15 kernel, but maybe some changes got backported. I'll work on a fix for that tomorrow which then should unlock the SRU.

Just wanted to say thank you Vincent, Christian, and it's really great
to see you working together this way!

Mark

As expected I was able to reproduce locally in autopkgtest.

The bionic kernel headers didn't change actually (which is good!).
The test creepign into Bionic fail is actually the inverse:
1. gettimeofday changed upstream
2. upstream clknetsim adapted
3. since Eoan we download defined versions matching the release
4. the bionic test still was the "uncontrolled" cloning of latest master
   which now no more matches bionic kernel headers

Focal uses clknetsim_ver=79ffe44, but that is for the new kernel behavior.
Eoan used clknetsim_ver=156b8e4 which works well in Bionic.

Furthermore between Eoan and Focal the test has gotten a bigger rewrite and some fixes.

In the SRU-spirit lets keep the changes minimal by:
- picking the version from Eoan
- add just one small fixups
  (ignoring stderr on make since gcc warnings trigger false positives).

That should do it, I'll be back here once I have some proven bionic test results.

The new build now has all tests fixed and going well on all architectures.
Seen on https://bileto.ubuntu.com/excuses/4021/bionic.html

I'll upload this to bionic-unapproved as 3.2-4ubuntu4.4 for evaluation by the SRU Team.

P.S. I found this also started to block some other SRUs e.g. kmod, so by that we will unblock more than just chrony itself.

This bug was fixed in the package chrony - 3.5-6ubuntu5

---------------
chrony (3.5-6ubuntu5) focal; urgency=medium

  * d/t/control: harden time-sources-from-dhcp-servers test for systemd change
    (LP: #1873031)

chrony (3.5-6ubuntu4) focal; urgency=medium

  * debian/postrm:
    - Stop starting systemd-timesyncd in postrm. This is no longer relevant
      since systemd-timesyncd is a standalone package declaring
      Conflicts/Replaces/Provides: time-daemon. (Closes 955773, LP: #1872183)

 -- Christian Ehrhardt <email address hidden> Wed, 15 Apr 2020 18:23:10 +0200

Hello Mark, or anyone else affected,

Accepted chrony into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/chrony/3.2-4ubuntu4.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted chrony (3.2-4ubuntu4.4) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

chrony/3.2-4ubuntu4.4 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#chrony

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Christian, from what I remember chrony 3.2 needs [1] to render the “smoothtime” test more reliable on some architectures. Could you please give it a try?

[1] https://git.tuxfamily.org/chrony/chrony.git/commit/?id=a4bd7f18000214ea4967a6a4d84a4a3e48d88a86

Yeah - thanks Vincent I think I also remember that, but the tests already worked in when I prepped it (running on the same infra) in comment #17. If it continues to be unreliable in Bionic I can add that fix as well.

/me stares at ppc64 for working when I pre-test and failing on the final upload

I need to retrigger the chrony tests as triggered by the kmod SRU anyway. Overall that will tomorrow morning give me some insight at what rate of success/fail we might look at.

Summing up the rate we have 1/14 fails, all green to release the SRU for now.
Much better than the 100% fail rate we had before since the gettimeofday header change :-)

If the rate goes up again we can consider adding the fix, but for now I'd think unbreaking the upgrade to focal is more urgent and we should go on releasing this change.

Re-tagging as tested after a quick re-verify

This bug was fixed in the package chrony - 3.2-4ubuntu4.4

---------------
chrony (3.2-4ubuntu4.4) bionic; urgency=medium

  * fix autopkgtest to continue working in Bionic
    - take Eoans version of d/t/upstream-simulation-test-suite
    - use a defined version of clknetsim
    - ignore compiler warnings when running make
    - d/p/update_processing_of_packet_log.patch: Two new fields have been
      added to the packet log, which broke some of the simulation tests.

chrony (3.2-4ubuntu4.3) bionic; urgency=medium

  * d/postrm: gracefully ignore issues on starting systemd-timesyncd
    (LP: #1872183)

 -- Christian Ehrhardt <email address hidden> Thu, 16 Apr 2020 07:47:40 +0200

The verification of the Stable Release Update for chrony has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.