boot-and-services in tests-in-lxd autopkgtest fails in groovy on s390x

Thanks for this bug Rbalint.
That the new glibc/cryptsetup worked is odd as nothing on the issue indicates they are related. There was nothing in the queue - for the sake of some test data I have restarted the packages currently blocked on this with all-proposed which will pull in e.g. gblibc/cryptsetup.

Let us check in some hours how that worked out ...

@Rbalint - hi the two most recent tests confirm that it still seems broken.
Maybe the good run was a one-off lucky run ... :-/

I'd ask you to look into that a bit deeper as it will again hold back plenty of things in groovy soon.

Also please do note that while almost all of the recent tests failed - almost all of the packages already migrated. Why is that - there is no test hint in place.
For example gdm3/3.37.90-1ubuntu1 and iptables/1.8.4-3ubuntu3 (but more than just those, policykit-1, dpdk, util-linux, ...) NEVER had a valid test [1], but they migrated to groovy-release.
That seems even more wrong, any idea why/how that happened?

There might be a follow up issue of people using tools that "retry everything that had no good run" but since somehow things magically migrated the recent long retry-lists mostly contain packages that are already migrated.

For the sake of seeing if it might be "blunt retries" and since the s390x test queue is empty I'm retrying it multiple times (hopefully those won't be coalesced).
The last not-yet weird-migrated entry is apparmor.

[1]: https://autopkgtest.ubuntu.com/packages/s/systemd/groovy/s390x

Old hints we only had:

# failing due to multiple reasons LP: #1892358
force-badtest systemd/246-2ubuntu1
# failing due to livecd-rootfs creating broken LXD images LP: #1877078
force-reset-test systemd/246.2-1ubuntu1

But was then removed in
revno: 4988
committer: Steve Langasek <email address hidden>
branch nick: hints-ubuntu
timestamp: Fri 2020-09-11 13:38:38 -0700
message:
  drop unused hints; fix typo in package name

Maybe that rule let the packages pass that I wondered about, but it is unclear what made 246.4-1ubuntu1 itself pass as there was no good run of that either ?!?.

Overall for now it seems to me that we'd have to add:
force-badtest systemd/246.4-1ubuntu1/s390x

The retries were reproducible failing as well.

[09:20] <cpaelzer> rbalint: btw the test on s390x really seems still broken
[09:20] <cpaelzer> we need either a fix or a test hint
[09:21] --> falcojr (~falcojr@184.55.64.152) has joined this channel.
[09:24] <rbalint> cpaelzer, i'm ok with the hint and i've retriggered the tests with cryptsetup and glibc to see if they still help
[09:25] <cpaelzer> ok rbalint, given you are the maintainer and you agree I think I'll prep a hint and once you ack there (for an audit trail) we can push it
[09:27] <rbalint> cpaelzer, thanks, i'm also working on the next systemd point release but it shows regressions so it will take a few days

Here it is:
https://code.launchpad.net/~paelzer/britney/hints-ubuntu-groovy-systemd-s390x-fails/+merge/390809

For now the hint is in groovy, but the next systemd upload needs to resolve it one (fix) or the other (mark this one as flaky) way.

OK, the next upload passes on s390x, so this will likely be resolved.

I can't start s390x VMs to reproduce it for some reason so I'm just disabling this test in the next upload.

This bug was fixed in the package systemd - 246.6-1ubuntu1

---------------
systemd (246.6-1ubuntu1) groovy; urgency=medium

  [ Dan Streetman ]
  * Fix resolved.conf Cache= default to match default (LP: #1895418)
    Files:
    - debian/patches/lp1895418.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=dd4d93d94ebe2cf416f6b5a5eb59a16432cbc47b

  [ Balint Reczey ]
  * Bugfix-only upload
  * Merge from Debian unstable
    - core: propagate unit start limit hit state to triggering path unit (LP: #1891657)
  * Skip test_rsyslog in s390x containers.
    The test is failing almost all the time. (LP: #1895576)
    File: debian/tests/boot-and-services
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ff8d286de52daa8b0856213ae52e66b81474fb57
  * debian/tests/tests-in-lxd: Don't create the lxd test image twice
    File: debian/tests/tests-in-lxd
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=e858e7c41b84bc4c34e8af79d6e4b6114fe23952

systemd (246.6-1) unstable; urgency=medium

  * New upstream version 246.6
  * Rebase patches

 -- Balint Reczey <email address hidden> Thu, 24 Sep 2020 21:27:11 +0200