AMD Ryzen 3000 series fails to boot

Bug #1835809 reported by Jeremy Soller
50
This bug affects 6 people
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Fix Released
High
Balint Reczey
Disco
Fix Released
Undecided
Balint Reczey
Eoan
Fix Released
High
Balint Reczey

Bug Description

[Impact]

 * Systems with AMD Ryzen 3000 series CPUs don't boot.

[Test Case]

 * Boot with fixed systemd on an AMD Ryzen 3000 series system.

[Regression Potential]

 * The fix itself is very small, it ignores known to be faulty random values returned by the rdrand instruction and use a different random source. Those values can still be returned by a properly working rdrand implementation in 2 in 2^32 cases on 32 bit arches and in 2 in 2^64 cases on 64 bit arches, but the fallback to the other random source ensures that in those rare occasions a random number can be generated.

[Original Bug Text]

On the new AMD Ryzen 3000 series CPUs, there is an issue with systemd preventing the boot process from completing. This issue does not affect the older systemd version in 18.04, but affects the 19.04 version.

Here is a screenshot showing what happens:
https://www.phoronix.net/image.php?id=ryzen-3700x-3900x-linux&image=amd_zen2_14_show

I am currently testing a patch to systemd, derived from this pull request:
https://github.com/systemd/systemd/pull/12536

This is a high severity issue, as I do not believe there is no potential workaround without either a firmware update or an ISO respin.

I have attached a rebase of the potential patch on the current 19.04 version of systemd for reference. I will provide more details after testing.

Revision history for this message
Jeremy Soller (jackpot51) wrote :
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "Rebase of https://github.com/systemd/systemd/pull/12536" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Revision history for this message
Jeremy Soller (jackpot51) wrote :
Revision history for this message
Jeremy Soller (jackpot51) wrote :

I have tested the patch posted, and it produces a bootable system with 19.04. We will be releasing the patched version of systemd in the following PPAs:

Pop!_OS - https://launchpad.net/~system76/+archive/ubuntu/pop
Ubuntu support for System76 computers - https://launchpad.net/~system76-dev/+archive/ubuntu/stable

Please let me know what needs to happen to get this patch into Ubuntu 19.04

Revision history for this message
Sebastien Bacher (seb128) wrote :

Thanks for your bug report & patch!

I'm subscribing ubuntu-sponsors which is what is needed for the bug to get on the sponsoring queue (http://reqorts.qa.ubuntu.com/reports/sponsoring/index.html)

I'm also tagging the bug rls-dd-incoming and rls-ee-incoming so it gets on the list of targetted to review which should help visibility

Changed in systemd (Ubuntu):
importance: Undecided → High
tags: added: rls-dd-incoming rls-ee-incoming
Revision history for this message
Éric Hoffman (ehoffman-videotron) wrote :

Good findings!

Thank you so much for testing the rdrand theory.

I guess a read fix would require a new ISO, which could easily be made as a dot release (19.04.1).

Regards,
Eric

Revision history for this message
Jeremy Soller (jackpot51) wrote :

Sebastien, thanks for getting eyes on it!

Éric, these systems will not boot the 19.04 ISO without either new firmware fixing the rdrand instruction, or a new ISO updating systemd with this patch.

Revision history for this message
Adam Conrad (adconrad) wrote :

FWIW, I think not booting a 19.04 ISO is probably fine, and it's not worth spinning a point release for this. "I can't install a 9mo release on this hardware" is unforunate, but not world-ending.

However, we should definitely SRU this for people who are likely to take their EXISTING 19.04 system with a Ryzen 1xxx/2xxx and drop in a 3xxx and find it doesn't boot anymore. Also, obviously, we should make sure it's fixed for 19.10.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in systemd (Ubuntu):
status: New → Confirmed
Balint Reczey (rbalint)
Changed in systemd (Ubuntu):
assignee: nobody → Balint Reczey (rbalint)
status: Confirmed → In Progress
Changed in systemd (Ubuntu Disco):
status: New → In Progress
Balint Reczey (rbalint)
description: updated
Revision history for this message
Adam Conrad (adconrad) wrote : Please test proposed package

Hello Jeremy, or anyone else affected,

Accepted systemd into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/240-6ubuntu5.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Disco):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-disco
Changed in systemd (Ubuntu Disco):
assignee: nobody → Balint Reczey (rbalint)
Revision history for this message
Adam Conrad (adconrad) wrote :

Note that the fix is committed to git for eoan, but not uploaded as systemd currently fails its testsuite on eoan. This needs investigation.

Revision history for this message
Jeremy Soller (jackpot51) wrote :

Thanks, I will get it tested soon

Revision history for this message
Łukasz Zemczak (sil2100) wrote : Autopkgtest regression report (systemd/240-6ubuntu5.2)

All autopkgtests for the newly accepted systemd (240-6ubuntu5.2) for disco have finished running.
There have been regressions in tests triggered by the package. Please visit the sru report page and investigate the failures.

https://people.canonical.com/~ubuntu-archive/pending-sru.html#disco

Revision history for this message
Jeremy Soller (jackpot51) wrote :

I tested 240-6ubuntu5.2, it fixes the issue on Ubuntu 19.04

tags: added: verification-done-disco
removed: verification-needed-disco
tags: removed: verification-needed
Revision history for this message
Matthew Geier (matthew-sleeper) wrote :

>> FWIW, I think not booting a 19.04 ISO is probably fine,

Unless like me, you have just upgraded your hardware to a Ryzen 3600x and find you have to downgrade from the version you are using on your old machine to use the new machine you just purchased.....

Revision history for this message
Balint Reczey (rbalint) wrote :

@matthew-sleeper & All,

The current options workaround options are installing 18.04/18.10, then upgrading to 19.04.
Using the netboot installer to install 19.04 on AMD 3000 series machines may also work after the fix for this but gets released to -updates:
http://cdimage.ubuntu.com/netboot/disco/

I personally believe that this bug would warrant at least spinning a half-official installer for amd64, but @adconrad can make this call, not me.

Revision history for this message
Jakub Mankowski (1-jakub) wrote :

Hi, could you confirm that this bug is regarding all 3 gen Ryzen processors? I mean that 3400G (with integrated GPU) processor has Zen+ architecture but others have Zen2. Thanks.

Revision history for this message
Jakub Mankowski (1-jakub) wrote :

Hi, I can confirm now that newly installed 19.04 on 3400G is booting without any problems.

Revision history for this message
Bryan Quigley (bryanquigley) wrote :
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 240-6ubuntu5.2

---------------
systemd (240-6ubuntu5.2) disco; urgency=medium

  [ Jeremy Soller ]
  * random-util: eat up bad RDRAND values seen on AMD CPUs.
    This fixes AMD Ryzen 3000 series failing to boot (LP: #1835809)

 -- Balint Reczey <email address hidden> Tue, 09 Jul 2019 11:52:55 +0200

Changed in systemd (Ubuntu Disco):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

tags: removed: rls-dd-incoming rls-ee-incoming
Revision history for this message
Alexander Stohr (alexander-stohr) wrote :

AMD has pulled back their first BIOS/Microcode fix due to problems with some boards switching down from PCI 4.0 to PCI 2.0 - the release seemingly only went as far as the big board vendors. the board vendors got order to not re-distribute that version in any form. sounds reasonable to me. indeed its already indicated that AMD still hopes to provide a solution in a near time frame.

Revision history for this message
Alexander Stohr (alexander-stohr) wrote :

Maybe unrelated (but those communications channel might be worth watching for related news):
Robert Hallock of AMD has announced a new beta driver for the chipset that is/was available via google drive. It is said this fixes something relevant for at least the game Destiny 2. As the test phase was closed already a quick follow up in about 1 day was promised.
https://www.reddit.com/r/Amd/comments/ciajef/placeholder_update_on_whea_warnings_destiny_2_and/evayush/
https://drive.google.com/file/d/1mojPNWcOuuhxvdBkYJlkLbsDXwGLImGp/view

Revision history for this message
Alexander Stohr (alexander-stohr) wrote :

A bunch of updates went public - the AGESA component update is still due as its pushed to the end user via the integration and qualification pipeline of the board vendors. It got said this will take several weeks.
https://community.amd.com/community/gaming/blog/2019/07/30/community-update-5-let-s-talk-clocks-voltages-and-destiny-2

Revision history for this message
Giacomo Graziosi (g.graziosi) wrote :

Does anyone here have a work around to install Ubuntu 19.04 without upgrading from a previous version?

Revision history for this message
Balint Reczey (rbalint) wrote :
Revision history for this message
TJ (tj) wrote :

Today in IRC we were asked if the ficed package is in the 19.04 ISOs. It isn't so they will fail to boot on AMD Ryzen 3000 CPUs.

Is there a plan to re-build the 19.04 ISOs? If not there ought to at least be a prominent addition to the 19.04 release notes.

Revision history for this message
Ben Hamilton (exharris) wrote :

IS this fix now implemented to the latest ISO available for download?

Changed in systemd (Ubuntu Eoan):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 240-6ubuntu13

---------------
systemd (240-6ubuntu13) eoan; urgency=medium

  * Drop s390x seccomp fix causing regression on s390x
    Files:
    - debian/patches/src-shared-seccomp-util.c-Add-mmap-definitions-for-s390.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=da95e1d022e94a4d3ce0b69bd6eb398c95d09f24

 -- Balint Reczey <email address hidden> Mon, 26 Aug 2019 17:02:54 +0200

Changed in systemd (Ubuntu Eoan):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.