armv8 paca: poly1305 users see segfaults when pointer authentication in use on AWS Graviton 3 instances

Bug #1960863 reported by Matthew Ruffell
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
openssl (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Released
High
Matthew Ruffell

Bug Description

[Impact]

Support for hardware pointer authentication for armv8 systems was merged in openssl 1.1.1f, but it contains a bug in the implementation for poly1305 message authenticated code routines, which causes the calling program to fail pointer authentication, which causes the program to crash with a segmentation fault.

You can easily test it by accessing any website that uses poly1305. There is no workaround except use a different MAC.

[Testcase]

This bug applies to armv8 systems which support pointer authentication. Start an armv8 instance, such as a c7g graviton 3 instance on AWS, and make sure the paca flag is present in lscpu:

$ grep paca /proc/cpuinfo
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs paca pacg dcpodp svei8mm svebf16 i8mm bf16 dgh rng

Next, attempt to connect to any website that uses poly1305 MAC.

$ curl https://services.gradle.org/distributions/gradle-7.2-bin.zip --output gradle-7.2.bin
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Segmentation fault (core dumped)

There is a test package available in the following ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/sf327917-test

Install it, and poly1305 operations will no longer segfault.

[Where problems could occur]

The patch changes the order of operations for loading the SP and checking the AUTIASP against it, from checking the AUTIASP against nothing then loading the correct SP to check with, to the correct loading the SP and then checking the AUTIASP against the SP.

This only changes one code path for armv8 systems, and other architectures are not affected. This is also only limited to poly1305 MAC.

If a regression were to occur, it would only affect users of poly1035 MAC on armv8 with pacs support.

[Other info]

The fix landed upstream in openssl 1.1.1i with the following commit:

commit 5795acffd8706e1cb584284ee5bb3a30986d0e75
Author: Ard Biesheuvel <email address hidden>
Date: Tue Oct 27 18:02:40 2020 +0100
Subject: crypto/poly1305/asm: fix armv8 pointer authentication
Link: https://github.com/openssl/openssl/commit/5795acffd8706e1cb584284ee5bb3a30986d0e75

This commit is already present in Impish onward. Only Focal needs the fix.

Changed in openssl (Ubuntu):
status: New → Fix Released
Changed in openssl (Ubuntu Focal):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Matthew Ruffell (mruffell)
tags: added: focal sts
Revision history for this message
Matthew Ruffell (mruffell) wrote :

Attached is a debdiff for openssl on Focal

tags: added: sts-sponsor
tags: added: sts-sponsor-mfo
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

The changes look good to me.

It built correctly (ie, build-time tests) on all archs on PPAs w/ -updates and -proposed.
It passed autopkgtests on focal/arm64 against the PPA build too.

Thus, no test regressions in our arm64 cpus, and the synthetic test passes on newer arm64 cpu (tested by user).

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Hi @ubuntu-security,

I believe we should ask for your ack, as this is openssl.
If that's right, could you please review it and comment?

The changes look ok technically and w.r.t. test results
(details in sru template and comment above.)

Thanks!

Revision history for this message
Seth Arnold (seth-arnold) wrote :

None of us are ARM architecture experts but the upstream code nearby doesn't look like it's changed since this patch was introduced:

https://github.com/openssl/openssl/blame/master/crypto/poly1305/asm/poly1305-armv8.pl
https://github.com/openssl/openssl/blame/OpenSSL_1_1_1-stable/crypto/poly1305/asm/poly1305-armv8.pl
https://github.com/openssl/openssl/blame/openssl-3.0/crypto/poly1305/asm/poly1305-armv8.pl

The debdiff looks like it makes sense to me, and it all feels pretty plausible.

Thanks

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Seth, ack; thanks!

Matthew, I uploaded the changes to Focal; thanks for the SRU!

Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello Matthew, or anyone else affected,

Accepted openssl into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl/1.1.1f-1ubuntu2.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in openssl (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Robie Basak (racb) wrote :

This was a perfectly researched and written up bug and patch and made SRU review easy. Thank you!

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (openssl/1.1.1f-1ubuntu2.11)

All autopkgtests for the newly accepted openssl (1.1.1f-1ubuntu2.11) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

python3.8/3.8.10-0ubuntu1~20.04.2 (armhf, amd64)
linux-hwe-5.11/5.11.0-60.60 (amd64)
linux-azure-5.11/5.11.0-1029.32~20.04.2 (amd64)
mysql-8.0/8.0.28-0ubuntu0.20.04.3 (i386)
linux-gcp-5.13/5.13.0-1015.18~20.04.1 (amd64)
linux-oem-5.13/5.13.0-1029.36 (amd64)
kopanocore/8.7.0-7ubuntu1 (amd64)
python3.9/3.9.5-3ubuntu0~20.04.1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#openssl

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Matthew Ruffell (mruffell) wrote :

Performing verification for openssl on Focal.

An affected user performed the verification, due to c7g instance types being in "Preview" state on Amazon AWS, and not generally accessible.

The user started a c7g instance, and checked they had openssl 1.1.1f-1ubuntu2.10 from -updates.

They attempted to use the poly1035 MAC downloading the file from the testcase:

$ curl https://services.gradle.org/distributions/gradle-7.2-bin.zip --output gradle-7.2.bin
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Segmentation fault (core dumped)

They can reproduce the issue.

They then enabled -proposed from ports.ubuntu.com mirror, and installed openssl 1.1.1f-1ubuntu2.11.

They again tried downloading the file:

$ curl https://services.gradle.org/distributions/gradle-7.2-bin.zip --output gradle-7.2.bin
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0

(note the file doesn't actually download due to curl not automatically following 301 redirects):

$ curl https://services.gradle.org/distributions/gradle-7.2-bin.zip --output gradle-7.2.bin --verbose
...
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
...
< HTTP/1.1 301 Moved Permanently
< Location: https://downloads.gradle-dn.com/distributions/gradle-7.2-bin.zip
...

curl does not segfault, and exits successfully. The package in -proposed fixes the issue. Happy to mark as verified.

tags: added: verification-done verification-done-focal
removed: sts-sponsor verification-needed verification-needed-focal
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for openssl has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssl - 1.1.1f-1ubuntu2.11

---------------
openssl (1.1.1f-1ubuntu2.11) focal; urgency=medium

  * Fixup pointer authentication for armv8 systems that support it when
    using the poly1305 MAC, preventing segmentation faults. (LP: #1960863)
    - d/p/lp-1960863-crypto-poly1305-asm-fix-armv8-pointer-authenticat.patch

 -- Matthew Ruffell <email address hidden> Tue, 15 Feb 2022 10:10:01 +1300

Changed in openssl (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.