Ubuntu
openssh package

Issue on sshd finds correct private key for a certificate when using ssh-agent

Bug #1952421 reported by Utkarsh Gupta on 2021-11-26

This bug affects 2 people

	Status	Importance	Assigned to
openssh (Ubuntu)	Fix Released	Undecided	Unassigned
Focal	Fix Released	Undecided	Chloé Smith
Hirsute	Fix Released	Undecided	Chloé Smith
Impish	Fix Released	Undecided	Utkarsh Gupta

Bug Description

Reported as https://bugzilla.mindrot.org/show_bug.cgi?id=3254 upstream

[Impact]

* HostCertificate and HostKeyAgent are not working together in sshd due
   to a mismatched certificate's public key and private key. The function ` `sshkey_equal_public()`` incorrectly compares the certificate's public
  key with a private key, never finding a match. The impact is that sshd
  cannot use said certificate *even though* its private key is indeed in
  ssh-agent.

* What it should do is compare the certificate's public key with a public key in `sensitive_data`.

* Having this SRU-ed is a direct ask from one of the major cloud partners.
  They are currently using a customised version of the package to work
  around this issue, and we would like them to use a package directly from
  our own archive.

* Looping through sensitive_data.host_pubkeys[j] *instead* of
sensitive_data.host_keys[j] fixes the issue

[https://github.com/openssh/openssh-portable/blob/V_8_4/sshd.c#L1936]

/* Find matching private key */
for (j = 0; j < options.num_host_key_files; j++) {
  if (sshkey_equal_public(key,
   sensitive_data.host_keys[j])) {
   sensitive_data.host_certificates[j] = key;
    break;
   }
}

vs.

/* Find matching private key */
for (j = 0; j < options.num_host_key_files; j++) {
  if (sshkey_equal_public(key,
   sensitive_data.host_pubkeys[j])) {
   sensitive_data.host_certificates[j] = key;
    break;
   }
}

[Test Plan]

* Due to the empirical nature of this bug, the test is quite straight
   forward. *Without* the fix, one cannot use certificates to authenticate
   successfully (e.g. ``sshd -c /path/to/certificate.pem``)
   whereas with the fix (assuming the certificate matches a host key) you
   can create a channel.

[Where problems could occur]

* This has already been fixed both upstream and in Jammy without issue.
However, if a regression where to happen it would probably be in one of
two ways:

     * A dependency/reverse-dependency issue stemming from the version
       bump that will happen if this fix is ported. We mitigate this risk
       by testing for these exact types of regression,
       and by selecting carefully what to label this new version.

     * Accidentally breaking a set up that was made to work around this
       bug in the first place. The risk of this is lower, as the most
       likely fix is the one being implemented here anyway. Though
       to mitigate this more we can describe exactly what is happening
       with the fix in the changelog.

This affects every version of openssh back until Focal, at least.

See original description

Tags:

Related branches

~sergiodj/ubuntu/+source/openssh:lp1966591-ssh-keygen-permission-focal

Superseded for merging into ubuntu/+source/openssh:ubuntu/devel

Canonical Server packageset reviewers: Pending requested 2022-03-31

Canonical Server: Pending requested 2022-03-31

~kajiya/ubuntu/+source/openssh:lp1952421-sshd-matching-certs-focal

On hold for merging into ubuntu/+source/openssh:ubuntu/focal-devel

Utkarsh Gupta (community): Approve on 2021-12-08

~kajiya/ubuntu/+source/openssh:lp1952421-sshd-matching-certs

Merged into ubuntu/+source/openssh:ubuntu/hirsute-devel at revision 1e5a2f97881c0f4e5baaf1ab988a226b72340512

Utkarsh Gupta (community): Approve on 2021-12-02

Canonical Server packageset reviewers: Pending requested 2021-12-01

~utkarsh/ubuntu/+source/openssh:lp1952421-matching-certs

Merged into ubuntu/+source/openssh:ubuntu/impish-devel at revision 785da1daa2a504b88e402832bfa5c6a493d6b83c

Chloé Smith (community): Approve on 2021-11-29

Christian Ehrhardt  (community): Approve on 2021-11-26

Canonical Server: Pending requested 2021-11-26

git-ubuntu import: Pending requested 2021-11-26

Utkarsh Gupta (utkarsh) on 2021-11-26

Changed in openssh (Ubuntu):
status:	New → Fix Released

Chloé Smith (kajiya) on 2021-11-29

description:

updated

Utkarsh Gupta (utkarsh) on 2021-11-29

Changed in openssh (Ubuntu Impish):
assignee:	nobody → Utkarsh Gupta (utkarsh)
Changed in openssh (Ubuntu Hirsute):
assignee:	nobody → Chloé Smith (kajiya)
Changed in openssh (Ubuntu Focal):
assignee:	nobody → Chloé Smith (kajiya)

Revision history for this message

Brian Murray (brian-murray) wrote on 2021-11-30: Please test proposed package

Hello Utkarsh, or anyone else affected,

Accepted openssh into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:8.4p1-6ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in openssh (Ubuntu Impish):
status:	New → Fix Committed
tags:	added: verification-needed verification-needed-impish

Revision history for this message

Robie Basak (racb) wrote on 2021-12-08:

Hello Utkarsh, or anyone else affected,

Accepted openssh into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:8.4p1-5ubuntu1.2 in a few hours, and then in the -proposed repository.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in openssh (Ubuntu Hirsute):
status:	New → Fix Committed
tags:	added: verification-needed-hirsute
Changed in openssh (Ubuntu Focal):
status:	New → Fix Committed
tags:	added: verification-needed-focal

Revision history for this message

Robie Basak (racb) wrote on 2021-12-08:

Hello Utkarsh, or anyone else affected,

Accepted openssh into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:8.2p1-4ubuntu0.4 in a few hours, and then in the -proposed repository.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message

Ubuntu SRU Bot (ubuntu-sru-bot) wrote on 2021-12-08: Autopkgtest regression report (openssh/1:8.2p1-4ubuntu0.4)

All autopkgtests for the newly accepted openssh (1:8.2p1-4ubuntu0.4) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.44.1-1ubuntu1 (ppc64el)
nova/2:21.2.3-0ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#openssh

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message

Ubuntu SRU Bot (ubuntu-sru-bot) wrote on 2021-12-08: Autopkgtest regression report (openssh/1:8.4p1-5ubuntu1.2)

All autopkgtests for the newly accepted openssh (1:8.4p1-5ubuntu1.2) for hirsute have finished running.
The following regressions have been reported in tests triggered by the package:

ganeti/3.0.1-2 (armhf)
diffoscope/171 (armhf)
dropbear/2020.81-3 (armhf)
libssh/0.9.5-1ubuntu0.1 (armhf)
libnetapp-perl/500.002-2 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/hirsute/update_excuses.html#openssh

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message

Utkarsh Gupta (utkarsh) wrote on 2021-12-09:

Hello,

All the 2 (for Focal) and 5 (for Hirsute) tests were re-triggered and are passing now, so there's no real regression. We'll proceed with the verification from our end shortly. TIA! \o/

Revision history for this message

Chloé Smith (kajiya) wrote on 2022-01-03:

Hey everyone,

I can confirm the fix has been tested by our friends at Google (Anthos) for Focal 20.04, using the same patch used in 1:8.2p1-4ubuntu0.4 but *not* by using the package from focal-proposed itself.
Hopefully this still suffices? Please let me know if not and I'll re-run the verification again using an instance pulling from focal-proposed.

[RATIONALE]

Need SSH to authenticate a ``HostCertificate`` and an SSH agent that holds the corresponding host private key.

The sshd_config has the following directives:

-- HostCertificate the public host certificate whose public key matches the private key stored in the ssh agent

-- HostKey the public key of the host keypair

-- HostKeyAgent the socket of the ssh agent that holds the host private key

Before the patch, this combination didn't work - even though it authenticated successfully the setup behaved as if ``HostCertificate`` was never configured (i.e. it authenticated using only the public key and the private key in the ssh agent).

[VERIFICATION OF FIX]
````
sh-agent -a /path/agent-socket
SSH_AUTH_SOCK=/path/agent-socket ssh-add -k /path/hostkey
````

Then ran ``sshd`` with:

````
HostCertificate /path/hostkey-cert.pub
HostKey /path/hostkey.pub
HostKeyAgent /path/agent-socket
````

Then configured the CA trust anchor on the client's side.
(localhost was used, but it would be the same if a second host is used as a client)

````
ssh -vv localhost
````

shows the host certificate was seen and used.

tags:

added: verification-done-focal
removed: verification-needed-focal

Revision history for this message

Chris Halse Rogers (raof) wrote on 2022-01-05:

For SRU verification we require that the actual package from the -proposed archive is tested. This is to avoid the chance that the patch is mis-applied in the package uploaded to the archive, or that the archive build environment differs in a way that affects the patch.

We'll also need this to be verified for Hirsute and Impish before we can release it for Focal, to ensure that upgrades do not regress this bug.

Revision history for this message

Chloé Smith (kajiya) wrote on 2022-01-06:

Hirsute verification

[INSTALLED PKG VERSION]
kajiya@chloe-HAL:~/Documents/work$ apt-cache policy openssh-server
openssh-server:
  Installed: 1:8.4p1-5ubuntu1.2
  Candidate: 1:8.4p1-5ubuntu1.2
  Version table:
*** 1:8.4p1-5ubuntu1.2 500
        500 http://gb.archive.ubuntu.com/ubuntu hirsute-proposed/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu hirsute-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1:8.4p1-5ubuntu1.1 500
        500 http://gb.archive.ubuntu.com/ubuntu hirsute-updates/main amd64 Packages
     1:8.4p1-5ubuntu1 500
        500 http://gb.archive.ubuntu.com/ubuntu hirsute/main amd64 Packages

[PROCEDURE]
Create the keys/certs needed
``ssh-keygen -t rsa -b 4096 -f host_ca -C host_ca`` (no passphrase)
``ssh-keygen -f ssh_host_rsa_key -N '' -b 4096 -t rsa``
``ssh-keygen -s host_ca -I localhost -h -n localhost -V +52w ssh_host_rsa_key.pub``

Copied ssh_host_rsa_key* files over to /etc/ssh and added the following to /etc/ssh/sshd_config
``HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub``

Restarted sshd using systemctl and added ``@cert-authority localhost ssh-rsa abcdefg`` (ssh-rsa abcdefg is the contents of host_ca.pub) to ~/.ssh/known_hosts

Finally, running
``ssh -vv localhost 2>&1 | grep "Server host certificate"`` gives

ssh -vv kajiya@localhost 2>&1 | grep "Server host certificate"
debug1: Server host certificate: <email address hidden> SHA256:ufStWAPad1IQ08xMPM1iF4u4JHEaeAuQcD3qoe8yJ9A, serial 0 ID "localhost" CA ssh-rsa SHA256:3iVQ6wcBeoRO3S12jO8K34Do8HbVTPxiBp3rNzCngGc valid from 2022-01-05T17:20:00 to 2023-01-04T17:21:17
debug2: Server host certificate hostname: localhost

which tells us the certificate was seen and used

Revision history for this message

Chloé Smith (kajiya) wrote on 2022-01-06 (last edit on 2022-01-06):

#10

Impish verification

[INSTALLED PKG VERSION]
kajiya@chloe-HAL:~$ apt-cache policy openssh-server
openssh-server:
  Installed: 1:8.4p1-6ubuntu2.1
  Candidate: 1:8.4p1-6ubuntu2.1
  Version table:
*** 1:8.4p1-6ubuntu2.1 400
        400 http://gb.archive.ubuntu.com/ubuntu impish-proposed/main amd64 Packages
        400 http://archive.ubuntu.com/ubuntu impish-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1:8.4p1-6ubuntu2 500
        500 http://gb.archive.ubuntu.com/ubuntu impish/main amd64 Packages

Copied ssh_host_rsa_key* files over to /etc/ssh and added the following to /etc/ssh/sshd_config
``HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub``

Restarted sshd using systemctl and added ``@cert-authority localhost ssh-rsa abcdefg`` (ssh-rsa abcdefg is the contents of host_ca.pub) to ~/.ssh/known_hosts

Finally, running
ssh -vv kajiya@localhost 2>&1 | grep "Server host certificate" gives

debug1: Server host certificate: <email address hidden> SHA256:pprTqBvT2oazgTsfPF+RD47ca/W1U4JCgq5fl7m1LkA, serial 0 ID "localhost" CA ssh-rsa SHA256:l3PYuQBJMLruGeASt+BKEDGLDlk5NHx59cwW6/Qgzs4 valid from 2022-01-05T22:11:00 to 2023-01-04T22:12:07
debug2: Server host certificate hostname: localhost

which tells us the certificate was seen and used

Revision history for this message

Chloé Smith (kajiya) wrote on 2022-01-06:

#11

Focal verification

[INSTALLED PKG VERSION]
chlo@BIG-HAL:~$ apt-cache policy openssh-server
openssh-server:
  Installed: 1:8.2p1-4ubuntu0.4
  Candidate: 1:8.2p1-4ubuntu0.4
  Version table:
*** 1:8.2p1-4ubuntu0.4 400
        400 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1:8.2p1-4ubuntu0.3 500
        500 http://gb.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
     1:8.2p1-4ubuntu0.2 500
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
     1:8.2p1-4 500
        500 http://gb.archive.ubuntu.com/ubuntu focal/main amd64 Packages

Copied ssh_host_rsa_key* files over to /etc/ssh and added the following to /etc/ssh/sshd_config
``HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub``

Restarted sshd using systemctl and added ``@cert-authority localhost ssh-rsa abcdefg`` (ssh-rsa abcdefg is the contents of host_ca.pub) to ~/.ssh/known_hosts

Finally, running

ssh -vv chlo@localhost 2>&1 | grep "Server"
debug1: Server host certificate: <email address hidden> SHA256:s2gq1xBSdetCarwElgQd0NbjJbiE3iLDxFtJqDhBFF4, serial 0 ID "localhost" CA ssh-rsa SHA256:v8ZgezKD9Zw/Ns8I0W6mfvxCAo9jv3WznUYAFhfPfCU valid from 2022-01-05T22:46:00 to 2023-01-04T22:47:11
debug2: Server host certificate hostname: localhost

which tells us the certificate was seen and used

tags:

added: verification-done-hirsute verification-done-impish
removed: verification-needed-hirsute verification-needed-impish

Revision history for this message

Chloé Smith (kajiya) wrote on 2022-01-06:

#12

Apologies, I changed the tags *after* posting all the comments just to make it more confusing...

Revision history for this message

Launchpad Janitor (janitor) wrote on 2022-01-06:

#13

This bug was fixed in the package openssh - 1:8.4p1-6ubuntu2.1

---------------
openssh (1:8.4p1-6ubuntu2.1) impish; urgency=medium

  * d/p/match-host-certs-w-public-keys.patch: Add patch
    to match host certificates agianst host public keys.
    (LP: #1952421)

-- Utkarsh Gupta <email address hidden> Fri, 26 Nov 2021 17:25:36 +0530

Changed in openssh (Ubuntu Impish):
status:	Fix Committed → Fix Released

Revision history for this message

Brian Murray (brian-murray) wrote on 2022-01-06: Update Released

#14

The verification of the Stable Release Update for openssh has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2022-01-06:

#15

This bug was fixed in the package openssh - 1:8.4p1-5ubuntu1.2

---------------
openssh (1:8.4p1-5ubuntu1.2) hirsute; urgency=medium

  * d/p/match-host-certs-w-public-keys.patch: Add patch
    to match host certificates agianst host public keys.
    (LP: #1952421)

-- Chloé S <email address hidden> Wed, 01 Dec 2021 14:12:42 +0000

Changed in openssh (Ubuntu Hirsute):
status:	Fix Committed → Fix Released

Revision history for this message

Launchpad Janitor (janitor) wrote on 2022-01-06:

#16

This bug was fixed in the package openssh - 1:8.2p1-4ubuntu0.4

---------------
openssh (1:8.2p1-4ubuntu0.4) focal; urgency=medium

  * d/p/match-host-certs-w-public-keys.patch: Add patch
    to match host certificates agianst host public keys.
    (LP: #1952421)

-- Chloé S <email address hidden> Thu, 02 Dec 2021 22:38:52 +0000

Changed in openssh (Ubuntu Focal):
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntuopenssh package

Issue on sshd finds correct private key for a certificate when using ssh-agent

Bug Description

Related branches

Other bug subscribers

Remote bug watches

Ubuntu
openssh package