SSH 1.99 clients fail to connect to openssh-server 1:7.6p1-4ubuntu0.3

The attachment "protocol_major_version_mismatch_regression.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Thanks Kyle for the great report and prepping a fix already.
offending: 97f4d3083 is in >=1%7.6p1-1
fix: 9e9c4a7e5 is in >=1%7.7p1-1
fix: c9c1bba06 is in >=1%7.7p1-1

Matching that with versions in Ubuntu means only Bionic should be affected.

 openssh | 1:5.9p1-5ubuntu1 | precise | source
 openssh | 1:5.9p1-5ubuntu1.10 | precise-security | source
 openssh | 1:5.9p1-5ubuntu1.10 | precise-updates | source
 openssh | 1:6.6p1-2ubuntu1 | trusty | source
 openssh | 1:6.6p1-2ubuntu2.13 | trusty-security | source
 openssh | 1:6.6p1-2ubuntu2.13 | trusty-updates | source
 openssh | 1:7.2p2-4 | xenial | source
 openssh | 1:7.2p2-4ubuntu2.8 | xenial-security | source
 openssh | 1:7.2p2-4ubuntu2.8 | xenial-updates | source
 openssh | 1:7.2p2-4ubuntu2.9 | xenial-proposed | source
 openssh | 1:7.6p1-4 | bionic | source
 openssh | 1:7.6p1-4ubuntu0.3 | bionic-security | source
 openssh | 1:7.6p1-4ubuntu0.3 | bionic-updates | source
 openssh | 1:7.6p1-4ubuntu0.4 | bionic-proposed | source
 openssh | 1:7.9p1-10 | disco | source
 openssh | 1:8.0p1-6build1 | eoan | source
 openssh | 1:8.0p1-6ubuntu0.1 | eoan-proposed | source
 openssh | 1:8.1p1-5 | focal | source
 openssh | 1:8.2p1-4 | focal-proposed | source

@CJWatson - are you also doing the openssh SRUs or would you expect us to handle that?

Assigned to cjwatson for now, but feel free to tell us you want us to drive the SRU for this and we can change it.

I have time to maintain openssh in Debian, but in general I don't have cycles to deal with SRUs, so please could somebody else take care of that part?

Yep, thanks cjwatson

@Kyle - in prep for an SRU - do you have steps to reproduce this e.g. with which Ubuntu based client/options one can easily send 1.99 on a connection attempt?

The easiest way I found to reproduce was to monkey patch the python paramiko library. I've attached a short script which can be used to test a host. It requires either python-paramiko or python3-paramiko to run.

Thanks Kyle,
I agree the testcase is great and works in my tests.
This is exactly what I needed to craft the SRU template as needed.

But OTOH about severity of this, as it will mean everyone having ssh installed (which is almost every installation out there) will have to download and install a new package. I was wondering if there is a (can be more complex and doesn't have to have step-by-step instructions) real use-case that is making this bug more severe by breaking it. If there isn't I'm tempted to say it is a correct bug and fix, but doesn't qualify to do the SRU on its own.
We might then still prep it completely but hold it in -proposed to only release it together with some other more severe update that will force a new download anyway.

Looking forward to your answer and adding the SRU template for now ...

I've redone the patches following the usual patch guidelines and opened an MP with these at:
=> https://code.launchpad.net/~paelzer/ubuntu/+source/openssh/+git/openssh/+merge/380138

Autopkgtests are complete on the PPA at
https://bileto.ubuntu.com/#/ticket/3962
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3962/+packages

Tests all passed or are known force-badtest cases already.

Waiting for Kyle's response to properly handle the severity of this ...

Merge Proposal review is complete, but waiting on some feedback that helps to classify the severity and urgency correctly.

Depending on that the options will be:
- actually unimportant: don't SRU it at all
- some reasonable cases exists, but are very rare: SRU it but hold the release in block-proposed until the next "important" update comes
- reasonable case for the Ubuntu community, SRU right away

Setting the task to incomplete while waiting on that feedback

Canonical client has opened a case regarding this as a current issue preventing them from upgrading their systems from 14.04 to 18.04.

This blocker is due to the version of openssh-server on Bionic not allowing the SolarWinds monitoring agent to establish a successful SSH connection, as it advertises SSH-1.99 as the protocol.

I have updated the Impact statement in the initial comment with further information and references. Please re-evaluate this bug as having a current impact on some clients.

As per Mark's description, this seems to fall in the

> - some reasonable cases exists, but are very rare: SRU it but hold the release in block-proposed until the next "important" update comes

case described above.

ACK from the security team on the changes in the MP.

Hello Kyle, or anyone else affected,

Accepted openssh into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:7.6p1-4ubuntu0.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Thank you for the detailed test case, rationale and security-team review of the patches! I have accepted those into bionic-proposed and added, for the time being, a block-proposed-bionic tag. I am still not sure about the severity of this bug.

As I don't work on the openssh package, I'd like to leave the decision to the Server and Security teams. Are there any other changes scheduled to be released shortly? Do you think it's worth releasing to -updates even now? From the SRU perspective I'm +0, since it always is a concern that many users will get the update, but on the other hand if a certain set of users is broken, medium prio bugs are also good to fix. If it's decided that we want it, please remove the block tag.

Thanks for fixing this! This helps with the long-standing problem with Solarwinds SSH client: https://support.solarwinds.com/SuccessCenter/s/article/SAM-s-Linux-Unix-Script-monitor-fails-to-connect-on-a-server-running-OpenSSH-7-6

Server-Team: As you see in the bug-history we (Server Team) have ourselves stopped working on this believing it might be too much of a corner case waiting for it to come back. But that come-back has happened by even more people reporting to be affected. Therefore - as much as it initially seems to be just a corner case - as of today I do believe that there is a real case for this fix and releasing it IMHO seems to be the right choice.

Validated according to test case from description:

root@bionic-ssh:~# python3 test_bug_1863930.py localhost
Server is patched
root@bionic-ssh:~# dpkg -l | grep openssh
ii openssh-client 1:7.6p1-4ubuntu0.6 amd64 secure shell (SSH) client, for secure access to remote machines
ii openssh-server 1:7.6p1-4ubuntu0.6 amd64 secure shell (SSH) server, for secure access from remote machines
ii openssh-sftp-server 1:7.6p1-4ubuntu0.6 amd64 secure shell (SSH) sftp server module, for SFTP access from remote machines

Given we have an ACK from both Server and Security and this is affecting multiple users, I'll remove the blocked tag as well.

This bug was fixed in the package openssh - 1:7.6p1-4ubuntu0.6

---------------
openssh (1:7.6p1-4ubuntu0.6) bionic; urgency=medium

  * fix clients advertising version 1.99 (LP: #1863930)
    - d/p/lp-1863930-Fix-logic-bug-in-sshd_exchange_identification.patch
    - d/p/lp-1863930-unbreak-clients-that-advertise-protocol.patch

 -- Christian Ehrhardt <email address hidden> Tue, 03 Mar 2020 07:47:02 +0100

The verification of the Stable Release Update for openssh has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.