How to downgrade gcc to 9.3.0-17ubuntu1

Asked by Stuart MacDonald

I use nVidia's drivers from https://www.nvidia.com/Download/index.aspx to set up my nVidia T500 GPU for Cuda processing. (The onboard Intel graphics runs the monitors.)

Trying to install NVIDIA-Linux-x86_64-510.68.02.run fails with:
(exact error message to be provided later, but is a gcc version mismatch between the gcc that compiled the kernel and the currently installed gcc)

This is a pretty critical failure. nVidia is correct: one should build kernel drivers with the same version of compiler that built the kernel, to avoid subtle and frustrating bugs.

Weird. Let's see:
$ cat /proc/version
Linux version 5.13.0-27-generic (buildd@lgw01-amd64-045) (gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #29~20.04.1-Ubuntu SMP Fri Jan 14 00:32:30 UTC 2022

Okay, the kernel was compiled with gcc (Ubuntu 9.3.0-17ubuntu1~20.04).

What's my version?
$ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0

So there is a mismatch.

Sure enough, when I check, I see that gcc upgraded during today's updates:
$ zgrep --color=always --no-filename " upgrade " /var/log/dpkg.log* | sort | grep -i "g\(cc\|++\)"
...
2021-07-06 18:28:01 upgrade gcc-10-base:amd64 10-20200411-0ubuntu1 10.3.0-1ubuntu1~20.04
2021-07-06 18:28:02 upgrade libgcc-s1:amd64 10-20200411-0ubuntu1 10.3.0-1ubuntu1~20.04
2021-07-06 18:29:03 upgrade gcc-9:amd64 9.3.0-10ubuntu2 9.3.0-17ubuntu1~20.04
2021-07-06 18:29:05 upgrade libgcc-9-dev:amd64 9.3.0-10ubuntu2 9.3.0-17ubuntu1~20.04
2021-07-06 18:29:07 upgrade gcc-9-base:amd64 9.3.0-10ubuntu2 9.3.0-17ubuntu1~20.04
...
2022-03-08 15:58:21 upgrade g++-9:amd64 9.3.0-17ubuntu1~20.04 9.4.0-1ubuntu1~20.04
2022-03-08 15:58:21 upgrade gcc-9:amd64 9.3.0-17ubuntu1~20.04 9.4.0-1ubuntu1~20.04
2022-03-08 15:58:22 upgrade libgcc-9-dev:amd64 9.3.0-17ubuntu1~20.04 9.4.0-1ubuntu1~20.04
2022-03-08 15:58:23 upgrade gcc-9-base:amd64 9.3.0-17ubuntu1~20.04 9.4.0-1ubuntu1~20.04
...
2022-05-10 10:15:57 upgrade g++-9:amd64 9.4.0-1ubuntu1~20.04 9.4.0-1ubuntu1~20.04.1
2022-05-10 10:15:58 upgrade gcc-9:amd64 9.4.0-1ubuntu1~20.04 9.4.0-1ubuntu1~20.04.1
2022-05-10 10:16:00 upgrade libgcc-9-dev:amd64 9.4.0-1ubuntu1~20.04 9.4.0-1ubuntu1~20.04.1
2022-05-10 10:16:01 upgrade gcc-9-base:amd64 9.4.0-1ubuntu1~20.04 9.4.0-1ubuntu1~20.04.1

Easy peasy, I'll downgrade gcc-9-base to the previous version 9.3.0-17ubuntu1. (It's actually a couple of versions back, but, it's the most recent 9.3 package, and, I think I had this same install failure in Mar but didn't have time to dig into it then.) Let's see:
$ sudo apt update
...
$ sudo apt policy gcc-9-base
gcc-9-base:
  Installed: 9.4.0-1ubuntu1~20.04.1
  Candidate: 9.4.0-1ubuntu1~20.04.1
  Version table:
 *** 9.4.0-1ubuntu1~20.04.1 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
        100 /var/lib/dpkg/status
     9.3.0-10ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

That's... weird. Where are 9.3.0-17ubuntu and 9.4.0-1ubuntu1?

I'd really like not to downgrade all the way to 9.3.0-10ubuntu2 if possible.

Question information

Language:
English Edit question
Status:
Answered
For:
Ubuntu Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Manfred Hampl (m-hampl) said :
#1

You did not provide details about your system, e.g. the Ubuntu release that you are running (I assume 20.04).

The gcc 9.3.0-17ubuntu1~20.04 packages can be found via https://launchpad.net/ubuntu/+source/gcc-9/+publishinghistory and https://launchpad.net/ubuntu/+source/gcc-9/9.3.0-17ubuntu1~20.04

Because of interdependencies you most probably have to downgrade not only the gcc-9 package, but also several other ones (gcc-9-base, cpp-9 and more)

Another potential option is using the Ubuntu-provided nvidia-510 packages instead of those from the nvidia web pages.
See https://launchpad.net/ubuntu/+source/nvidia-graphics-drivers-510 (currently 510.60.02-0ubuntu0.20.04.2 with 510.68.02 already available in PPAs).

Revision history for this message
Stuart MacDonald (studog-ubuntuone) said :
#2

Apologies. Correct. Ubuntu 20.04.

I saw the publishing history before posting my question. I guess what's missing is I don't know how to use the information on the publshing page to make apt install that specific version. When I try currently I get:

$ sudo apt install gcc-9-base=9.3.0-17ubuntu1ntu1
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Version '9.3.0-17ubuntu1' for 'gcc-9-base' was not found

Please point me at some documentation that describes how to use the publishing history to make apt install that version?

I'm assuming that if I downgrade gcc-9-base that will cause all the dependencies to downgrade too.

I have used the provided drivers in the past but like to be current.

Revision history for this message
Stuart MacDonald (studog-ubuntuone) said :
#3

The compiler failure message:

=======================================================================
DKMS make.log for nvidia-510.68.02 for kernel 5.13.0-27-generic (x86_64)
Tue May 10 17:35:43 EDT 2022
make[1]: Entering directory '/usr/src/linux-headers-5.13.0-27-generic'

Compiler version check failed:

The major and minor number of the compiler used to
compile the kernel:

gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, GNU ld (GNU Binutils for Ubuntu) 2.34

does not match the compiler used here:

cc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

It is recommended to set the CC environment variable
to the compiler that was used to compile the kernel.

The compiler version check can be disabled by setting
the IGNORE_CC_MISMATCH environment variable to "1".
However, mixing compiler versions between the kernel
and kernel modules can result in subtle bugs that are
difficult to diagnose.

*** Failed CC version check. Bailing out! ***

make[2]: *** [/var/lib/dkms/nvidia/510.68.02/build/Kbuild:194: cc_version_check] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [Makefile:1879: /var/lib/dkms/nvidia/510.68.02/build] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.13.0-27-generic'
make: *** [Makefile:82: modules] Error 2
=======================================================================

The nVidia package has a flag to skip the version check, and that worked to force the build/install for me. It's not ideal due to the potential for unusual bugs though.

Revision history for this message
Manfred Hampl (m-hampl) said :
#4

You cannot use apt to downgrade to a version that has already been removed from the repositories (because being superseded), but you have to do that manually.

Suggested steps:

dpkg -l | grep 9.4.0-1ubuntu1~20.04.1
This gives a list of the gcc-related packages currently installed in the 9.4-* version.

Download the *.deb files for these packages from https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/ppa/+build/19784872 and put them into an empty directory
cd into that directory
sudo dpkg --install --force-downgrade *.deb

Can you help with this problem?

Provide an answer of your own, or ask Stuart MacDonald for more information if necessary.

To post a message you must log in.