Ubiquity removes kernel headers, fails to build nonfree drivers

This works completely fine for me via the normal method of searching for nvidia in software center and clicking "install"

What kernel headers did you have installed before? The correct kernel headers are installed by default for all Ubuntu architectures, to prevent exactly this. Also, nvidia-current has a dependency on "linux-headers-generic | linux-headers", so it should either pull -generic if you did not have any installed, or use an existing one if you have another flavour (like linux-headers-server).

Hi Martin,

I redid an installation to try to determine what the deal is with installed packages.

First, something that *may* be affecting behavior here. This system is behind a non-transparent proxy, so it is unable to access the internet without extra configuration.

Also, notice that on this particular test I used a system different from the original reported one, and I'm cheating a bit, as it doesn't actually have an nvidia chipset. But the package *should* build correctly. I can repeat this if needed on the actual nvidia system (running some other tests on it but should be available in a few hours). But I expect the results would be the same.

Here's what I did:

1- Installed Ubuntu 12.10 from a USB stick. This was a mostly default install. I did NOT click on the "install 3rd party software" checkbox.
2- Once the system booted I did dpkg --list, it's a largish list so I'll attach it as as file. I notice only a linux-headers-3.5.0-17 package is installed:

$ grep linux-header packagelist.txt
ii linux-headers-3.5.0-17 3.5.0-17.28 all Header files related to Linux kernel version 3.5.0

3- I configured the proxy for apt-get by adding this under /etc/apt/apt.conf.d/80proxy:

Acquire::http::proxy "my-super-secret-proxy";

4- I then did sudo apt-get update, and afterwards I *do* see a bit more information about linux headers:

$ dpkg --list "linux-header*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-====================-====================-====================================================================
un linux-headers <none> (no description available)
un linux-headers-3 <none> (no description available)
ii linux-headers-3.5.0-17 3.5.0-17.28 all Header files related to Linux kernel version 3.5.0
un linux-headers-generic <none> (no description available)

Do notice that linux-headers-generic is *not* installed.

5- sudo apt-get install nvidia-current. Results in this (notice the linux-headers-generic wasn't listed as a package to install, and it fails to build the driver.

$ sudo apt-get install nvidia-current

Reading package lists... Done
Building dependency tree
Reading state information... Done

The following extra packages will be installed:
  dkms fakeroot nvidia-settings python-xkit screen-resolution-extra
Suggested packages:
  dpkg-dev debhelper
The following NEW packages will be installed:
  dkms fakeroot nvidia-current nvidia-settings python-xkit screen-resolution-extra
0 upgraded, 6 newly installed, 0 to remove and 23 not upgraded.
Need to get 69.7 MB of archives.
After this operation, 209 MB of additional disk space will be used.
Do you want to continue [Y/n]? y
[Snip, fetching packages]
Fetched 69.7 MB in...

List of packages installed after clean 12.10 installation.

 subscribe apw
 subscribe albertomilone

Hello Daniel,

Daniel Manrique [2012-10-24 16:08 -0000]:
> 1- Installed Ubuntu 12.10 from a USB stick. This was a mostly default install. I did NOT click on the "install 3rd party software" checkbox.
> 2- Once the system booted I did dpkg --list, it's a largish list so I'll attach it as as file. I notice only a linux-headers-3.5.0-17 package is installed:
>
> $ grep linux-header packagelist.txt
> ii linux-headers-3.5.0-17 3.5.0-17.28 all Header files related to Linux kernel version 3.5.0

This seems to be the root of the problem. You need
linux-headers-3.5.0-17-generic and linux-headers-generic as well, and
they are indeed on the default image:

  http://releases.ubuntu.com/12.10/ubuntu-12.10-desktop-amd64.manifest

So this looks like a bug during installation. Perhaps ubiquity removes
the kernel headers for some reason? Can you please attach
/var/log/installer/syslog?

> 3- I configured the proxy for apt-get by adding this under
> /etc/apt/apt.conf.d/80proxy:

The proxy can certainly play a role in here, but as the kernel headers
are already in the live system and thus should be copied to the
installation, it does not need network access to get them.

> 5- sudo apt-get install nvidia-current. Results in this (notice the
> linux-headers-generic wasn't listed as a package to install, and it
> fails to build the driver.

Ah, I see the problem: nvidia-current depends on
"linux-headers-generic | linux-headers", the intention is that you can
also run e. g. the -lowlatency flavour and build against
linux-headers-lowlatency instead. However, linux-headers-3.5.0-17
provides "linux-headers, linux-headers-3", which is a bad thing to do
as they are incomplete! Only linux-headers-3.5.0-17-generic should
provide linux-headers (and does). Andy, what do you think about this?

Thanks,

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1070427

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

This does not need Apport logs, it's a dependency problem.

@pitti -- I concur the common headers are not even guarenteed to exist so it is not reasonable to have a way to depend on them, the linux-headers in my mind does mean 'there are valid binary headers' so I would expect that to be only on the complete headers. Will spin some patches.

Obvlously for this case simply installing linux-headers-lowlatency would resolve things.

Andy, this is indeed a combination of two bugs. linux-headers-generic ought to be installed by default, and we are still debugging why ubiquity removed it. But if it's not there, nvidia-current's dependencies out to pull it in, and not consider linux-headers-3.5.0-17 sufficient for this.

Hi, I'm attaching /var/log/installer/syslog and debug as requested. Also, please don't hesitate to ask for more information, I can even install with debug-ubiquity if needed.

I said I'd attach the debug file but it has nothing interesting so I won't. Do let me know if you want one produced with debug-ubiquity.

Status changed to 'Confirmed' because the bug affects multiple users.

Oh, but it has! Apparently this is a bug with the new behaviour for signed kernels:

Oct 24 15:22:09 ubuntu ubiquity: Removing linux-signed-generic ...
Oct 24 15:22:10 ubuntu ubiquity: Removing linux-signed-image-generic ...
Oct 24 15:22:10 ubuntu ubiquity: Removing linux-signed-image-3.5.0-17-generic ...
Oct 24 15:22:11 ubuntu ubiquity: Generating grub.cfg ...
Oct 24 15:22:11 ubuntu ubiquity:
Oct 24 15:22:11 ubuntu ubiquity: Found linux image: /boot/vmlinuz-3.5.0-17-generic
Oct 24 15:22:11 ubuntu ubiquity: Found initrd image: /boot/initrd.img-3.5.0-17-generic
Oct 24 15:22:11 ubuntu ubiquity: Found memtest86+ image: /boot/memtest86+.bin
Oct 24 15:22:11 ubuntu ubiquity: done
Oct 24 15:22:11 ubuntu ubiquity:
Oct 24 15:22:11 ubuntu ubiquity: Purging configuration files for linux-signed-image-3.5.0-17-generic ...
[...]
Oct 24 15:23:29 ubuntu ubiquity: Removing linux-generic ...
Oct 24 15:23:30 ubuntu ubiquity: Removing linux-headers-generic ...
Oct 24 15:23:30 ubuntu ubiquity: Removing linux-headers-3.5.0-17-generic ...

So for some reason ubiquity removes the kernel header packages.

I have this problem too. I did a fresh install (as the upgrade from 12.04 to 12.10 introduced a lot of other bugs that I hope will go away with a fresh install), and since, for some reason, the open source driver crashes constantly on my system (I had to install the gnome fallback in order to even run for more than a few seconds in X mode), I tried installing the nvidia drivers.

It fails for the same reason as above. I noticed that the .17 kernel headers were installed, but not the .18 ones -- and realized I did an upgrade immediately after getting up and running, as the first thing.

So it does indeed look as if the headers are installed correctly initially by the installer, but when the kernel is upgraded, the new headers are not installed (likely because the linux-headers-generic is not installed/removed).

The fix is as above (sudo apt-get install linux-headers-generic # will also cause DKMS to rebuild the nvidia drivers) -- but I guess a new user will have some challenges with this.

Unfortunately the open source driver still has too many bugs on my hardware (GT 520 type of card), so I need the nvidia one.

Hello Daniel, or anyone else affected,

Accepted ubiquity into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/ubiquity/2.10.22 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This bug was fixed in the package ubiquity - 2.13.6

---------------
ubiquity (2.13.6) raring; urgency=low

  * Don't remove kernel headers just because we're removing signed kernel
    images of the same flavour (LP: #1070427).
  * Automatic update of included source packages: yaboot-installer
    1.1.25ubuntu2.
 -- Colin Watson <email address hidden> Thu, 29 Nov 2012 17:13:37 +0000

Status changed to 'Confirmed' because the bug affects multiple users.

I know point releases are not planned for non-LTS releases. But shouldn't there be an exception if there are serious problems with ubiquity. I'd argue that a majority of users will install binary drivers which leads to unexperienced users being left with unusable systems. The interwebs is full with reports of people talking about re-installing Ubuntu over and over and always failing when they try to install the nvidia drivers.

For ubuntu-test-cases project: this type of bug should be tested for in the desktop ISO smoke testing jenkins job.

Upon installation using precise-daily, the system had precise-proposed enabled via /etc/apt/sources.list.d/proposed.list.
Upon installation on nvidia-current and updated 3.2 linux-image was installed from precise-proposed.

Installing nvidia-current, pulled in updated linux-image from proposed (3.2.0-35-generic).
Post-install compilation of nvidia-current (295.40-0ubuntu1.1) dkms module failed for 3.5 kernel.
Installing nvidia-current-updates (304.64-0ubuntu0.1) from precise-updates successfully built dmks module for the 3.5 kernel.

The linux-headers-generic & linux-headers-generic-lts-quantal were available on first boot and did not require manual installation.
Verification was performed in a AMD64 kvm virtual machine.
Please consider this bug as passing verification, but please note the nvidia-current FTBFS dkms above ( I don't know if that is intentional or not, e.g. I was doing manual apt-get install instead of via ubuntu-drivers-common).

Another problem is (at least for 12.10) that each time kernel is updated you have to munually install new kernel headers. It's not a problem for me but for new users it's a nogo. From a reputation perspective it's critical.

ergo-proxy [2012-12-15 12:31 -0000]:
> Another problem is (at least for 12.10) that each time kernel is updated
> you have to munually install new kernel headers.

Only if you uninstall the linux-generic metapackage. It is installed
by default, and keeps kernel headers in sync with the kernel.

Martin

Hi Martin, thx for the tip but this package is not installed by default on 12.10x64. I checked twice, one time on my desktop and second time when I made a fresh installation on VB.

v-done based on Dmitrijs' comment. If there are further issues I think we should deal with them separately at this point.

@ergo-proxy: The fix for this bug has not (yet?) been backported to 12.10, and may be at least a partial cause of not having linux-generic installed. In other words, I don't think that's "another problem", it's the very problem described in this bug.

This bug was fixed in the package ubiquity - 2.10.23

---------------
ubiquity (2.10.23) precise-proposed; urgency=low

  * Honour base-installer/kernel/altmeta when deciding which kernels to
    install or keep installed.

ubiquity (2.10.22) precise-proposed; urgency=low

  [ Dmitrijs Ledkovs ]
  * Make user-setup-encrypted-swap wait until partitioning has finished
    before attempting to adjust /target/etc/fstab. (LP: #1024343)
    (LP: #1068178)

  [ Colin Watson ]
  * Don't remove kernel headers just because we're removing signed kernel
    images of the same flavour (LP: #1070427).

ubiquity (2.10.21) precise-proposed; urgency=low

  [ Colin Watson ]
  * Fix missing parentheses that caused removable installation media
    sometimes to be selected as the default GRUB device (LP: #987418).
  * Support UEFI Secure Boot (LP: #1075181):
    - Try to install a signed kernel if base-installer asks for one, and
      don't leave signed kernels installed if it doesn't.
    - If the SecureBoot EFI variable is set, then ensure that
      grub-efi-amd64-signed and shim-signed remain installed.
    - Copy the signed kernel from /cdrom if it is not in the squashfs. If
      there is a signed kernel there but no unsigned one, then use sbattach
      to remove the signature and construct the unsigned kernel on the fly.
  * Automatic update of included source packages: base-installer
    1.122ubuntu7.2, grub-installer 1.68ubuntu5.1.

  [ Mario Limonciello ]
  * Don't let oem-config crash from an invalid server return on the timezone
    page. (LP: #887879)

  [ Dmitrijs Ledkovs ]
  * Do not preseed grub-install, if we are not proceeding to install. This
    should fix ValueError, I/O operation on closed file (LP: #1027648)
    (LP: #792652)
  * Prevent progress label to expand & shrink the window (LP: #1046241)

  [ Jonathan Riddell ]
  * Change from a KApplication to a QApplication to avoid using DBus, DBus
    as needed by KApplication no longer works with our multiple user changes
    LP: #1055967
 -- Colin Watson <email address hidden> Thu, 06 Dec 2012 17:20:32 +0000

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Hi,
I have the same issue. Ununtu 12.10, amd 64.
I had to install kernel headers to use nvidia-current driver.
nvidia-current requires kernel headers but it dosen't has it as dependency.

I think I may have the same thing, but not on initial install of the system. My system worked, and then the unity GUI update installer decided I needed a new kernel. (apt-get upgrade appeared to think the kernel was pinned.) Now I have a system I don't know how to fix, presently booted from a DVD, and a request on ubuntu-forums for help recovering.

Also 12.10.

It would be nice if that graphical installer SHOWED ITS OUTPUT to users. Laterly I haven't even been able to get it to open into a window; it just wants to do the installation with a tiny progress bar in the icon on the tool bar. And when I use "sudo apt-get update; sudo apt-get upgrade" I don't get the same results as with the tool.

My Quantal install went bad - would not let me login so I did a clean install twice and finally seen the 2nd time that if I installed linux-headers-generic first and then nvidia-current-updates it worked correctly. That was a lot of hassle.

I'm wondering what the status of this bug is as it relates to 12.04 LTS, or if that would be considered a separate issue. The Ubiquity issue causing linux-headers-generic not to be installed was introduced in 12.10. However, in 12.04 LTS it is still possible for linux-generic or linux-headers-generic to be manually removed, and the NVIDIA drivers do not depend on them.

In my case linux-generic and linux-headers-generic were uninstalled. When or why this happened I am not sure, and it may be been done intentionally by me, but in any case the NVIDIA packages should pull down all of the dependencies they require. When the headers are not installed jockey reports that the installation was successful even though the kernel module failed to be built. Installing the drivers using apt-get reports the same "Module build for the currently running kernel was skipped since the
kernel source for this kernel does not seem to be installed." message as the original bug reporter. Installing the headers fixes the problem.

Googling for '12.04 "kernel source for this kernel does not seem to be installed"' brings up plenty of reports for both NVIDIA and wireless drivers. Perhaps having dkms depend on linux-headers-generic would be possible? Whatever the proper solution, drivers in 12.04 should install properly, pulling down any dependencies they need.

Also, does the 'Fix Released' status for 12.10+ cover existing systems, or only newly installed systems? If the fix is in Ubiquity that does not seem to fully address all the possible cases of this bug arising.

This bug was nominated against a series that is no longer supported, ie raring. The bug task representing the raring nomination is being closed as Won't Fix.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

This bug was nominated against a series that is no longer supported, ie quantal. The bug task representing the quantal nomination is being closed as Won't Fix.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

quantal has seen the end of its life and is no longer receiving any updates. Marking the quantal task for this ticket as "Won't Fix".

There is no valid way in dpkg to represent the dependency on the "linux headers you need installed for the kernel you are running". The nearest would be a "linux-headers-generic | linux-headers-server" style or, but this triggers the first one that exists to install, not the one you need.