/boot/initrd.img-*.old-dkms files left behind

Status changed to 'Confirmed' because the bug affects multiple users.

Might be causing/aggravating this issue with full /boot:

https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1357093

Debian Bug:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717584

This affects 16.04 LTS as well:

$ dkms --version
dkms: 2.2.0.3

The Bourne shell script code to remove the backups could be added in kernel_prerm.d_dkms (or currently non-existing kernel_postrm.d_dkms) in dkms package source, but if the package was removed, there would be no code to remove the backups in conjunction with kernel removing. Thus, the code could be added to kernel/postrm.d/initramfs-tools (or currently non-existing kernel/prerm.d/initramfs-tools) in initramfs-tools package source, too:

# This is applied from make_initrd function in dkms command, which
# creates the possible initrd backup file.
remove_initrd_backup() {
 for initrd in "initrd-$1.img" "initramfs-$1.img" "initrd.img-$1" "initrd-$1"; do
  rm -fv /boot/"${initrd}".old-dkms \
   /boot/efi/efi/redhat/"${initrd}".old-dkms
 done
}

remove_initrd_backup "$1"

Here is the code as a file to retain text formatting.

Or update-initramfs could do more comprehensive deletion in its delete function:

rm -f "${initramfs}" "${initramfs}".*

instead of

rm -f "${initramfs}" "${initramfs}.bak"

Also found this:

https://github.com/dell/dkms/issues/2

It's known upstream in debian and dkms, but whether dkms or initramfs-tools should be responsible for cleaning up files left behind by dkms...I dunno. What you posted sounded like it made sense to me.

I just wanted to note at least dkms has an upstream bug report (which was opened May 2016).

Why not adding removing to both? That way update-initramfs would remove the extra ones even if dkms was uninstalled.

Merged upstream :D

Status changed to 'Confirmed' because the bug affects multiple users.

Will the fix be included in Zesty or Artful?

DKMS is a Dell project but not Project Sputnik. Marking as invalid for dell-sputnik.

According to commant #8
patch against current https://git.launchpad.net/~usd-import-team/ubuntu/+source/initramfs-tools/tree/update-initramfs
Removes all leftover initramfs files, even after dkms is uninstalled.

The attachment "patch against update-imitramfs" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Note that the patch in #15 removes the leftovers related to the kernel version being removed currently. Removing leftovers related to already purged kernels still have to be done manually, for example by linux-purge https://www.bountysource.com/issues/38300038-feature-request-the-command-should-work-like-this together with other extra kernel related stuff.

This bug was fixed in the package dkms - 2.3-3ubuntu2

---------------
dkms (2.3-3ubuntu2) artful; urgency=medium

  * 0014-remove-initrd-backup.patch: When removing a kernel also remove
    related .old-dkms file from /boot. (LP: #1515513)

 -- Brian Murray <email address hidden> Fri, 15 Sep 2017 15:33:17 -0700

While I've uploaded this fix to Artful for dkms, its not quite clear to me how the extra files end up getting created. I've a couple of systems using dkms and nvidia and I don't see any .old-dkms files on them. For this to SRU'ed, which I'm happy to do, we'd need a test case which describes how to recreate the issue. Could someone provide some information on how to recreate the issue of .old-dkms files filling up /boot? Thanks in advance!

It's in the DKMS itself. Not sure what the main branch is, but here is an example from some branch:
http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/vivid/dkms/vivid-updates/view/head:/dkms#L277

I'm looking for specific steps to recreate the problem on an Ubuntu system. What steps do I need to take for dkms to leave to a bunch of .old-dkms files in boot?

1. Have nvidia-3## and/or virtualbox installed.
2. Update kernel.

My testing indicates that the .old-dkms files are only created if "remake_initrd" is yes in /usr/sbin/dkms and that is controlled by "REMAKE_INITRD" in a dkms.conf file. From what I can tell most dkms.conf files do not contain the REMAKE_INITRD option. So I don't think this effects a lot of people.

I found some commonly used dkms packages that do set REMAKE_INITRD.

broadcom-sta-6.30.223.271/debian/broadcom-sta-dkms.dkms:REMAKE_INITRD="YES"
flashcache-3.1.3+git20150701/src/dkms.conf:REMAKE_INITRD=yes
r8168-8.044.02/debian/r8168-dkms.dkms:REMAKE_INITRD="YES"

And zfs sets REMAKE_INITRD sometimes.

Sorry I didn't test it. Turns out in my case it's Intel drivers

/var/lib/dkms $ grep REMAKE_INITRD * -rF
i915-4.6.3-4.4.0/1/build/dkms.conf:REMAKE_INITRD=yes

Hello Robin, or anyone else affected,

Accepted dkms into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dkms/2.2.0.3-2ubuntu11.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Robin, or anyone else affected,

Accepted dkms into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dkms/2.3-3ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

The verification of this has failed as the fix was incorrect.

Removing linux-image-4.10.0-19-generic (4.10.0-19.21) ...
Examining /etc/kernel/prerm.d.
run-parts: executing /etc/kernel/prerm.d/dkms 4.10.0-19-generic /boot/vmlinuz-4.10.0-19-generic
removed '/boot/initrd.img-4.10.0-19-generic.old-dkms'
dkms: removing: r8168 8.043.02 (4.10.0-19-generic) (x86_64)

-------- Uninstall Beginning --------
Module: r8168
Version: 8.043.02
Kernel: 4.10.0-19-generic (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

r8168.ko:
 - Uninstallation
   - Deleting from: /lib/modules/4.10.0-19-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod...

Backing up initrd.img-4.10.0-19-generic to /boot/initrd.img-4.10.0-19-generic.old-dkms
Making new initrd.img-4.10.0-19-generic
(If next boot fails, revert to initrd.img-4.10.0-19-generic.old-dkms image)
update-initramfs...

DKMS: uninstall completed.
Examining /etc/kernel/postrm.d .

The .old-dkms file is removed before a new one is created (see the Backing up bits). So the remove_initrd_backup run needs to be reordered for after the dkms call.

I've uploaded fixes to all releases previously identified as worth fixing which remove the .old-dkms file after its been created.

Brian, Where you have uploaded the code? The code at https://code.launchpad.net/ubuntu/+source/dkms is old.

Hello Robin, or anyone else affected,

Accepted dkms into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dkms/2.3-3ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Robin, or anyone else affected,

Accepted dkms into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dkms/2.2.0.3-2ubuntu11.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I've verified that the version of dkms, 2.3-3ubuntu1.2, in zesty-proposed now does the right thing.

Backing up initrd.img-4.10.0-19-generic to /boot/initrd.img-4.10.0-19-generic.old-dkms
Making new initrd.img-4.10.0-19-generic
(If next boot fails, revert to initrd.img-4.10.0-19-generic.old-dkms image)
update-initramfs...

DKMS: uninstall completed.
removed '/boot/initrd.img-4.10.0-19-generic.old-dkms'
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 4.10.0-19-generic /boot/vmlinuz-4.10.0-19-generic
update-initramfs: Deleting /boot/initrd.img-4.10.0-19-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 4.10.0-19-generic /boot/vmlinuz-4.10.0-19-generic
Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-4.10.0-37-generic
Found initrd image: /boot/initrd.img-4.10.0-37-generic
Found linux image: /boot/vmlinuz-4.10.0-28-generic
Found initrd image: /boot/initrd.img-4.10.0-28-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
done
bdmurray@clean-zesty-amd64:~$ ls -lh /boot/
total 149M
-rw-r--r-- 1 root root 1.4M Jun 29 02:14 abi-4.10.0-28-generic
-rw-r--r-- 1 root root 1.4M Oct 6 12:45 abi-4.10.0-37-generic
-rw-r--r-- 1 root root 201K Jun 29 02:14 config-4.10.0-28-generic
-rw-r--r-- 1 root root 201K Oct 6 12:45 config-4.10.0-37-generic
drwxr-xr-x 5 root root 4.0K Oct 10 09:33 grub
-rw-r--r-- 1 root root 42M Oct 10 09:26 initrd.img-4.10.0-28-generic
-rw-r--r-- 1 root root 41M Oct 10 09:26 initrd.img-4.10.0-28-generic.old-dkms
-rw-r--r-- 1 root root 42M Oct 10 09:29 initrd.img-4.10.0-37-generic
-rw-r--r-- 1 root root 179K Jan 28 2016 memtest86+.bin
-rw-r--r-- 1 root root 181K Jan 28 2016 memtest86+.elf
-rw-r--r-- 1 root root 181K Jan 28 2016 memtest86+_multiboot.bin
-rw------- 1 root root 3.6M Jun 29 02:14 System.map-4.10.0-28-generic
-rw------- 1 root root 3.6M Oct 6 12:45 System.map-4.10.0-37-generic
-rw------- 1 root root 7.3M Jun 29 02:14 vmlinuz-4.10.0-28-generic
-rw------- 1 root root 7.3M Oct 6 12:45 vmlinuz-4.10.0-37-generic

The verification of this also passed for Xenial, Ubuntu 16.04.

run-parts: executing /etc/kernel/prerm.d/dkms 4.4.0-31-generic /boot/vmlinuz-4.4.0-31-generic
dkms: removing: r8168 8.041.00 (4.4.0-31-generic) (x86_64)

-------- Uninstall Beginning --------
Module: r8168
Version: 8.041.00
Kernel: 4.4.0-31-generic (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

r8168.ko:
 - Uninstallation
   - Deleting from: /lib/modules/4.4.0-31-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod....

Backing up initrd.img-4.4.0-31-generic to /boot/initrd.img-4.4.0-31-generic.old-dkms
Making new initrd.img-4.4.0-31-generic
(If next boot fails, revert to initrd.img-4.4.0-31-generic.old-dkms image)
update-initramfs....

DKMS: uninstall completed.
removed '/boot/initrd.img-4.4.0-31-generic.old-dkms'
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 4.4.0-31-generic /boot/vmlinuz-4.4.0-31-generic
update-initramfs: Deleting /boot/initrd.img-4.4.0-31-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 4.4.0-31-generic /boot/vmlinuz-4.4.0-31-generic
Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-4.4.0-97-generic
Found initrd image: /boot/initrd.img-4.4.0-97-generic
Found linux image: /boot/vmlinuz-4.4.0-38-generic
Found initrd image: /boot/initrd.img-4.4.0-38-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
done
bdmurray@clean-xenial-amd64:~$ ls -lh /boot/
total 132M
-rw-r--r-- 1 root root 1.2M Sep 6 2016 abi-4.4.0-38-generic
-rw-r--r-- 1 root root 1.2M Sep 19 13:29 abi-4.4.0-97-generic
-rw-r--r-- 1 root root 186K Sep 6 2016 config-4.4.0-38-generic
-rw-r--r-- 1 root root 187K Sep 19 13:29 config-4.4.0-97-generic
drwxr-xr-x 5 root root 4.0K Oct 10 09:42 grub
-rw-r--r-- 1 root root 36M Oct 10 09:39 initrd.img-4.4.0-38-generic
-rw-r--r-- 1 root root 36M Oct 10 09:38 initrd.img-4.4.0-38-generic.old-dkms
-rw-r--r-- 1 root root 37M Oct 10 09:41 initrd.img-4.4.0-97-generic
-rw-r--r-- 1 root root 179K Jan 28 2016 memtest86+.bin
-rw-r--r-- 1 root root 181K Jan 28 2016 memtest86+.elf
-rw-r--r-- 1 root root 181K Jan 28 2016 memtest86+_multiboot.bin
-rw------- 1 root root 3.7M Sep 6 2016 System.map-4.4.0-38-generic
-rw------- 1 root root 3.8M Sep 19 13:29 System.map-4.4.0-97-generic
-rw------- 1 root root 6.8M Sep 6 2016 vmlinuz-4.4.0-38-generic
-rw------- 1 root root 6.8M Sep 19 13:29 vmlinuz-4.4.0-97-generic

This bug was fixed in the package dkms - 2.3-3ubuntu3

---------------
dkms (2.3-3ubuntu3) artful; urgency=medium

  * 0014-remove-initrd-backup.patch: When removing a kernel also remove
    related .old-dkms file from /boot after its been created not before.
    (LP: #1515513)

 -- Brian Murray <email address hidden> Thu, 05 Oct 2017 13:10:42 -0700

This bug was fixed in the package dkms - 2.3-3ubuntu1.2

---------------
dkms (2.3-3ubuntu1.2) zesty; urgency=medium

  * 0014-remove-initrd-backup.patch: When removing a kernel also remove
    related .old-dkms file from /boot after its been created not before.
    (LP: #1515513)

 -- Brian Murray <email address hidden> Thu, 05 Oct 2017 11:45:16 -0700

The verification of the Stable Release Update for dkms has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package dkms - 2.2.0.3-2ubuntu11.5

---------------
dkms (2.2.0.3-2ubuntu11.5) xenial; urgency=medium

  * 0014-remove-initrd-backup.patch: When removing a kernel also remove
    related .old-dkms file from /boot after its been created not before.
    (LP: #1515513)

 -- Brian Murray <email address hidden> Thu, 05 Oct 2017 13:14:20 -0700