Ubuntu

Upgrade from 17.10 to 18.04 LTS out of diskspace on /boot

Asked by Michael Mazzotta

Hi. I'm upgrading my servers from 17.10 to 18.04 LTS. However, when it gets to the kernel updates it runs out of disk space in /boot.
####
Processing triggers for ureadahead (0.100.0-20) ...
Processing triggers for initramfs-tools (0.130ubuntu3.3) ...
update-initramfs: Generating /boot/initrd.img-4.13.0-46-generic

gzip: stdout: No space left on device
E: mkinitramfs failure cpio 141 gzip 1
update-initramfs: failed for /boot/initrd.img-4.13.0-46-generic with 1.
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Processing triggers for linux-image-4.15.0-36-generic (4.15.0-36.39) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-4.15.0-36-generic
ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/initramfs-tools.0.crash'

gzip: stdout: No space left on device
E: mkinitramfs failure cpio 141 gzip 1
update-initramfs: failed for /boot/initrd.img-4.15.0-36-generic with 1.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
dpkg: error processing package linux-image-4.15.0-36-generic (--configure):
 installed linux-image-4.15.0-36-generic package post-installation script subprocess returned error exit status 1
Processing triggers for systemd (237-3ubuntu10.3) ...
Processing triggers for libapache2-mod-php7.2 (7.2.10-0ubuntu0.18.04.1) ...
Errors were encountered while processing:
 initramfs-tools
 linux-image-4.15.0-36-generic
ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/linux-image-4.15.0-36-generic.0.crash'
Exception during pm.DoInstall(): E:Sub-process /usr/bin/dpkg returned an error code (1)

*** Send problem report to the developers?
####

At this point I'm able to exit the installation. I manually removed the old kernels created during the update to free up more space in /boot. When I enter update commands there is nothing to update so it seems to be done. But I don't know what it was trying to finish.

This is what is left in /boot after the removal of older initrd.img-4.15.0-xx-generic that where added during the upgrade process.

# ls -ltr /boot
total 129610
drwx------ 2 root root 12288 Jun 1 2014 lost+found
-rw-r--r-- 1 root root 184840 Jan 28 2016 memtest86+_multiboot.bin
-rw-r--r-- 1 root root 184380 Jan 28 2016 memtest86+.elf
-rw-r--r-- 1 root root 182704 Jan 28 2016 memtest86+.bin
-rw------- 1 root root 7870224 Jun 11 19:25 vmlinuz-4.13.0-46-generic
-rw------- 1 root root 3884611 Jun 11 19:25 System.map-4.13.0-46-generic
-rw-r--r-- 1 root root 255 Jun 11 19:25 retpoline-4.13.0-46-generic
-rw-r--r-- 1 root root 213212 Jun 11 19:25 config-4.13.0-46-generic
-rw-r--r-- 1 root root 1501528 Jun 11 19:25 abi-4.13.0-46-generic
-rw------- 1 root root 4046393 Sep 24 10:08 System.map-4.15.0-36-generic
-rw-r--r-- 1 root root 0 Sep 24 10:08 retpoline-4.15.0-36-generic
-rw-r--r-- 1 root root 216954 Sep 24 10:08 config-4.15.0-36-generic
-rw-r--r-- 1 root root 1537821 Sep 24 10:08 abi-4.15.0-36-generic
-rw------- 1 root root 8277752 Sep 24 11:54 vmlinuz-4.15.0-36-generic
-rw-r--r-- 1 root root 50368264 Oct 2 20:14 initrd.img-4.13.0-46-generic
-rw-r--r-- 1 root root 53703477 Oct 2 20:14 initrd.img-4.15.0-36-generic
drwxr-xr-x 5 root root 1024 Oct 2 20:14 grub

I looks like it's a mix of old 4.13.0-46 and new 4.15.0-36 files. Did the upgrade complete?
The server is working but I rather it complete without a space issue. I don't have a way to expand /boot without a rebuild.
Is there a way to run do-release-upgrade without it updating the new kernels? The process does not autoremove them. If it just does one kernel then I can do the others after the upgrade is done.
Is there another option other than do-release-upgrade that gives me more control over the update process?

Thanks for the help.

Question information

Language:
English Edit question
Status:
Solved
For:
Ubuntu Edit question
Assignee:
No assignee Edit question
Solved by:
actionparsnip
Solved:
Last query:
Last reply:
Revision history for this message
actionparsnip (andrew-woodhead666) said : 		#1

What is the output of:

uname -a; dpkg -l | grep linux-image | grep -v head

Thanks

Revision history for this message
Michael Mazzotta (mazz44) said : 		#2

Sorry for the delay. I replied to the email and expected it would have updating this thread with my reply. Here is the data.

uname -a; dpkg -l | grep linux-image | grep -v head

Linux scratchubuntu 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

rc linux-image-3.13.0-106-generic 3.13.0-106.153 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-3.13.0-108-generic 3.13.0-108.155 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-3.13.0-115-generic 3.13.0-115.162 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-3.13.0-116-generic 3.13.0-116.163 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-3.13.0-117-generic 3.13.0-117.164 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-3.13.0-119-generic 3.13.0-119.166 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-3.13.0-121-generic 3.13.0-121.170 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-3.13.0-123-generic 3.13.0-123.172 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-3.13.0-125-generic 3.13.0-125.174 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-3.13.0-126-generic 3.13.0-126.175 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-3.13.0-128-generic 3.13.0-128.177 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-3.13.0-129-generic 3.13.0-129.178 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-3.13.0-132-generic 3.13.0-132.181 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-3.13.0-133-generic 3.13.0-133.182 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-3.13.0-24-generic 3.13.0-24.47 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-3.13.0-27-generic 3.13.0-27.50 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-3.13.0-30-generic 3.13.0-30.55 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-3.13.0-40-generic 3.13.0-40.69 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc linux-image-4.10.0-37-generic 4.10.0-37.41 amd64 Linux kernel image for version 4.10.0 on 64 bit x86 SMP
rc linux-image-4.10.0-38-generic 4.10.0-38.42 amd64 Linux kernel image for version 4.10.0 on 64 bit x86 SMP
rc linux-image-4.13.0-16-generic 4.13.0-16.19 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
rc linux-image-4.13.0-17-generic 4.13.0-17.20 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
rc linux-image-4.13.0-19-generic 4.13.0-19.22 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
rc linux-image-4.13.0-21-generic 4.13.0-21.24 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
rc linux-image-4.13.0-25-generic 4.13.0-25.29 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
ii linux-image-4.13.0-31-generic 4.13.0-31.34 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
rc linux-image-4.13.0-32-generic 4.13.0-32.35 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
rc linux-image-4.13.0-36-generic 4.13.0-36.40 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
ii linux-image-4.13.0-38-generic 4.13.0-38.43 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
rc linux-image-4.13.0-39-generic 4.13.0-39.44 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
rc linux-image-4.13.0-41-generic 4.13.0-41.46 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
ii linux-image-4.13.0-46-generic 4.13.0-46.51 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
ii linux-image-4.15.0-36-generic 4.15.0-36.39 amd64 Signed kernel image generic
rc linux-image-4.4.0-97-generic 4.4.0-97.120 amd64 Linux kernel image for version 4.4.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-106-generic 3.13.0-106.153 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-108-generic 3.13.0-108.155 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-115-generic 3.13.0-115.162 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-116-generic 3.13.0-116.163 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-117-generic 3.13.0-117.164 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-119-generic 3.13.0-119.166 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-121-generic 3.13.0-121.170 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-123-generic 3.13.0-123.172 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-125-generic 3.13.0-125.174 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-126-generic 3.13.0-126.175 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-128-generic 3.13.0-128.177 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-129-generic 3.13.0-129.178 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-132-generic 3.13.0-132.181 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-133-generic 3.13.0-133.182 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-24-generic 3.13.0-24.47 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-27-generic 3.13.0-27.50 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-30-generic 3.13.0-30.55 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-3.13.0-40-generic 3.13.0-40.69 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc linux-image-extra-4.10.0-37-generic 4.10.0-37.41 amd64 Linux kernel extra modules for version 4.10.0 on 64 bit x86 SMP
rc linux-image-extra-4.10.0-38-generic 4.10.0-38.42 amd64 Linux kernel extra modules for version 4.10.0 on 64 bit x86 SMP
rc linux-image-extra-4.13.0-16-generic 4.13.0-16.19 amd64 Linux kernel extra modules for version 4.13.0 on 64 bit x86 SMP
rc linux-image-extra-4.13.0-17-generic 4.13.0-17.20 amd64 Linux kernel extra modules for version 4.13.0 on 64 bit x86 SMP
rc linux-image-extra-4.13.0-19-generic 4.13.0-19.22 amd64 Linux kernel extra modules for version 4.13.0 on 64 bit x86 SMP
rc linux-image-extra-4.13.0-21-generic 4.13.0-21.24 amd64 Linux kernel extra modules for version 4.13.0 on 64 bit x86 SMP
rc linux-image-extra-4.13.0-25-generic 4.13.0-25.29 amd64 Linux kernel extra modules for version 4.13.0 on 64 bit x86 SMP
ii linux-image-extra-4.13.0-31-generic 4.13.0-31.34 amd64 Linux kernel extra modules for version 4.13.0 on 64 bit x86 SMP
rc linux-image-extra-4.13.0-32-generic 4.13.0-32.35 amd64 Linux kernel extra modules for version 4.13.0 on 64 bit x86 SMP
rc linux-image-extra-4.13.0-36-generic 4.13.0-36.40 amd64 Linux kernel extra modules for version 4.13.0 on 64 bit x86 SMP
ii linux-image-extra-4.13.0-38-generic 4.13.0-38.43 amd64 Linux kernel extra modules for version 4.13.0 on 64 bit x86 SMP
rc linux-image-extra-4.13.0-39-generic 4.13.0-39.44 amd64 Linux kernel extra modules for version 4.13.0 on 64 bit x86 SMP
rc linux-image-extra-4.13.0-41-generic 4.13.0-41.46 amd64 Linux kernel extra modules for version 4.13.0 on 64 bit x86 SMP
ii linux-image-extra-4.13.0-46-generic 4.13.0-46.51 amd64 Linux kernel extra modules for version 4.13.0 on 64 bit x86 SMP
rc linux-image-extra-4.4.0-97-generic 4.4.0-97.120 amd64 Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP
ii linux-image-generic 4.15.0.36.38 amd64 Generic Linux kernel image

Revision history for this message
actionparsnip (andrew-woodhead666) said : 		#3

Oh jesus, OK lets sort this mess out:

sudo dpkg -P `dpkg -l | grep ^rc | awk {'print $2'}`
sudo apt-get autoremove

Then please give the output of:

dpkg -l | grep linux-image | grep -v head | grep -v extra

Thanks

Revision history for this message
actionparsnip (andrew-woodhead666) said : 		#4

All the lines that start with 'rc' are removed but residual configuration exists outside of your home folder. The first command removes that we can then see the kernels that are really installed and remove the old ones to clear space in /boot

Revision history for this message
Michael Mazzotta (mazz44) said : 		#5

Looks cleaner. It was probably a mess since I was manually deleting image files since the system would not purge them when use autoremove or autoclean. Also, 'sudo apt-get autoremove' did not remove any files.

dpkg -l | grep linux-image | grep -v head | grep -v extra
ii linux-image-4.13.0-31-generic 4.13.0-31.34 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
ii linux-image-4.13.0-38-generic 4.13.0-38.43 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
ii linux-image-4.13.0-46-generic 4.13.0-46.51 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
ii linux-image-4.15.0-36-generic 4.15.0-36.39 amd64 Signed kernel image generic
ii linux-image-generic 4.15.0.36.38 amd64 Generic Linux kernel image

sudo apt-get autoremove
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.

The space used by boot does not seem to have changed.

ls -ltr /boot
total 129610
drwx------ 2 root root 12288 Jun 1 2014 lost+found
-rw-r--r-- 1 root root 184840 Jan 28 2016 memtest86+_multiboot.bin
-rw-r--r-- 1 root root 184380 Jan 28 2016 memtest86+.elf
-rw-r--r-- 1 root root 182704 Jan 28 2016 memtest86+.bin
-rw------- 1 root root 7870224 Jun 11 19:25 vmlinuz-4.13.0-46-generic
-rw------- 1 root root 3884611 Jun 11 19:25 System.map-4.13.0-46-generic
-rw-r--r-- 1 root root 255 Jun 11 19:25 retpoline-4.13.0-46-generic
-rw-r--r-- 1 root root 213212 Jun 11 19:25 config-4.13.0-46-generic
-rw-r--r-- 1 root root 1501528 Jun 11 19:25 abi-4.13.0-46-generic
-rw------- 1 root root 4046393 Sep 24 10:08 System.map-4.15.0-36-generic
-rw-r--r-- 1 root root 0 Sep 24 10:08 retpoline-4.15.0-36-generic
-rw-r--r-- 1 root root 216954 Sep 24 10:08 config-4.15.0-36-generic
-rw-r--r-- 1 root root 1537821 Sep 24 10:08 abi-4.15.0-36-generic
-rw------- 1 root root 8277752 Sep 24 11:54 vmlinuz-4.15.0-36-generic
-rw-r--r-- 1 root root 50368264 Oct 2 20:14 initrd.img-4.13.0-46-generic
-rw-r--r-- 1 root root 53703477 Oct 2 20:14 initrd.img-4.15.0-36-generic
drwxr-xr-x 5 root root 1024 Oct 2 20:14 grub

I'm running this from a snapshot so I can revert the server back before the upgrade. Will using the sudo dpkg -P `dpkg -l | grep ^rc | awk {'print $2'}` help with the space issue? Before the upgrade, I cleared out /boot so it had 22% used but that was still not enough space.

Thanks.

Revision history for this message
Best actionparsnip (andrew-woodhead666) said : 		#6

sudo apt-get --purge remove linux-image-4.13.0-31-generic
sudo apt-get --purge remove linux-image-4.13.0-38-generic
sudo apt-get --purge autoremove

Should give you enough space

Revision history for this message
actionparsnip (andrew-woodhead666) said : 		#7

Purging the old config is good, it also makes your package list easier to read because it will only show installed packages rather than the ones with fluff left behind.

Nice that you have snapshots. Bit of insurance is always good.

Revision history for this message
Manfred Hampl (m-hampl) said : 		#8

What is the size of your /boot partition?

df -h | grep boot

It seems to me that on your system the size of initrd.img-* is considerably larger than what its size is on other systems.
On your system the kernel packages in /boot for a certain version need almost 70 MB space.

Revision history for this message
Michael Mazzotta (mazz44) said : 		#9

I ran the purge commands and it gave a FATAL message looking for the System.map file.

# sudo apt-get --purge remove linux-image-4.13.0-31-generic
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  linux-headers-4.13.0-31 linux-headers-4.13.0-31-generic
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  linux-image-4.13.0-31-generic* linux-image-extra-4.13.0-31-generic*
0 upgraded, 0 newly installed, 2 to remove and 3 not upgraded.
After this operation, 235 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 218296 files and directories currently installed.)
Removing linux-image-extra-4.13.0-31-generic (4.13.0-31.34) ...
depmod: FATAL: could not load /boot/System.map-4.13.0-31-generic: No such file or directory

Then this command did find 4.13.0-31 files to remove
# sudo apt-get --purge autoremove

# df -h | grep boot
/dev/sda1 236M 136M 88M 61% /boot

I created the servers with 14.04 LTS. I don't remember selecting a size for the /boot partition.

I'm going to revert the server back to 17.10 and use what you shared for commands to see if it help clean up enough of the kernels so it does not run out of space during the upgrade. It will take at least an hour to run through it.

Revision history for this message
Michael Mazzotta (mazz44) said : 		#10

All the pre-clean commands solved the issue. When I ran the update it did not run into space problems with /boot. Thank you.

Revision history for this message
Michael Mazzotta (mazz44) said : 		#11

Thanks actionparsnip, that solved my question.