blk_update_request: I/O error

Bug #1526537 reported by Marcin
46
This bug affects 6 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Expired
Medium
Unassigned

Bug Description

On my laptop I am using the ubuntu in persistent mode with a backing files: casper-rw, home-rw. For some time in the new editions of ubuntu [ 15.04 / 15.10 / 16.04 daily (15.12) ] I meet with error: blk_update_request: I/O error, dev loop ... , sector ... . am sure it is not a hardware problem. Please, could you try reproduce this issue

`/proc/cmdline`

BOOT_IMAGE=/Ubuntu/16.04/desktop-amd64/casper/vmlinuz.efi append noprompt boot=casper intel_iommu=on toram showmounts persistent-path=/Ubuntu/16.04/desktop-amd64 persistent iso-scan/filename=/Ubuntu/16.04/xenial-desktop-amd64.iso cdrom-detect/try-usb=true

`kern.log`

Dec 15 21:49:25 ThinkPad kernel: [ 11.507021] Bluetooth: BNEP socket layer initialized
Dec 15 21:49:25 ThinkPad kernel: [ 11.513208] blk_update_request: I/O error, dev loop0, sector 1376280
Dec 15 21:49:25 ThinkPad kernel: [ 11.544996] blk_update_request: I/O error, dev loop0, sector 1364008
Dec 15 21:49:26 ThinkPad kernel: [ 11.655626] IPv6: ADDRCONF(NETDEV_UP): wlp3s0: link is not ready
Dec 15 21:49:26 ThinkPad kernel: [ 11.656563] IPv6: ADDRCONF(NETDEV_UP): enp0s25: link is not ready
Dec 15 21:49:26 ThinkPad kernel: [ 11.884604] IPv6: ADDRCONF(NETDEV_UP): enp0s25: link is not ready
Dec 15 21:49:26 ThinkPad kernel: [ 12.080950] usbcore: registered new interface driver cdc_wdm
Dec 15 21:49:26 ThinkPad kernel: [ 12.089352] qmi_wwan 2-1.4:1.0: cdc-wdm0: USB WDM device
Dec 15 21:49:26 ThinkPad kernel: [ 12.089629] qmi_wwan 2-1.4:1.0 wwan0: register 'qmi_wwan' at usb-0000:00:1d.0-1.4, WWAN/QMI device, 86:f3:f1:57:ac:04
Dec 15 21:49:26 ThinkPad kernel: [ 12.089851] usbcore: registered new interface driver qmi_wwan
Dec 15 21:49:27 ThinkPad kernel: [ 13.014113] blk_update_request: I/O error, dev loop2, sector 776360
Dec 15 21:49:27 ThinkPad kernel: [ 13.026856] blk_update_request: I/O error, dev loop2, sector 777792
Dec 15 21:49:27 ThinkPad kernel: [ 13.406901] e1000e: enp0s25 NIC Link is Up 100 Mbps Full Duplex, Flow Control: Rx/Tx
Dec 15 21:49:27 ThinkPad kernel: [ 13.406909] e1000e 0000:00:19.0 enp0s25: 10/100 speed: disabling TSO
Dec 15 21:49:27 ThinkPad kernel: [ 13.406948] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s25: link becomes ready
Dec 15 21:49:27 ThinkPad kernel: [ 13.411598] blk_update_request: I/O error, dev loop0, sector 1364024
Dec 15 21:49:27 ThinkPad kernel: [ 13.525480] blk_update_request: I/O error, dev loop2, sector 778064
Dec 15 21:49:28 ThinkPad kernel: [ 13.733896] Bluetooth: RFCOMM TTY layer initialized
Dec 15 21:49:28 ThinkPad kernel: [ 13.733906] Bluetooth: RFCOMM socket layer initialized
Dec 15 21:49:28 ThinkPad kernel: [ 13.733912] Bluetooth: RFCOMM ver 1.11
Dec 15 21:49:28 ThinkPad kernel: [ 13.758583] blk_update_request: I/O error, dev loop0, sector 1360048
Dec 15 21:49:28 ThinkPad kernel: [ 13.765377] blk_update_request: I/O error, dev loop0, sector 1364032
Dec 15 21:49:28 ThinkPad kernel: [ 13.778920] blk_update_request: I/O error, dev loop0, sector 1364040
Dec 15 21:49:28 ThinkPad kernel: [ 13.785037] blk_update_request: I/O error, dev loop0, sector 1364048
Dec 15 21:49:28 ThinkPad gnome-session-binary[3083]: Entering running state
Dec 15 21:49:37 ThinkPad kernel: [ 22.759620] blk_update_request: 15 callbacks suppressed
Dec 15 21:49:37 ThinkPad kernel: [ 22.759623] blk_update_request: I/O error, dev loop0, sector 1364072
Dec 15 21:49:37 ThinkPad kernel: [ 22.803618] hidraw: raw HID events driver (C) Jiri Kosina
Dec 15 21:49:37 ThinkPad kernel: [ 22.806751] Bluetooth: HIDP (Human Interface Emulation) ver 1.2
Dec 15 21:49:37 ThinkPad kernel: [ 22.806758] Bluetooth: HIDP socket layer initialized
Dec 15 21:49:37 ThinkPad kernel: [ 22.811095] hid-generic 0005:0D62:01BF.0001: unknown main item tag 0x0
Dec 15 21:49:37 ThinkPad kernel: [ 22.811211] input: Bluetooth Mouse as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.4/1-1.4:1.0/bluetooth/hci0/hci0:11/0005:0D62:01BF.0001/input/input17
Dec 15 21:49:37 ThinkPad kernel: [ 22.811384] hid-generic 0005:0D62:01BF.0001: input,hidraw0: BLUETOOTH HID v1.39 Mouse [ Bluetooth Mouse] on ec:55:f9:fa:c6:1d
Dec 15 21:50:41 ThinkPad kernel: [ 86.683594] blk_update_request: I/O error, dev loop2, sector 2343552
Dec 15 21:50:41 ThinkPad kernel: [ 86.690626] blk_update_request: I/O error, dev loop2, sector 2339064
Dec 15 21:50:41 ThinkPad kernel: [ 86.703606] blk_update_request: I/O error, dev loop2, sector 2057600
Dec 15 21:50:41 ThinkPad kernel: [ 86.713265] blk_update_request: I/O error, dev loop2, sector 2343592
Dec 15 21:50:41 ThinkPad kernel: [ 86.715556] blk_update_request: I/O error, dev loop2, sector 2057608
Dec 15 21:50:41 ThinkPad kernel: [ 86.861531] blk_update_request: I/O error, dev loop2, sector 2339104
Dec 15 21:50:42 ThinkPad kernel: [ 87.954169] blk_update_request: I/O error, dev loop2, sector 2343632
---
ApportVersion: 2.19.3-0ubuntu2
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: marcin 3226 F.... pulseaudio
CasperVersion: 1.366
DistroRelease: Ubuntu 16.04
LiveMediaBuild: Ubuntu 16.04 LTS "Xenial Xerus" - Alpha amd64 (20151202)
MachineType: LENOVO 4291PR5
Package: linux (not installed)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/Ubuntu/16.04/desktop-amd64/casper/vmlinuz.efi append noprompt boot=casper intel_iommu=on toram showmounts persistent-path=/Ubuntu/16.04/desktop-amd64 persistent userfullname=MarcinCelebucki username=marcin hostname=hostname iso-scan/filename=/Ubuntu/16.04/xenial-desktop-amd64.iso cdrom-detect/try-usb=true
ProcVersionSignature: Ubuntu 4.2.0-19.23-generic 4.2.6
PulseList:
 Error: command ['pacmd', 'list'] failed with exit code 1: Home directory not accessible: Permission denied
 No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-4.2.0-19-generic N/A
 linux-backports-modules-4.2.0-19-generic N/A
 linux-firmware 1.154
Tags: xenial
UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
Uname: Linux 4.2.0-19-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 05/14/2015
dmi.bios.vendor: LENOVO
dmi.bios.version: 8DET70WW (1.40 )
dmi.board.asset.tag: Not Available
dmi.board.name: 4291PR5
dmi.board.vendor: LENOVO
dmi.board.version: Not Available
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvr8DET70WW(1.40):bd05/14/2015:svnLENOVO:pn4291PR5:pvrThinkPadX220:rvnLENOVO:rn4291PR5:rvrNotAvailable:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 4291PR5
dmi.product.version: ThinkPad X220
dmi.sys.vendor: LENOVO

Marcin (marcincelebucki)
affects: lvm2 (Ubuntu) → linux-meta (Ubuntu)
Brad Figg (brad-figg)
affects: linux-meta (Ubuntu) → linux (Ubuntu)
Marcin (marcincelebucki)
tags: added: wily
Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1526537

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.4 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4-rc5-wily

Changed in linux (Ubuntu):
importance: Undecided → Medium
Revision history for this message
Marcin (marcincelebucki) wrote : AlsaInfo.txt

apport information

tags: added: apport-collected xenial
description: updated
Revision history for this message
Marcin (marcincelebucki) wrote : CRDA.txt

apport information

Revision history for this message
Marcin (marcincelebucki) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Marcin (marcincelebucki) wrote : IwConfig.txt

apport information

Revision history for this message
Marcin (marcincelebucki) wrote : JournalErrors.txt

apport information

Revision history for this message
Marcin (marcincelebucki) wrote : Lspci.txt

apport information

Revision history for this message
Marcin (marcincelebucki) wrote : Lsusb.txt

apport information

Revision history for this message
Marcin (marcincelebucki) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Marcin (marcincelebucki) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Marcin (marcincelebucki) wrote : ProcModules.txt

apport information

Revision history for this message
Marcin (marcincelebucki) wrote : RfKill.txt

apport information

Revision history for this message
Marcin (marcincelebucki) wrote : UdevDb.txt

apport information

Revision history for this message
Marcin (marcincelebucki) wrote : WifiSyslog.txt

apport information

Revision history for this message
Marcin (marcincelebucki) wrote :

Joseph, I encounter this problem also in ubuntu 15.04 with a linux-image-3.19.0-15-generic.

I can append some additional information how to reproduce this issue. After my analysis I found dependence: drive filesystem vs backing file filesystem:

Create backing file on a drive that is formatted to NTFS
dd if=/dev/zero of=./Test.img bs=1M count=1024

Find free loop device
losetup -f

Connect backing file with a loop device [ losetup -f ], for example: /dev/loop3
losetup /dev/loop3 ./Test.img

Create filesystem on backing file via loop device
mkfs.ext4 /dev/loop3

Mount backing file via loop device into /mnt directory
mount -t ext4 /dev/loop3 /mnt/

Write some data on backing file via loop device
dd if=/dev/zero of=/mnt/Test.file bs=1M count=512

After this procedure you will be see errors on a kernel log:
[ 1370.381087] blk_update_request: I/O error, dev loop3, sector 25864
[ 1370.789277] blk_update_request: I/O error, dev loop3, sector 29960
[ 1371.193360] blk_update_request: I/O error, dev loop3, sector 34056

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
penalvch (penalvch)
tags: added: latest-bios-1.40
Revision history for this message
penalvch (penalvch) wrote :

Marcin, at your earliest convenience, could you please test the latest upstream kernel available from the very top line at the top of the page from http://kernel.ubuntu.com/~kernel-ppa/mainline/?C=N;O=D (the release names are irrelevant for testing, and please do not test the daily folder)? Install instructions are available at https://wiki.ubuntu.com/Kernel/MainlineBuilds . This will allow additional upstream developers to examine the issue.

If testing on your main install would be inconvenient, one may:
1) Install Ubuntu to a different partition and then test this there.
2) Backup, or clone the primary install.

If the latest kernel did not allow you to test to the issue (ex. you couldn't boot into the OS) please make a comment in your report about this, and continue to test the next most recent kernel version until you can test to the issue. Once you've tested the upstream kernel, please comment on which kernel version specifically you tested. If this issue is fixed in the mainline kernel, please add the following tags by clicking on the yellow circle with a black pencil icon, next to the word Tags, located at the bottom of the report description:
kernel-fixed-upstream
kernel-fixed-upstream-X.Y-rcZ

Where X, and Y are the first two numbers of the kernel version, and Z is the release candidate number if it exists.

If the mainline kernel does not fix the issue, please add the following tags:
kernel-bug-exists-upstream
kernel-bug-exists-upstream-X.Y-rcZ

Please note, an error to install the kernel does not fit the criteria of kernel-bug-exists-upstream.

Once testing of the latest upstream kernel is complete, please mark this report's Status as Confirmed. Please let us know your results.

Thank you for your understanding.

tags: added: vivid
Changed in linux (Ubuntu):
status: Confirmed → Incomplete
Marcin (marcincelebucki)
tags: added: kernel-bug-exists-4.3.0-2-generic
Marcin (marcincelebucki)
tags: added: kernel-bug-exists-4.4.0-040400rc5-generic
removed: apport-collected kernel-bug-exists-4.3.0-2-generic latest-bios-1.40 vivid wily xenial
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
tags: added: apport-collected kernel-bug-exists-upstream-4.3.0-2-generic latest-bios-1.40 vivid wily xenial
removed: kernel-bug-exists-4.4.0-040400rc5-generic
tags: added: kernel-bug-exists-upstream-4.4.0-040400rc5-generic
removed: kernel-bug-exists-upstream-4.3.0-2-generic
penalvch (penalvch)
tags: added: kernel-bug-exists-upstream kernel-bug-exists-upstream-4.4-rc5
removed: kernel-bug-exists-upstream-4.4.0-040400rc5-generic
Revision history for this message
penalvch (penalvch) wrote :

Marcin, was this issue not reproducible in a release prior to 15.04?

Changed in linux (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Marcin (marcincelebucki) wrote :

Yes, this issue not reproducible in a release prior to 15.04 [ ex. 14.04.3 ]

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
penalvch (penalvch) wrote :

Marcin, the next step is to fully commit bisect from kernel 3.13 to 4.2 in order to identify the last good kernel commit, followed immediately by the first bad one. This will allow for a more expedited analysis of the root cause of your issue. Could you please do this following https://wiki.ubuntu.com/Kernel/KernelBisection ?

Please note, finding adjacent kernel versions is not fully commit bisecting.

Thank you for your understanding.

Helpful bug reporting tips:
https://wiki.ubuntu.com/ReportingBugs

tags: added: regression-release
Changed in linux (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

Changed in linux (Ubuntu):
status: Incomplete → Expired
Revision history for this message
D. Charles Pyle (dcharlespyle) wrote :
Download full text (25.7 KiB)

Still seeing this in 4.4.0 kernels on Ubuntu 16.04.

$ dmesg | grep -i blk
[ 5.660121] systemd[1]: systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN)
[ 60.473635] blk_update_request: I/O error, dev loop0, sector 151271776
[ 64.663131] blk_update_request: I/O error, dev loop0, sector 193202168
[ 67.053226] blk_update_request: I/O error, dev loop0, sector 155758640
[ 67.119192] blk_update_request: I/O error, dev loop0, sector 155758648
[ 67.189707] blk_update_request: I/O error, dev loop0, sector 155758656
[ 67.353011] blk_update_request: I/O error, dev loop0, sector 154296456
[ 67.411390] blk_update_request: I/O error, dev loop0, sector 155758664
[ 67.469595] blk_update_request: I/O error, dev loop0, sector 154296464
[ 67.529317] blk_update_request: I/O error, dev loop0, sector 155758672
[ 67.586518] blk_update_request: I/O error, dev loop0, sector 155758680
[ 67.669838] blk_update_request: I/O error, dev loop0, sector 155758688
[ 67.746886] blk_update_request: I/O error, dev loop0, sector 155758696
[ 72.829706] blk_update_request: 8 callbacks suppressed
[ 72.829712] blk_update_request: I/O error, dev loop0, sector 155758768
[ 73.164340] blk_update_request: I/O error, dev loop0, sector 155758776
[ 73.672569] blk_update_request: I/O error, dev loop0, sector 155758784
[ 80.571864] blk_update_request: I/O error, dev loop0, sector 155758792
[ 80.660314] blk_update_request: I/O error, dev loop0, sector 67514472
[ 80.735506] blk_update_request: I/O error, dev loop0, sector 158408744
[ 80.789497] blk_update_request: I/O error, dev loop0, sector 67514480
[ 80.843609] blk_update_request: I/O error, dev loop0, sector 67514488
[ 80.898603] blk_update_request: I/O error, dev loop0, sector 67514496
[ 80.957007] blk_update_request: I/O error, dev loop0, sector 67514504
[ 99.165416] blk_update_request: I/O error, dev loop0, sector 158408720
[ 100.452488] blk_update_request: I/O error, dev loop0, sector 158409144
[ 100.452497] blk_update_request: I/O error, dev loop0, sector 158409184
[ 106.492835] blk_update_request: I/O error, dev loop0, sector 67515384
[ 107.409988] blk_update_request: I/O error, dev loop0, sector 155760968
[ 705.902224] blk_update_request: I/O error, dev loop0, sector 158410984
[ 707.433389] blk_update_request: I/O error, dev loop0, sector 158411328
[ 710.156894] blk_update_request: I/O error, dev loop0, sector 154299240
[ 711.846610] blk_update_request: I/O error, dev loop0, sector 158411440
[ 712.162845] blk_update_request: I/O error, dev loop0, sector 158411664
[ 713.730702] blk_update_request: I/O error, dev loop0, sector 158411680
[ 800.300872] blk_update_request: I/O error, dev loop0, sector 154299328
[ 880.661945] blk_update_request: I/O error, dev loop0, sector 155762080
[ 881.002646] blk_update_request: I/O error, dev loop0, sector 158411848
[ 883.469054] blk_update_request: I/O error, dev loop0, sector 158411968
[ 1489.772068] blk_update_request: I/O error, dev loop0, sector 158412376
[ 2018.027178] blk_update_request: I/O error, dev loop0...

Revision history for this message
hakuna_matata (hakuna-matata) wrote :

I can confirm that the error messages on Ubuntu 16.04 (daily) still exist but it seems that it is not as bad as it looks.

I followed the steps which Marcin described in comment #16. Instead of

> Write some data on backing file via loop device
> dd if=/dev/zero of=/mnt/Test.file bs=1M count=512

I copied a known iso file to the created loop device
dd if=lubuntu-14.04.4-desktop-amd64.iso of=/mnt/Test.file bs=1M

I got also some error messages as described in previous comments but md5sum said

md5sum lubuntu-14.04.4-desktop-amd64.iso && md5sum /mnt/Test.file

0b0fc8652afe0761edc3ed33e1b199f0 lubuntu-14.04.4-desktop-amd64.iso
0b0fc8652afe0761edc3ed33e1b199f0 /mnt/Test.file

Then I unmounted the test image and checked it for errors.

umount /mnt/
fsck -pvcf ./Test.img

...and I got:

fsck from util-linux 2.27.1
./Test.img: Updating bad block inode.

          12 inodes used (0.02%, out of 65536)
           0 non-contiguous files (0.0%)
           0 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 3/1
      201308 blocks used (76.79%, out of 262144)
           0 bad blocks
           1 large file

           1 regular file
           2 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           0 symbolic links (0 fast symbolic links)
           0 sockets
------------
           3 files

Revision history for this message
penalvch (penalvch) wrote :

D. Charles Pyle / hakuna_matata, it will help immensely if you filed a new report with the Ubuntu repository kernel (not mainline/upstream) via a terminal:
ubuntu-bug linux

Please feel free to subscribe me to it.

For more on why this is helpful, please see https://wiki.ubuntu.com/ReportingBugs.

Revision history for this message
Lancillotto (antonio-petricca) wrote :

I have the same problem on Ubuntu 16.04 and Kernel 4.10.0-38-generic .

Revision history for this message
penalvch (penalvch) wrote :

Lancillotto (antonio-petricca), it will help immensely if you filed a new report with Ubuntu via a terminal:
ubuntu-bug linux

Please feel free to subscribe me to it.

Revision history for this message
Lancillotto (antonio-petricca) wrote :

I solved my problem!

Recently I have published a guide on how to install linux on loopback LVM devices booting from USB (so without have to install grub on the internal disk, leaving it as original). Here is the guide: https://github.com/DareDevil73/linux-on-loopback-usb .

Then I fallen in the freeze issue on high memory load and I observed an abnormal swap space usage (all the RAM eaten, and swap usage near to zero). Obviously the LVM swap partition was mounted and working properly, but I don't known why the kernel did not use it as expected.

I tried an alternative solution. I have created a swap loopback file (not LVM) and the freeze is gone. Now the swap file is used as it would be, and the OS never freeze!

Please look at https://github.com/DareDevil73/linux-on-loopback-usb#known-issues to get deeper information.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.