fsck.ext4 fails to fix multiply-claimed blocks: can't find dup_blk

Can you send me the output of the following:

debugfs /dev/md/1
debugfs: stat <27528105>
debugfs: stat <27528089>
debugfs: quit

Thanks!!

(Keep in mind that I'm running debugfs after having deleted the file at inode 27528089 using rm.)

debugfs: stat <27528105>

Inode: 27528105 Type: regular Mode: 0600 Flags: 0x80000
Generation: 2927505238 Version: 0x00000000:00000001
User: 1000 Group: 1000 Size: 28672
File ACL: 0 Directory ACL: 0
Links: 2 Blockcount: 64
Fragment: Address: 0 Number: 0 Size: 0
 ctime: 0x52d35460:6c6ed008 -- Sun Jan 12 18:50:08 2014
 atime: 0x537a59e3:7b6818f8 -- Mon May 19 12:22:11 2014
 mtime: 0x52d35460:6c6ed008 -- Sun Jan 12 18:50:08 2014
crtime: 0x4e05aecd:1ddc9980 -- Sat Jun 25 02:47:57 2011
Size of extra inode fields: 28
EXTENTS:
(ETB0):73467268, (0):9280, (1):9281, (2):9282, (3):9283, (4):9284, (5):9285, (6):9286

stat <27528089>

Inode: 27528089 Type: regular Mode: 0600 Flags: 0x80000
Generation: 1720399890 Version: 0x00000000:00000001
User: 1000 Group: 1000 Size: 42789
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 88
Fragment: Address: 0 Number: 0 Size: 0
 ctime: 0x537ba775:61b6f090 -- Tue May 20 12:05:25 2014
 atime: 0x537ba763:3cca25fc -- Tue May 20 12:05:07 2014
 mtime: 0x537ba6bf:bec9095c -- Tue May 20 12:02:23 2014
crtime: 0x537ba6bf:bec9095c -- Tue May 20 12:02:23 2014
Size of extra inode fields: 28
EXTENTS:
(0-10):110330351-110330361

Right, I didn't notice the first time I read the bug report you had already deleted the inode to fix the problem. It would have been nice to have gotten a look at the inode in question so I could really see what was going on.

The "internal error: can't find dup_blk" error is one of those "this should never happen" situations, so it would be good to understand how and why it happened, and hopefully figure out how to reproduce it.

Can you send me the output of dumpe2fs -h /dev/md/1, just for the record?

Thanks!!

Yeah, sorry about that. If I had known you were likely to respond so quickly, I might have left it alone until I heard from you.

dumpe2fs 1.42.8 (20-Jun-2013)
Filesystem volume name: root
Last mounted on: /
Filesystem UUID: 3dd3e793-f873-4db2-8c68-801b217ba06e
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: (none)
Filesystem state: clean
Errors behavior: Remount read-only
Filesystem OS type: Linux
Inode count: 30285824
Block count: 121119454
Reserved block count: 2422389
Free blocks: 14803241
Free inodes: 29247104
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 995
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
Flex block group size: 16
Filesystem created: Fri Jun 24 17:47:00 2011
Last mount time: Tue May 20 12:03:32 2014
Last write time: Tue May 20 12:03:31 2014
Mount count: 2
Maximum mount count: -1
Last checked: Tue May 20 11:10:00 2014
Check interval: 2419200 (4 weeks)
Next check after: Tue Jun 17 11:10:00 2014
Lifetime writes: 6072 GB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
First orphan inode: 27269186
Default directory hash: half_md4
Directory Hash Seed: 814029bb-cac6-432f-9a6b-a7924a5e8f4a
Journal backup: inode blocks
Journal features: journal_incompat_revoke
Journal size: 128M
Journal length: 32768
Journal sequence: 0x00b205bb
Journal start: 237

Thanks, I was able to reproduce the problem using a test file system. The problem appears to have been caused by an extent-mapped inode getting written (or copied) to a different/wrong place in the inode table. I should be able to get this fixed now that I have a simple repro.

That's great news. Thanks for the quick response!

Hi,
I had the same error message doing while fsck'ing one of my disks.

I did not understand well the cause of the problem. Is fsck what caused the file-system corruption? (because of an "extent-mapped inode getting written (or copied) to a different/wrong place in the inode table"?)

This ticket is labelled e2fsprogs. May you explain if the planned fix is:
- to avoid file-system corruption
- to enable fsck to fix it
(- something else?)
 please?

The file system which I posted in #6 no longer triggers a bug in 1.42.12. It looks like it was fixed as a side effect of commit 9a1d614df217. @bigeel, what version of e2fsck were you using? Could you try using e2fsprogs 1.42.12 and see if that fixes the problem for you?

Thanks!!

Hi Theodore,

thanks for your really fast answer. I do not know how to get fsck version number, but I had the following output:
  fsck from util-linux 2.20.1
  e2fsck 1.42 (29-Nov-2011)

However I have run `e2fsck f_dup5.img` on your attachment at #6, and no error was reported, so I do not know how if I can trust the "1.42 (29-Nov-2011)" part:
  e2fsck 1.42 (29-Nov-2011)
  f_dup5.img: clean, 13/16 files, 43/100 blocks

Anyway, I do not think I am willing to run fsck again as it takes 4 days to complete, and the data is not that important.

What I am more worried about is what could have caused it, and how serious this is. Do you have a clue?
Would deleting (or cp file bak; rm file; mv bak file) solve the problem?
(at least I have done some cp/rm and the number of errors decreased, but I wonder whether it’s just shallow or the real problem is gone)

Best

Hello, for all those interested..

I'll be providing the fix by backporting/cherry-picking commit 9a1d614df217 from upstream mentioned by Theo so we can create a SRU for this...

inaddy@xxx:~/sources/upstream/e2fsprogs$ git tag --contains 9a1d614df217
v1.42.12

From rmadison all affected versions are:

- Precise (1.42-1ubuntu2.2)
- Trusty (1.42.9-3ubuntu1.2)
- Utopic (1.42.10-1.1ubuntu1)

Vivid is ok (1.42.12-1ubuntu2)

I already cherry-picked the commit and I'm solving conflicts. Then I'll test the fsck with the image provided by Theo to see if it solves the issue and ask for SRU.

Thank you!

Rafael Tinoco

Okay, I have provided one PPA for precise for this particular case:

https://launchpad.net/~inaddy/+archive/ubuntu/lp1321418

Cherry-picking 2 needed patches from upstream.

I have tested also the "fsck" with the filesystem that Theo provided and version from PPA fixes the problems.

Second fsck run shows:

inaddy@xxx:~/filesystem$ fsck.ext4 -f ./f_dup5.img
e2fsck 1.42 (29-Nov-2011)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
./f_dup5.img: 13/16 files (15.4% non-contiguous), 65/100 blocks

No errors.

I'm waiting for feedback from some users to provide debdiffs to be proposed as SRU for Precise, Trusty and Utopic.

Thank you very much for reporting this.

Rafael Tinoco

Sponsored for P/T.

Hello Forest, or anyone else affected,

Accepted e2fsprogs into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/e2fsprogs/1.42.9-3ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Forest, or anyone else affected,

Accepted e2fsprogs into precise-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/e2fsprogs/1.42-1ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Chris, I have no way to test this, because I no longer have a filesystem with multiply-claimed blocks.

This verified fine on Precise and Trusty using the FS image from comment #6. Thank you

This bug was fixed in the package e2fsprogs - 1.42-1ubuntu2.3

---------------
e2fsprogs (1.42-1ubuntu2.3) precise; urgency=low

  * fix rule-violating lblk->pblk mappings on bigalloc filesystems (LP: #1321418)

 -- Seyeong Kim <email address hidden> Tue, 01 Sep 2015 10:57:56 -0500

The verification of the Stable Release Update for e2fsprogs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package e2fsprogs - 1.42.9-3ubuntu1.3

---------------
e2fsprogs (1.42.9-3ubuntu1.3) trusty; urgency=medium

  * fix rule-violating lblk->pblk mappings on bigalloc filesystems (LP: #1321418)

 -- Seyeong Kim <email address hidden> Tue, 01 Sep 2015 07:08:12 -0500