cp dumps core when copying >4GB files to a VFAT filesystem

Bug #75574 reported by JK
126
This bug affects 1 person
Affects Status Importance Assigned to Milestone
coreutils (Ubuntu)
Invalid
Undecided
Unassigned
linux (Ubuntu)
Fix Released
Undecided
Unassigned
linux-source-2.6.20 (Baltix)
Invalid
Undecided
Unassigned
linux-source-2.6.20 (Ubuntu)
Won't Fix
Low
Unassigned

Bug Description

ProblemType: Crash
CrashCounter: 1
Date: Wed Dec 13 00:43:24 2006
Dependencies:
 libsepol1 1.12-1
 libselinux1 1.30-1ubuntu1
 tzdata 2006m-1ubuntu1
 libattr1 2.4.32-1ubuntu1
 libacl1 2.2.39-1ubuntu2
 belocs-locales-bin 2.4-1ubuntu6
 libc6 2.4-1ubuntu12
 locales 2.3.22
Disassembly:
 (no debugging symbols found)
 Using host libthread_db library "/lib/tls/i686/cmov/libthread_db.so.1".
 (no debugging symbols found)
 Core was generated by `cp tiger.tar /media/IPOOP/'.
 Program terminated with signal 25, File size limit exceeded.

(Trimmed the rest of this extremely long crash report from out of the description: should've been attached rather than included inline; in any case, it is no longer useful, other than to indicate the fact that it was aborted with signal 25 (SIGXFSZ) --Micah Cowan)

Related branches

Revision history for this message
didier (did447-deactivatedaccount) wrote :

VFAT, USB and file > 2GB?

Same issue than bug #63900?

Changed in coreutils:
status: Unconfirmed → Needs Info
Revision history for this message
JK (jk4-deactivatedaccount) wrote : Re: [Bug 75574] Re: cp crashes when copying large files to usb device

my operation appears to have been much simpler than bug 63900. while
the older bug was using a combination of openssh and scp. i'm see
that scp depends on coreutils which may include cp. in conclusion,
whether the two bugs are related, i can not be certain. to me, as a
novice bug reporter, they appear different, but perhaps since 63900
rely on coreutils, like i said earlier, it may be related.

On 12/13/06, didier <email address hidden> wrote:
> VFAT, USB and file > 2GB?
>
> Same issue than bug #63900?
>
> ** Changed in: coreutils (Ubuntu)
> Status: Unconfirmed => Needs Info
>
> --
> cp crashes when copying large files to usb device
> https://launchpad.net/bugs/75574
>

Revision history for this message
edschofield (schofield) wrote :

This bug is simple to reproduce: create a large (>4GB) tar file and copy it to a VFAT partition (e.g. an iPod). cp dumps core.

Preferably cp should not waste the user's time copying only the first 4GB of the file before raising an error, but should check in advance whether the file size will be exceeded before it starts copying. It should never be necessary to override this behaviour, since a filesystem's file size limits are static (unlike, for example, its free space).

Revision history for this message
Micah Cowan (micahcowan) wrote :

cp dies with SIGXFSZ, a signal from BSD that is generated when the device's file-size limit is exceeded, and whose default behavior is to dump core(!).

There may be issues on some filesystems where it can be difficult to determine the true file-size limit, and even after such a check, there is a race condition. Whether or not such a check is performed, however, it should be a simple matter to install a handler for SIGXFSZ.

Note that mv (others? prolly dd...) also has this issue (see duplicates).

Revision history for this message
Nigel Cunningham (nigelc) wrote :

Someone wrongly marked 103204 as a dupicate of this. How do you undo that?

Micah Cowan (micahcowan)
Changed in coreutils:
status: Needs Info → Confirmed
Revision history for this message
Micah Cowan (micahcowan) wrote :

Bug 63900 has some discussion regarding blocking/handling SIGXFSZ. I'm convinced that handling this signal is the right way to fix it; patching vfat to not generate the signal isn't appropriate, as other filesystems may still cause this.

IMO, SIGXFSZ should never have been given a default behavior to dump core; but now that that's been established (in BSD?), we can't do much about that. :-/ The commentor on bug 63900 is right that it's stupid to expect the writers of all tools to deal with this signal (grossly paraphrased), but we don't really have a choice now. :(

Revision history for this message
Micah Cowan (micahcowan) wrote :

To my dismay, SIGXFSZ is a POSIX-defined signal; and because the "cp" command is specified as having the "Default" behavior with regard to asynchronous events (signals), it will break POSIX to install a signal handler for this by default.

In my opinion, it would be well to diverge from POSIX in this respect, and provide the signal handler by default, except when POSIXLY_CORRECT has been specified.

Micah Cowan (micahcowan)
Changed in linux-source-2.6.20:
status: Unconfirmed → Confirmed
Revision history for this message
Micah Cowan (micahcowan) wrote :

After discussion with the coreutils guys: ( http://lists.gnu.org/archive/html/bug-coreutils/2007-04/msg00070.html ), and double-checking SUSv3, I believe I was mistaken in my belief that the standard allows this behavior. if you look at http://www.opengroup.org/onlinepubs/009695399/functions/pwrite.html, you'll see:

«If a write() requests that more bytes be written than there is room for (for example, [XSI] [Option Start] the process' file size limit or [Option End] the physical end of a medium), only as many bytes as there is room for shall be written.... [t]he next write of a non-zero number of bytes would give a failure return (except as noted below).»

Included in the exceptions "noted below" is the generation of SIGXFSZ when a write would exceed the "soft file size limit for the process". Nothing is said about hard, physical or file-system-based limits; therefore, there is clearly no license for write() to do anything other than to return -1 and set an error condition, EFBAD in this case.

So, I'm confirming for the latest kernel (confirmed for latest in feisty, via dd, and explicitly checked to ensure that usage limits are set to unlimited for file size), and rejecting for coreutils.

Changed in coreutils:
status: Confirmed → Rejected
Revision history for this message
Micah Cowan (micahcowan) wrote :

To be clear: it's not a vfat issue: vfat just determines where the file limit is, but as didier comments in bug 63900, the actual generation of SIGXFSZ upon /encountering/ file size limits is a default thing, and is found in mm/filemap.c (the one generated for exceeding rlimits is SUS-mandated and should remain: IMO, the others should be removed, leaving just the return of -EFBIG).

Revision history for this message
Micah Cowan (micahcowan) wrote :
Revision history for this message
Micah Cowan (micahcowan) wrote :

Posted to the kernel mailing list, which so far has a response from Alan Cox, which indicates that (1) at the time it was written, this behavior was mandated by standards from the Large File Summit, and so was correct, and (2) those standards may have been "subsumed" by SUSv3, so patches adjusting behavior to match that just might be accepted. :)

The previous patch doesn't completely address the problem (for instance, old ReiserFS 3.5 objects on a 3.6+ system). I'll do a little more thorough check for the problem, and submit a new patch.

Kernel mailing list thread: http://groups.google.com/group/fa.linux.kernel/browse_thread/thread/5bee5c7f7405233e/b8e06b88a8520973

Revision history for this message
Micah Cowan (micahcowan) wrote :

How the heck did I choose Baltix by accident? *sigh*...

Changed in linux-source-2.6.20:
assignee: nobody → micah-cowan
status: Confirmed → Rejected
assignee: micah-cowan → nobody
assignee: nobody → micah-cowan
status: Unconfirmed → Confirmed
Micah Cowan (micahcowan)
Changed in linux-source-2.6.20:
status: Confirmed → In Progress
Revision history for this message
edschofield (schofield) wrote :

Well Done, Micah! I'm glad this is in capable hands :)

Micah Cowan (micahcowan)
Changed in linux-source-2.6.20:
importance: Undecided → Low
Revision history for this message
Micah Cowan (micahcowan) wrote :

Patch has been submitted to Ubuntu's kernel-team: https://lists.ubuntu.com/archives/kernel-team/2007-May/001487.html

Revision history for this message
Micah Cowan (micahcowan) wrote :

Patch also sent to linux-kernel development mailing list: http://lkml.org/lkml/2007/5/23/146

Revision history for this message
edschofield (schofield) wrote : Re: [Bug 75574] Re: cp dumps core when copying >4GB files to a VFAT filesystem

On 5/23/07, Micah Cowan <email address hidden> wrote:
> Patch also sent to linux-kernel development mailing list:
> http://lkml.org/lkml/2007/5/23/146

Thank you! Well done!

Micah Cowan (micahcowan)
description: updated
Revision history for this message
Micah Cowan (micahcowan) wrote :

This patch is now in the mainline linux-2.6 tree: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=17973f5af741f1758ed57c5115ca394c22bee159

I'm hoping this means that the Ubuntu Kernel Team will be willing to give it the go-ahead for Gutsy.

Revision history for this message
edschofield (schofield) wrote :

Good news!

Revision history for this message
Yann Rouillard (yann-pleiades) wrote :

The patch wasn't applied in gutsy kernel. This bug will wait the next ubuntu release to be fixed.

Revision history for this message
zeddock (zeddock) wrote :

The low importance of this problem should be raised to high. Here is my argument:

There is an influx of new users coming into Linux Ubuntu from MS Windows. These users have users of training in certain processes for computer use. Fortunately one of these things is the need for good backups.

Although there is not yet an Open Source backup utility out there that makes them feel warm and fuzzy, there are a few ways to get a good backup.

One option for backups is tar. There are several posts that a new user will inevitably end up which tells them how to use tar to make a good backup. And all along the way this new user is told how simple and dependable this is.

Everything is great until they go to copy the tar, bz, tzg, zip, etc... for when the filesize goes beyond 4gig, and I assure you that will happenn VERY fast as they bring more and more stuff over from there old MS OS, the file gets truncated!!

The worse part is that often there is not much warning that it has happened! So the user depends on a new external/usb HD to be making backups to and usually doesn't know there is a problem until they go to restore. Error: End Of File!!!! Did I mention that most usb/external drives that they will be buying are formatted to FAT32 or 16??

Anyway, please reconsider this priority and kick it up to high. In the mean time, is there a noob way for users to patch if they need this option?

Thanx!

zeddock

Revision history for this message
Tommy Trussell (tommy-trussell) wrote :

> In the mean time, is there a noob way for users to patch if they need this option?

The "patch" is to format the storage device using a different file format, such as ext2 or ext3. VFAT cannot handle files larger than 4GB. The linux and gnome utilities need to handle the situation more gracefully, HOWEVER I believe the underlying size restriction is a limitation of the file system.

Revision history for this message
zeddock (zeddock) wrote :

Understood. Thanx for the clarification.
zeddock

On Dec 13, 2007 1:39 PM, Tommy Trussell <email address hidden> wrote:

> > In the mean time, is there a noob way for users to patch if they need
> this option?
>
> The "patch" is to format the storage device using a different file
> format, such as ext2 or ext3. VFAT cannot handle files larger than 4GB.
> The linux and gnome utilities need to handle the situation more
> gracefully, HOWEVER I believe the underlying size restriction is a
> limitation of the file system.
>
> --
> cp dumps core when copying >4GB files to a VFAT filesystem
> https://bugs.launchpad.net/bugs/75574
> You received this bug notification because you are a direct subscriber
> of the bug.
>

Micah Cowan (micahcowan)
Changed in linux-source-2.6.20:
assignee: micahcowan → nobody
Revision history for this message
Leann Ogasawara (leannogasawara) wrote :

Hi Everyone,

The Hardy Heron Alpha series is currently under development and contains an updated version of the kernel. It would be helpful if you could test the latest Hardy Alpha release: http://www.ubuntu.com/testing . You should be able to then test the new kernel via the LiveCD. If you can, please verify if this bug still exists or not and report back your results. We'll keep this report open against the actively developed kernel but against 2.6.20 this will be closed. Thanks.

Changed in linux:
status: New → Incomplete
Changed in linux-source-2.6.20:
status: In Progress → Won't Fix
Revision history for this message
Leann Ogasawara (leannogasawara) wrote :

The Ubuntu Kernel Team is planning to move to the 2.6.27 kernel for the upcoming Intrepid Ibex 8.10 release. As a result, the kernel team would appreciate it if you could please test this newer 2.6.27 Ubuntu kernel. There are one of two ways you should be able to test:

1) If you are comfortable installing packages on your own, the linux-image-2.6.27-* package is currently available for you to install and test.

--or--

2) The upcoming Alpha5 for Intrepid Ibex 8.10 will contain this newer 2.6.27 Ubuntu kernel. Alpha5 is set to be released Thursday Sept 4. Please watch http://www.ubuntu.com/testing for Alpha5 to be announced. You should then be able to test via a LiveCD.

Please let us know immediately if this newer 2.6.27 kernel resolves the bug reported here or if the issue remains. More importantly, please open a new bug report for each new bug/regression introduced by the 2.6.27 kernel and tag the bug report with 'linux-2.6.27'. Also, please specifically note if the issue does or does not appear in the 2.6.26 kernel. Thanks again, we really appreicate your help and feedback.

Revision history for this message
dharmax (dharmax) wrote :

Hi,

unfortunately i can't do that. Currently, i can't "play" with my OS -
the computer has to stay very stable...

Leann Ogasawara wrote:

> The Ubuntu Kernel Team is planning to move to the 2.6.27 kernel for the
> upcoming Intrepid Ibex 8.10 release. As a result, the kernel team would
> appreciate it if you could please test this newer 2.6.27 Ubuntu kernel.
> There are one of two ways you should be able to test:
>
> 1) If you are comfortable installing packages on your own, the linux-
> image-2.6.27-* package is currently available for you to install and
> test.
>
> --or--
>
> 2) The upcoming Alpha5 for Intrepid Ibex 8.10 will contain this newer
> 2.6.27 Ubuntu kernel. Alpha5 is set to be released Thursday Sept 4.
> Please watch http://www.ubuntu.com/testing for Alpha5 to be announced.
> You should then be able to test via a LiveCD.
>
> Please let us know immediately if this newer 2.6.27 kernel resolves the
> bug reported here or if the issue remains. More importantly, please
> open a new bug report for each new bug/regression introduced by the
> 2.6.27 kernel and tag the bug report with 'linux-2.6.27'. Also, please
> specifically note if the issue does or does not appear in the 2.6.26
> kernel. Thanks again, we really appreicate your help and feedback.
>
> ** Tags added: cft-2.6.27
>
>

Revision history for this message
didier (did447-deactivatedaccount) wrote :

Hi

It's fixed at least since 2.6.24. I can't test with newer kernel.

Revision history for this message
dharmax (dharmax) wrote :

great work :)

didier wrote:

> Hi
>
> It's fixed at least since 2.6.24. I can't test with newer kernel.
>
>

Revision history for this message
Leann Ogasawara (leannogasawara) wrote :

Based on didier's comment that this was resolved back with 2.6.24, it should not have regressed with 2.6.27 so I'm marking this "Fix Released" for now. Thanks.

Changed in linux:
status: Incomplete → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.