PHP5 under Apache2 on 64 bit system is not completely 64 bit

Was php5 compiled with -D_LARGEFILE_SOURCE and -D_FILE_OFFSET_BITS=64 on 64 bit systems?

Please provide a sample php script so I can try to replicate this.

Regards
chuck

Test file opened by the attached script, test.php, was 4,700,110,848 bytes in length. Under Ubuntu 10.04 beta 2, download socket was closed after transfering 405,143,552 bytes. (a 0xFFFFFFFF mask of the full file size).

Test needs to run on 64 bit system in order to properly work on files larger than 0xFFFFFFFF bytes in size.

Script works when run under Ubuntu 8.04 LTS (with updates) and fails when run under Ubuntu 10.04 beta 2 (with updates), both [amd64] bit Ubuntu based systems (running Intel CPUs).

An example (public) large file is: http://cdimage.debian.org/debian-cd/5.0.4/amd64/iso-dvd/debian-504-amd64-DVD-1.iso .

Bug also exists in current Ubuntu 9.10 release.

Tested using the test.php script previously attached, and using the suggested debian dvd .iso image as the input file (listed as test.txt in test.php script). .iso image is 4,697,126,912 bytes. Download socket is closed after 402,159,616 bytes, which again is the file size & 0xFFFFFFFF.

Attached is the apport report for php under Ubuntu 9.10, currrent release, latest updates.

Attached requested script to reproduce bug.

Bug does not exist in Debian 5.04 release (with updates).

I have confirmed this.

A 64-bit Lucid Apache server hosting a file of size

4697126912 bytes

and a client (happens to be Lucid also) downloading the file via the above test.php script only gets

402159616 bytes

I sent this bug upstream.

http://bugs.php.net/bug.php?id=52102

This turned out to be a duplicate of

http://bugs.php.net/bug.php?id=44522

Submitted: 2008-03-24 18:21 UTC

Actually, http://bugs.php.net/bug.php?id=44522 is concerned with uploading data to the server, which according to 44522 breaks at the 2G byte mark (a signed 32 bit int issue). This bug (564920, php # 52102) is an issue with downloading data from the server, which breaks at the 4G byte mark. Both are 32 bit vs 64 bit integer issues. So, this is not exactly a duplicate bug.

Peter:

Can you look at the patches to http://bugs.php.net/bug.php?id=44522 and see if we can apply them and test the results to see if that fixes the download issue?

Yes, we are doing just that.

Can you try the version in my ppa when its built?

thanks
chuck

Ok. I'm back to trying to test this fix. I think I have lucid-proposed added to my test configuration. I'm not seeing the build. A pointer to a web page documenting what I need to do next to access your "ppa" area and use it would be useful.

Thanks.

@R. Jones

Go to

https://launchpad.net/~zulcss/+archive/ppa

and follow instructions under "Adding this PPA to your system".

Install

php5 5.3.2-1ubuntu4.3

I tested this myself and the results were unsatisfactory (Apache error after about 10% download of the test ISO file). Can you confirm the same?

FYI, this is probably what caused the regression, but reverting it wouldn't be a proper fix:

http://svn.php.net/viewvc?view=revision&revision=280678

Here a patch that should fix this. It will stop php from mmaping large files. It may not be the approach upstream would like.

Marc, 4*1024*1024 (4MB) seems rather arbitrary. An ini setting would be a natural choice there. Upstream may or may not like it, but I think limiting mmap to a certain size actually makes sense.

I haven't tried the patch yet, does it solve the issue?

Well, it's not completely arbitrary. It's the size that was used previous to php 5.2.10. See _php_stream_mmap_range() in main/streams/mmap.c.

In my limited testing, it does solve the issue. Unfortunately, it also has the side effect of re-introducing php bug #48309 for files over 4MB.

Maybe making it size_t - 1 would be better?

So this is caused by a regression by the fix for php bug #48309. Marc patch does work but the side effect of the fix is that it re-introduces the same bug, so we need to find a better fix for bug #48309.

The plan going forward is the following:

- Make a PHP version with the patch available in a PPA.
- Ask for testers
- Make sure there isnt any more regressions
- Get it fixed in maverick
- Get it fixed for lucid through an SRU.
- Let upstream know about the regression and submit the patch upstream.

If you have any questions please let me know.

chuck

Can you try the version in ppa:zulcss/server-lucid-bug-fixes ?

thanks
chuck

Is ppa:zulcss/server-lucid-bug-fixes different from what is pointed at by https://launchpad.net/~zulcss/+archive/ppa ? The later is still showing a 2010/06/28 build date which is broken for trying to run PHP scripts under apache2. It is missing "libapache2-mod-php5{a} and php5-intl{a} from the build. aptitude then removes the existing libapache2-mod-php5 module and leaves the system broken for running PHP scripts under apache.

I'm not familiar enough with the ppa configuration to see a later php5 update.

And the documented script for adding ppa's to Ubuntu 10.04 (add-apt-repository ) is broken for access through firewalls.

Hi,

You can install the PPA manually as documented at https://help.launchpad.net/Packaging/PPA/InstallingSoftware, see the section "On older (pre 9.10) Ubuntu systems". After following the steps you should be able to install it without a problem.

chuck

I tested this and confirmed that the entire test ISO was properly downloaded.

I've now had a chance to install the PHP5 build from ppa:zulcss/server-lucid-bug-fixes. With that build, I was able to download a file > 2^32 bytes (over the 4.2Gb boundary) successfully. I still need to test the scripts I have running under Apache2 / PHP5 / Mysql 5.0 from Ubuntu 8.04 LTS to see if they continue to run under a Ubuntu 10.04 LTS configuration.

Installing the version of PHP5 from ppa:zulcss/server-lucid-bug-fixes using aptitude resulted in uninstalling a number of items (such as autoconf, automake, autotools-dev, libssl-dev, libtools, m4, shtool, zlib_g-dev), so it looks like there are some additional regression builds that will be needed to make the package system happy. Prior to the PHP5 install, the system was running Ubuntu desktop 10.04 LTS with the latest public standard production updates. The system had the previous PHP5 build installed and then backed out to the standard configuration prior to this update.

attaching debdiff of same thing as linked merge proposal

This bug was fixed in the package php5 - 5.3.3-1ubuntu3

---------------
php5 (5.3.3-1ubuntu3) maverick; urgency=low

  * debian/patches/lp564920-fix-big-files.patch: Fix downloading of large
    files (LP: #564920)
 -- Clint Byrum <email address hidden> Fri, 06 Aug 2010 13:10:17 -0700

Regarding the SRU to lucid:
https://code.launchpad.net/~clint-fewbar/ubuntu/lucid/php5/lucid-sru-lp564920/+merge/32803

IMPACT:

This bug fix is intended to make it possible to access/download large files. Users who attempt it now are given a file of the wrong size (silently) as the pointer appears to wrap.

DEV RELEASE STATUS(maverick):

This has been fixed in version 5.3.3-1ubuntu3 in Maverick

TEST CASE:

Please see https://bugs.launchpad.net/ubuntu/+source/php5/+bug/564920/comments/4

REGRESSION:

This fix is known to regress the upstream bug fix to php bug #48309

http://bugs.php.net/bug.php?id=48309

However, that bug is far less critical than this one, as it can be worked around in php code, while this cannot.

Uploaded to lucid-proposed

Accepted php5 into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

FYI, on monday we will be releasing security updates for php that will have versioning newer than the package currently in -proposed. I will upload an updated package to -proposed once the security updates come out.

Marc:

Will the next version have the proper support for handling large files (> 2^32 bytes) in it?

Test case for this is listed above.

The security update coming out monday will not. Monday, a new package that does contain the fix will be uploaded to -proposed for the required testing waiting period.

The proposed fix (in lucid-proposed as 5.3.2-1ubuntu4.3.) corrects this bug. I have now tested version 5.3.2-1ubuntu4.3 and successfully downloaded a large file (> 2^32 bytes) via a dynamic PHP script on a 64 bit Ubuntu 10.04.1 based system.

The proposed fix (in lucid-proposed as 5.3.2-1ubuntu4.3.) corrects this bug. I have now tested version 5.3.2-1ubuntu4.3 and successfully downloaded a large file (> 2^32 bytes) via a dynamic PHP script on a 64 bit Ubuntu 10.04.1 based system.

Just to add details:

Test was with the PHP script ( https://bugs.edge.launchpad.net/ubuntu/+source/php5/+bug/564920/+attachment/1338515/+files/test.php ) listed in comment 3 of the bug report ( https://bugs.edge.launchpad.net/ubuntu/+source/php5/+bug/564920/comments/4 ). My large test file on one test system is 4697126912 bytes (4.4G) in size.

Do you need any further input from me at this time? Once the next proposed update comes out, do you need anything from me other than to re-run the test at that time and provide similar comments on the test results?

Thanks for testing it!

The packages that will get pushed to -proposed on monday will contain the same bug fix, on top of the security updates. Once the one week waiting period is up, they will get pushed to -updates. Since it's the same fix you've already tested, I don't think the SRU team will require more testing. Of course, feel free to test them and comment here if you'd like.

Marc,

Now that this has been verified, would it be possible to fold this fix into the security update, to avoid yet another upload?

I can officially move this version to -updates on Monday morning or Sunday if that would help.

Martin,

No problem, I will incorporate the fix into the security updates. No need to move the version currently in -proposed.

Thanks!

This bug was fixed in the package php5 - 5.3.2-1ubuntu4.5

---------------
php5 (5.3.2-1ubuntu4.5) lucid-security; urgency=low

  * SECURITY UPDATE: denial of service and possible memory corruption via
    negative size in HTTP chunked encoding stream
    - debian/patches/CVE-2010-1866.patch: prevent chunk_size from
      overflowing in ext/standard/filters.c.
    - CVE-2010-1866
  * SECURITY UPDATE: arbitrary code execution via empty SQL query
    - debian/patches/CVE-2010-1868.patch: use ecalloc instead of emalloc in
      ext/sqlite/sqlite.c.
    - CVE-2010-1868
  * SECURITY UPDATE: denial of service via fnmatch stack consumption
    - debian/patches/CVE-2010-1917.patch: limit size of pattern in
      ext/standard/file.c.
    - CVE-2010-1917
  * SECURITY UPDATE: arbitrary memory disclosure and possible code
    execution via phar extension
    - debian/patches/CVE-2010-2094.patch: use correct format string in
      ext/phar/dirstream.c, ext/phar/stream.c.
    - CVE-2010-2094
    - CVE-2010-2950
  * SECURITY UPDATE: sensitive information disclosure or arbitrary code
    execution via use-after-free in SplObjectStorage unserializer
    - debian/patches/CVE-2010-2225.patch: fix logic in
      ext/spl/spl_observer.c, ext/standard/{php_var.h,var_unserializer.*},
      add tests to ext/spl/tests.
    - CVE-2010-2225
  * SECURITY UPDATE: sensitive information disclosure via error messages
    - debian/patches/CVE-2010-2531.patch: don't display data when flushing
      output buffer in ext/standard/{var.c,php_var.h}, fix tests in
      ext/standard/tests/general_functions.
    - CVE-2010-2531
  * SECURITY UPDATE: arbitrary session variable modification via crafted
    session variable name
    - debian/patches/CVE-2010-3065.patch: handle PS_UNDEF_MARKER marker in
      ext/session/session.c.
    - CVE-2010-3065
  * debian/patches/lp564920-fix-big-files.patch: Fix downloading of large
    files (LP: #564920)
 -- Marc Deslauriers <email address hidden> Fri, 17 Sep 2010 08:14:26 -0400

Now that a fix has been released for Ubuntu 10.04, what needs to happen to get this fixed in the PHP 5 code base? PHP bug http://bugs.php.net/bug.php?id=52102 was closed out as a duplicate of http://bugs.php.net/bug.php?id=44522 , which it isn't an exact duplicate? How do we get the PHP5 code base updated with an acceptable fix?