update Python3 for trusty

So is the plan to bring 3.4.1 into Trusty then?

Status changed to 'Confirmed' because the bug affects multiple users.

Is there any chance we could get latest upstream 3.4 SRU'd soon? We're tracking a couple of relatively serious regressions issues fixed upstream but still present in Trusty. See bug 1367907 and bug 1382607 for details.

Based on E-mail discussion with Barry and Matthias, it sounds like the plan now is to SRU MRE Python 3.4.3 into Trusty once it's available (due out February 22, 2015 according to the official release schedule).

Hello Matthias, or anyone else affected,

Accepted python-urllib3 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-urllib3/1.7.1-1ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Brian, was comment #5 a mis-update? This bug is about getting newer Python 3.4 into Trusty, not a newer python-urllib3 module.

I upgraded to https://launchpad.net/ubuntu/+source/python-urllib3/1.7.1-1ubuntu3 and checked my Python 3 version:

Actual:

$ python3
Python 3.4.0 (default, Apr 11 2014, 13:05:11)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.

Expected:

$ python3
Python 3.4.2 (default, ???, ???)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.

there was no python3.4 upload yet. python3-urllib references the wrong bug number. should be lp #1433324.

@brian-murray, @doko, do you know what's blocking this bug?

On Jun 17, 2015, at 12:08 PM, Matthias Klose wrote:

>+ To validate this SRU, I'm proposing to use the results from the test
>+ rebuild, plus evaluating the testsuite results of the python3.4 package
>+ itself.

+1

Status changed to 'Confirmed' because the bug affects multiple users.

Reviewing the debdiff of the package uploaded to the queue.
 - debian/patches/apport-support.dpatch, debian/patches/doc-build.diff, debian/patches/installed-testsuite.diff,debian/patches/issue21097.diff : why have these patches been removed? I find no references to these in the changelog.

- debian/rules:
@@ -221,21 +231,22 @@
     stamps/stamp-check stamps/stamp-pystone stamps/stamp-pybench
        touch $@

-PROFILE_EXCLUDES = test_compiler test_distutils test_platform test_subprocess \
+PROFILE_EXCLUDES = test_compiler test_distutils test_subprocess \
        test_multiprocessing test_socketserver \
        test_thread test_threaded_import test_threadedtempfile \
        test_threading test_threading_local test_threadsignals \
        test_concurrent_futures test_ctypes \
        test_dbm_dumb test_dbm_ndbm test_pydoc test_sundry \
- test_signal test_ioctl test_gdb
+ test_signal test_ioctl test_gdb test_ensurepip test_venv
 @@ -416,6 +426,7 @@
 endif
 TESTOPTS = -j 1 -w -u$(TEST_RESOURCES)
 TEST_EXCLUDES =
+TEST_EXCLUDES += test_ensurepip test_venv
 ifeq ($(on_buildd),yes)
   TEST_EXCLUDES += test_tcl test_codecmaps_cn test_codecmaps_hk \
         test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw \

Why are the ensurepip and venv tests being skipped? Considering the great deal of work that's been put into ensurepip and venv in this version, don't we want this to pass the tests?

On 07/10/2015 07:58 AM, Steve Langasek wrote:
> Reviewing the debdiff of the package uploaded to the queue.
> - debian/patches/apport-support.dpatch,

unused ("dpatch"). left over.

> debian/patches/doc-build.diff

needed to build with sphinx 0.5, trusty has 1.2

> debian/patches/installed-testsuite.diff

this just disabled a test, fixed it upstream.

> debian/patches/issue21097.diff

applied upstream

> why have these patches been removed? I find no references to these in the changelog.

> - debian/rules:
> @@ -221,21 +231,22 @@
> stamps/stamp-check stamps/stamp-pystone stamps/stamp-pybench
> touch $@
>
> -PROFILE_EXCLUDES = test_compiler test_distutils test_platform test_subprocess \
> +PROFILE_EXCLUDES = test_compiler test_distutils test_subprocess \
> test_multiprocessing test_socketserver \
> test_thread test_threaded_import test_threadedtempfile \
> test_threading test_threading_local test_threadsignals \
> test_concurrent_futures test_ctypes \
> test_dbm_dumb test_dbm_ndbm test_pydoc test_sundry \
> - test_signal test_ioctl test_gdb
> + test_signal test_ioctl test_gdb test_ensurepip test_venv
> @@ -416,6 +426,7 @@
> endif
> TESTOPTS = -j 1 -w -u$(TEST_RESOURCES)
> TEST_EXCLUDES =
> +TEST_EXCLUDES += test_ensurepip test_venv
> ifeq ($(on_buildd),yes)
> TEST_EXCLUDES += test_tcl test_codecmaps_cn test_codecmaps_hk \
> test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw \
>
> Why are the ensurepip and venv tests being skipped? Considering the
> great deal of work that's been put into ensurepip and venv in this
> version, don't we want this to pass the tests?

this would add build dependencies on all the setuptools and pip packages
currently in universe, because we remove the internal copies in the packages.
these tests just failed in the 3.4.0 packages.

Hello Matthias, or anyone else affected,

Accepted python3.4 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python3.4/3.4.3-1ubuntu1~14.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Status changed to 'Confirmed' because the bug affects multiple users.

Building trusty daily desktop ISOs with the new python3.4 causes ubiquity to fail to start. I've saved two ISOs where the only delta is the python version to show the issue:

http://people.canonical.com/~adconrad/lp1348954/

In the above directory, 20150728 is broken, while 20150728.1 works.

it looks like LP: #1326707 is the only Python3 related issue in ubiquity, but not backported to trusty, although python 3.4 is the default.

I'm attaching a backport for it. How could we test this changed ubiquity on the image?

Hello Matthias, or anyone else affected,

Accepted python3.4 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python3.4/3.4.3-1ubuntu1~14.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hm. That python3-stdlib-extensions SRU would have been easier to SRU if there wasn't a bunch of changes to dead code in debian/rules...

Hello Matthias, or anyone else affected,

Accepted python3-stdlib-extensions into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python3-stdlib-extensions/3.4.3-1~14.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

The ubiquity cherry-pick resolved the issue mentioned in comment #18

the test rebuilds also had binutils, gcc-4.8 and a proposed python2.7 update. Looking at the test results with a python3 hat only. All ftbfs in the test-rebuild-20150501-updates-trusty ppa are unrelated to python3, the python-apt i
ssue is the same as in the test-rebuild-20150317-trusty ppa.

in the test-rebuild-20150317-trusty archive the following packages fail to build:

bzr
        unrelated to python3
celery
        unrelated to python3
cinder
        unrelated to python3
click-apparmor
        unrelated, another SRU: 1451459
firefox
        unrelated to python3
flite
        unrelated to python3
gcc-4.7
        unrelated to python3
gcc-4.8
        unrelated to python3
gccgo-4.9
        unrelated to python3
gnome-sharp2
        unrelated to python3
grantlee
        unrelated to python3
graphite2
        unrelated to python3
gtk-sharp2
        unrelated to python3
hfsutils
        unrelated to python3
icu
        unrelated to python3
keystone
        unrelated to python3
libqtdbusmock
        unrelated, not a regression, another SRU: 1451512
libqtdbustest
        unrelated, not a regression, another SRU: 1451507
libreoffice
        builds in the trusty-updates test rebuild
libreoffice-voikko
        unrelated to python3
libxfont
        unrelated to python3
lintian
        unrelated to python3
linux
        unrelated to python3
llvm-toolchain-3.4
        unrelated to python3
location-service
        unrelated to python3
mir
        unrelated to python3
mysql-5.5
        unrelated to python3
neutron
        unrelated to python3
openjdk-7
        unrelated to python3
openvpn
        unrelated to python3
oxide-qt
        unrelated to python3
patch
        unrelated to python3
platform-api
        unrelated to python3
ps3-kboot
        unrelated to python3
pygresql
        unrelated to python3, postgres C header not found, 1451530
python-apt
        unrelated to python2.7 and python3: test failure
========================
FAIL: test_add_key_from_server_mitm (test_auth.TestAuthKeys)
Verify that the key fingerprint is verified after download
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/doko/tmp/python-apt-0.9.3.5ubuntu1/tests/test_auth.py", line 216, in test_add_key_from_server_mitm
    cm.exception)
AssertionError: recv from 'hkp://localhost:19191' failed for '0101010178F7FE5C3E65D8AF8B48AD6246925553'

----------------------------------------------------------------------
Ran 88 tests in 13.779s

FAILED (failures=1)

python-django
        unrelated to python3
python-docutils
        unrelated to python3, builds with the updated python3.4, python2.7.9 issue?
python-eventlet
        unrelated to python3
python-glanceclient
        unrelated to python3
python-greenlet
        unrelated to python3
serf
        unrelated to python3
spice
        unrelated to python3
thunderbird
        unrelated to python3
tomcat7
        unrelated to python3
u-boot
        unrelated to python3
unity-webapps-qml
        unrelated to python3
usbredir
        unrelated to python3
webbrowser-app
        unrelated to python3
yaboot
        unrelated to python3

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

 affects ubuntu/trusty/python3-defaults
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJV/GRtAAoJEEUTBuC2oiinbxIIAJQrFTcP0PKq85ZM92pBpEHf
swNzHpGX6wx1C+rCCoJaDrDKSNIO2FFDNH1/wvIdhaVHGSmP/4Ib8o1JE0JNMo43
9Xj2nMo+5KdHSrmKOqsHR0Sl4NeWRULND15zcUg5fvY/Thx0BDdoWjMOtuboHNmA
gPWGO5ofWOu9c3oaWhxf0TthOutP88b038+f7/dBKnj4Z6H4MOBsdcY7norBhGKo
C/in86WA4l+LnOo+eI9N4p9CsJRFgYNa86v3P1L/AozI+cZv0odONdu2FGB8/lLO
4Cp/tTZerMeHkyl4LmO4YeLcdQekvAIqU2yt1Qis+43peR4xJLw1sIC4VY/W6h4=
=QYQ1
-----END PGP SIGNATURE-----

This bug was fixed in the package python3.4 - 3.4.3-1ubuntu1~14.04.1

---------------
python3.4 (3.4.3-1ubuntu1~14.04.1) trusty-proposed; urgency=medium

  * Backport issue #23844 from the 3.4 branch, replacing the 512 bit dh key
    with a 2014 bit one. Triggered by OpenSSL security update in
    trusty-security. LP: #1264554.
  * Fix expansion of makefile macros for _sysconfigdata. Issue #24705.

python3.4 (3.4.3-1ubuntu1~14.04) trusty-proposed; urgency=medium

  * SRU: Update Python3 for trusty. LP: #1348954.

python3.4 (3.4.3-1ubuntu1) vivid; urgency=medium

  * debian/tests: Use init system agnostic "service" command instead of
    upstart specific "stop". Also drop unnecessary "status" call right after
    stopping apport.

python3.4 (3.4.3-1) experimental; urgency=medium

  * Python 3.4.3 release.
  * Changes since 20141202 (3.4.2-4):
    - Issue #22896: Avoid using PyObject_AsCharBuffer(),
      PyObject_AsReadBuffer(), and PyObject_AsWriteBuffer().
    - Issue #21295: Revert some changes (issue #16795) to AST line numbers and
      column offsets that constituted a regression.
    - Issue #21408: The default __ne__() now returns NotImplemented if __eq__()
      returned NotImplemented.
    - Issue #23321: Fixed a crash in str.decode() when error handler returned
      replacment string longer than mailformed input data.
    - Issue #23048: Fix jumping out of an infinite while loop in the pdb.
    - Issue #23165: Perform overflow checks before allocating memory in the
      _Py_char2wchar function.
    - Issue #23099: Closing io.BytesIO with exported buffer is rejected now to
      prevent corrupting exported buffer.
    - Issue #23363: Fix possible overflow in itertools.permutations.
    - Issue #23364: Fix possible overflow in itertools.product.
    - Issue #23366: Fixed possible integer overflow in itertools.combinations.
    - Issue #23369: Fixed possible integer overflow in
      _json.encode_basestring_ascii.
    - Issue #23353: Fix the exception handling of generators in
      PyEval_EvalFrameEx(). At entry, save or swap the exception state even if
      PyEval_EvalFrameEx() is called with throwflag=0. At exit, the exception
      state is now always restored or swapped, not only if why is WHY_YIELD or
      WHY_RETURN.
    - Issue #18518: timeit now rejects statements which can't be compiled
      outside a function or a loop (e.g. "return" or "break").
    - Issue #23094: Fixed readline with frames in Python implementation of
      pickle.
    - Issue #23268: Fixed bugs in the comparison of ipaddress classes.
    - Issue #21408: Removed incorrect implementations of __ne__() which didn't
      returned NotImplemented if __eq__() returned NotImplemented. The default
      __ne__() now works correctly.
    - Issue #19996: :class:`email.feedparser.FeedParser` now handles
      (malformed) headers with no key rather than amusing the body has started.
    - Issue #23248: Update ssl error codes from latest OpenSSL git master.
    - Issue #23098: 64-bit dev_t is now supported in the os module.
    - Issue #23250: In the http.cookies module, capitalize "HttpOnly" and
      "Secure" as they are written in the standard.
    - Issue #23...

The verification of the Stable Release Update for python3.4 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package python3-stdlib-extensions - 3.4.3-1~14.04.2

---------------
python3-stdlib-extensions (3.4.3-1~14.04.2) trusty-proposed; urgency=medium

  * SRU, update python3.4 for trusty. LP: #1348954.

python3-stdlib-extensions (3.4.3-1) unstable; urgency=medium

  * Bump version to 3.4.3.

python3-stdlib-extensions (3.4.2-1) unstable; urgency=medium

  * Bump version to 3.4.2 release.

python3-stdlib-extensions (3.4.2~rc1-1) unstable; urgency=medium

  * Bump version to 3.4.2 release candidate 1.

python3-stdlib-extensions (3.4.1-3) unstable; urgency=medium

  * Build for blt 2.5. Closes: #753929.

python3-stdlib-extensions (3.4.1-2) unstable; urgency=medium

  * Require BLT version built for Tcl/Tk 8.6.

python3-stdlib-extensions (3.4.1-1) unstable; urgency=medium

  * Bump version to 3.4.1.
  * Remove python 3.3 sources.

 -- Matthias Klose <email address hidden> Wed, 17 Jun 2015 14:23:30 +0200

On 27.09.2015 23:52, Mathew Hodson wrote:
> ** No longer affects: python3-defaults (Ubuntu)

Mathew, any reason you close SRU tasks without any comment and reason? Now reopened.

Thanks, Matthias

Sorry, I was trying to get the Trusty task to appear, and I accidentally messed it up. I guess that's bug 110195.

python3.4 3.4.3-1ubuntu1~14.04.1 has been removed from trusty-updates due to bug #1500768 which is a regression introduced in this SRU. This SRU can be reconsidered once this regression has been resolved.

Bug #1499075 is another regression introduced by this SRU. A fix is available in python-botocore/trusty-proposed, so please release that fix either before/alongside this SRU to avoid reintroducing this regression.

Bug #1503774 is another regression for this. It's missing the pyvenv scripts and docs that 3.2.0 shipped.

according to
http://packages.ubuntu.com/search?searchon=contents&keywords=pyvenv&mode=exactfilename&suite=trusty&arch=any
pyvenv never shipped in trusty.

Hello Matthias, or anyone else affected,

Accepted python3.4 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python3.4/3.4.3-1ubuntu1~14.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

verified that the urllib.request.urlopen tests in the bug description pass when certification is disabled, and fail when certification is enabled.

This upload includes a change to add a new configuration file for setting the site policy for whether to enforce signature signing.

I objected privately to the addition of this configuration file when Matthias proposed it. This adds complexity to the system both on upgrade and in 14.04 itself; the patch was proposed upstream and rejected; and the configuration file will cause the behavior of programs to be inconsistent across installations of Ubuntu. Furthermore, the claim in the changelog that this config file will be removed on upgrade to 15.04 is *false*; there is no code in the 15.04 version of python3.4 which implements this, and there is no python3.4 package in the SRU queue for vivid.

Users who upgraded to 3.4.3 previously in trusty-updates are currently stuck on an upgrade island as a result of the previous SRU having been backed out due to regressions. We need to resolve this problem quickly. The SRU that has been uploaded is not appropriate as a quick fix, it has longer-term consequences that need to be thought through carefully.

I am going to upload a new SRU that reverts the addition of this config file. The code patch can stay in place, it should implement the correct behavior with or without the config file actually being present (and I don't have an alternative implementation of this policy change to hand that we could quickly release). But if we're going to release this SRU with that code path, we should not be advising users to use a global config file to configure the site policy until this has been discussed more broadly.

On Oct 14, 2015, at 07:50 PM, Steve Langasek wrote:

>I am going to upload a new SRU that reverts the addition of this config
>file. The code patch can stay in place, it should implement the correct
>behavior with or without the config file actually being present (and I
>don't have an alternative implementation of this policy change to hand
>that we could quickly release). But if we're going to release this SRU
>with that code path, we should not be advising users to use a global
>config file to configure the site policy until this has been discussed
>more broadly.

I guess given Tyler Hicks' out-of-band feedback, we should just turn off
certificate checking for 14.04. Let's let the security team and/or bug
reports drive any change in this behavior.

On 14.10.2015 21:50, Steve Langasek wrote:
> This upload includes a change to add a new configuration file for
> setting the site policy for whether to enforce signature signing.
>
> I objected privately to the addition of this configuration file when
> Matthias proposed it.

> This adds complexity to the system both on
> upgrade and in 14.04 itself; the patch was proposed upstream and
> rejected;

The patch was not rejected upstream, because it wasn't seen as relevant. It was
somehow acknowledged that something like this is needed. Other options
discussed seemd to be worse. See http://bugs.python.org/issue23857

> and the configuration file will cause the behavior of programs
> to be inconsistent across installations of Ubuntu.

This claim seems to be wrong. The default is the same as in the released trusty.

> Furthermore, the
> claim in the changelog that this config file will be removed on upgrade
> to 15.04 is *false*; there is no code in the 15.04 version of python3.4
> which implements this, and there is no python3.4 package in the SRU
> queue for vivid.

This is in progress, just started with wily.

> Users who upgraded to 3.4.3 previously in trusty-updates are currently
> stuck on an upgrade island as a result of the previous SRU having been
> backed out due to regressions. We need to resolve this problem quickly.
> The SRU that has been uploaded is not appropriate as a quick fix, it has
> longer-term consequences that need to be thought through carefully.
 >
> I am going to upload a new SRU that reverts the addition of this config
> file. The code patch can stay in place, it should implement the correct
> behavior with or without the config file actually being present (and I
> don't have an alternative implementation of this policy change to hand
> that we could quickly release). But if we're going to release this SRU
> with that code path, we should not be advising users to use a global
> config file to configure the site policy until this has been discussed
> more broadly.

sure we can do that.

Yes, we should disable the certificate verification in 14.04 since 14.04 initially shipped with a version of Python 3.4 that did not perform certificate verification by default. We may decide to enable certificate verification by default in a future security update.

I tend to agree with Steve that the complexity of a global toggle for certificate verification is something that deserves some thought. In the out-of-band communication that Barry mentioned, I didn't ack or nack such a global toggle. I think some people will find it useful. However, it is something that can be rolled out separately from this urgent SRU.

Status changed to 'Confirmed' because the bug affects multiple users.

This bug was fixed in the package python3.4 - 3.4.3-1ubuntu1~14.04.3

---------------
python3.4 (3.4.3-1ubuntu1~14.04.3) trusty; urgency=medium

  * Remove the config file from the package, as there is no handling in
    place to deal with this config file on upgrade and it is not appropriate
    for inclusion in an urgent SRU.

 -- Steve Langasek <email address hidden> Wed, 14 Oct 2015 12:52:19 -0700

There has been a python3-defaults upload in the trusty queue for about a year, but there is no corresponding task nor description why a -defaults update is necessary. If this is still relevant, please update the bug and reupload.