Python should be compiled with USDT/dtrace user-space probes

Bug #1818778 reported by Trent Lloyd
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
python3.7 (Ubuntu)
Confirmed
Wishlist
Unassigned
python3.8 (Ubuntu)
Fix Released
Wishlist
Trent Lloyd
Disco
Fix Released
Undecided
Unassigned

Bug Description

Python3.6 and later support being compiled with USDT/dtrace user-space probes, this allows tools such as perf and BPF utilities (bpfcc-tools) to do runtime profiling and tracing of Python execution.

Ubuntu should compile Python with this support to enable better debugging of Python applications in production. perf/bpf tools can be attached to running production processes without having to restart them under a debugger which may destroy the situation you are trying to observe. This would be particularly useful with OpenStack deployments for example.

The current python3.6/python3.7/python3.8 packages do not build with this support, because it requires a build-dependency on systemtap-sdt-dev and the option --with-dtrace to be passed at compile time.

systemtap-sdt-dev is not currently in main, however, it was previously approved for a MIR in Bug #1203590 - it seems the original packages this was required for (Source Package: ust) was demoted to universe at some point and the systemtap-sdt-dev dependency went with it. So I am hopeful it will be re-approved for main inclusion and allow this to be done

Trent Lloyd (lathiat)
Changed in python3.8 (Ubuntu):
status: New → Confirmed
importance: Undecided → Wishlist
assignee: nobody → Trent Lloyd (lathiat)
Revision history for this message
Trent Lloyd (lathiat) wrote :

Attaching debdiff showing how to enable build-time support. debian/control is generated for the python version by debian/control.in so we patch both files, then enable --with-dtrace in the rules.

At this stage this patch is blocked by the MIR for systemtap-sdt-dev, and I will also file this upstream in Debian for the same

Revision history for this message
Trent Lloyd (lathiat) wrote :

Adding patch (git format-patch) against upstream Debian cpython-team/python3 git repository (canonical home of debian/ for python3 packages)

Trent Lloyd (lathiat)
Changed in python3.7 (Ubuntu):
status: New → Confirmed
importance: Undecided → Wishlist
assignee: nobody → Trent Lloyd (lathiat)
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "lp1818778-python3.8-enable-usdt.patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Revision history for this message
Trent Lloyd (lathiat) wrote :

You can unsubscribe ubuntu-sponsors from this

tags: removed: patch
Revision history for this message
Seth Arnold (seth-arnold) wrote :

The systemtap dependency may be easier to handle this time around than last time: we've slightly changed the rules about build-dependencies:

https://lists.ubuntu.com/archives/ubuntu-devel-announce/2016-April/001179.html

Be sure to pay attention to this point:

> - If you have a build-dependency which does not result in a runtime
> dependency, but *does* result in code being copied into the final
> package, you must ensure that this code copying is declared using the
> Built-Using header[5]. (This is an uncommon case; the vast majority of
> the affected packages are written in go, where our package toolchain
> handles this automatically today.)

Thanks

Revision history for this message
Simon Quigley (tsimonq2) wrote :

Unsubscribing the Ubuntu Sponsors Team as per the comment in #4.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python3.8 - 3.8.0~a4-3

---------------
python3.8 (3.8.0~a4-3) experimental; urgency=medium

  * Update symbols files.

 -- Matthias Klose <email address hidden> Wed, 15 May 2019 00:30:25 +0200

Changed in python3.8 (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Trent Lloyd (lathiat) wrote :

Matthias: Thanks for the upload in python3.8; I have verified this is working as expected

can we get the same upload for python3.7 since thats the default and it's quite hard to use python3.8 really.. maybe by f* we'll have python3.8 but at least for e* i'd like to get the same in for python3.7

Changed in python3.7 (Ubuntu):
assignee: Trent Lloyd (lathiat) → nobody
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Trent, or anyone else affected,

Accepted python3.8 into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python3.8/3.8.0-1~19.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python3.8 (Ubuntu Disco):
status: New → Fix Committed
tags: added: verification-needed verification-needed-disco
Mathew Hodson (mhodson)
tags: added: patch
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Trent, or anyone else affected,

Accepted python3.8 into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python3.8/3.8.0-3~19.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (3.4 KiB)

This bug was fixed in the package python3.8 - 3.8.0-3~19.04

---------------
python3.8 (3.8.0-3~19.04) disco-proposed; urgency=medium

  * SRU: LP: #1835737. Backport the final Python 3.8.0 release.

python3.8 (3.8.0-3) unstable; urgency=medium

  * python3.8-dev: Depend on zlib1g-dev, needed to link as an
    embedded interpreter.

python3.8 (3.8.0-2) unstable; urgency=medium

  * Fix a symlink to the shared debug library.
  * Install the python3.8d-embed pkg-config file.
  * Don't ship the python3-embed pkg-config file.

python3.8 (3.8.0-1) unstable; urgency=medium

  * Python 3.8.0 release.

python3.8 (3.8.0~rc1-3) unstable; urgency=medium

  * Apply proposed patch for issue 38368. LP: #1847036. Addresses: #941650.

python3.8 (3.8.0~rc1-2) unstable; urgency=medium

  * Python 3.8.0 release candidate 1.
  * Refresh patches.
  * Update symbols files.
  * Bump standards version.

python3.8 (3.8.0~b4-1) unstable; urgency=medium

  * Python 3.8.0 beta 4 release.

python3.8 (3.8.0~b3-1) unstable; urgency=medium

  * Python 3.8.0 beta 3 release.
  * Fix FTCBFS: (Helmut Grohne). Closes: #932077.
    + Build-Depends, when cross compiling we need python3.8.
    + Export CC, because dtrace needs it.
    + Honour DEB_BUILD_OPTIONS=nocheck more thoroughly.
    + Fix up sysconfigdata-name.diff.
  * Implement the native thread ids for the Hurd and KFreeBSD (Samuel
    Thibault). Closes: #931624.
  * Enable pgo/lto builds on arm64. Addresses: #934812.
  * Update symbols files.
  * Don't propagate lto flags to the _sysconfigdata module. Addresses: #934771.
  * d/patches/issue35998.diff: Disable TLS1.3 in the client on all platforms
    rather than just reducing the payload size (Michael Hudson-Doyle).

python3.8 (3.8.0~b2-5) unstable; urgency=high

  * Bump standards version.
  * Revert the KFreeBSD and Hurd fix for PyThread_get_thread_native_id,
    and use the implementation from Python3.7. Mark the symbol as linux only.
  * Don't encode the MACHDEP into the _sysconfigdata file name.

python3.8 (3.8.0~b2-4) unstable; urgency=medium

  * Fix the LTO build.

python3.8 (3.8.0~b2-3) unstable; urgency=medium

  * Define PY_HAVE_THREAD_NATIVE_ID for KFreeBSD and Hurd builds.
  * Fix issue #37504, building the texinfo docs (Dmitry Shachnev).
  * Enable LTO/PGO builds on some architectures.

python3.8 (3.8.0~b2-1) experimental; urgency=medium

  * Python 3.8.0 beta 2 release.
  * Refresh patches.
  * Update the symbols files.
  * Don't build the texinfo documentation, fails. See issue #37504.

python3.8 (3.8.0~b1-2) experimental; urgency=medium

  * Fix include links in python3.8-dbg.

python3.8 (3.8.0~b1-1) experimental; urgency=medium

  * Python 3.8.0 beta 1 release.
  * Refresh patches.
  * Add new importlib dependencies to python3-minimal.
  * Update symbols files.

python3.8 (3.8.0~a4-3) experimental; urgency=medium

  * Update symbols files.

python3.8 (3.8.0~a4-2) experimental; urgency=medium

  * Update to 20190514 from the trunk.
  * Fix interpreter name in the -dbg autopkg tests.
  * Enable dtrace/USDT probe support (--with-dtrace) and build-dep on
    systemtap-sdt-dev (Trent Lloyd). LP: #1818778.

python3.8 (3.8.0~a4-1) experimental; urgency=medium...

Read more...

Changed in python3.8 (Ubuntu Disco):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.