variations in python 2.7 breaks preppy

Asked by Robin Becker on 2019-07-29

in ubuntu 18.04 lts I have updated python from the original 2.7.15rc1 to 2.7.15+; unfortunately that breaks preppy < 2.7.0 which worked well before the upgrade. Is there somewhere I can see what patches have been made to Python 2.7.15?

I have built python 2.7.16 directly and the breakage is not present in that version.
The breakage seems to be in the tokenizer module which causes an extra newline token to appear in this code
#######################
from StringIO import StringIO
import tokenize, token
L = []
s='i'
tokenize.tokenize(StringIO(s.strip()).readline,lambda *a: L.append(a))
print('L=%s' % repr(L))
#######################
in 2.7.15rc1 [(1, 'i', (1, 0), (1, 1), 'i'), (0, '', (2, 0), (2, 0), '')], 2.7.15+ [(1, 'i', (1, 0), (1, 1), 'i'), (4, '', (1, 1), (1, 2), ''), (0, '', (2, 0), (2, 0), '')]

Question information

Language:
English Edit question
Status:
Solved
For:
Ubuntu python-defaults Edit question
Assignee:
No assignee Edit question
Solved by:
Manfred Hampl
Solved:
2019-07-30
Last query:
2019-07-30
Last reply:
2019-07-29
Manfred Hampl (m-hampl) said : #1

From which source did you install python 2.7.15+ ?
You probably better ask there, what their differences are to 2.7.15rc1.

If you are interested in details of the Ubuntu version, look at the package source.
https://launchpad.net/ubuntu/+source/python-defaults/2.7.15~rc1-1
https://launchpad.net/ubuntu/+archive/primary/+sourcefiles/python-defaults/2.7.15~rc1-1/python-defaults_2.7.15~rc1-1.tar.gz

Remark: Ubuntu's python 2.7.15~rc1-1 package was taken over from Debian without further modification.

Robin Becker (robin-reportlab) said : #2

Hi,
thanks for the info regarding sources.

So far as I know this python is coming from a package called python-minimal; I just used apt update/upgrade. Of course it is possible I have the wrong apt setup somewhere and it comes from the future eg bionic-backports.

I did not build it from source. I just assumed that this was a python provided by Ubuntu.

$ ls -l /usr/bin/python
lrwxrwxrwx 1 root root 9 Apr 16 2018 /usr/bin/python -> python2.7

$ dpkg -S /usr/bin/python2.7
python-minimal: /usr/bin/python2.7

$ aptitude search python | grep ^i | grep minimal
i libpython2.7-minimal - Minimal subset of the Python language (version 2.7)
i libpython3.6-minimal - Minimal subset of the Python language (version 3.6)
i A python-minimal - minimal subset of the Python language (default version)
i A python2.7-minimal - Minimal subset of the Python language (version 2.7)
i python3-minimal - minimal subset of the Python language (default python3 version)
i python3.6-minimal - Minimal subset of the Python language (version 3.6)

Manfred Hampl (m-hampl) said : #3

For diagnostic purposes, what is the output of the commands

uname -a
lsb_release -crid
apt policy python-minimal

Robin Becker (robin-reportlab) said : #4

$ lsb_release -crid
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic
rptlab@everest:~/devel/docengine-test
$ apt policy python-minimal
python-minimal:
  Installed: 2.7.15~rc1-1
  Candidate: 2.7.15~rc1-1
  Version table:
 *** 2.7.15~rc1-1 500
        500 http://gb.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
        100 /var/lib/dpkg/status

Manfred Hampl (m-hampl) said : #5

I do not see any "version 2.7.15+". According to the Ubuntu package management system you have version 2.7.15~rc1-1 installed, and that version stems from April 2018 without any change since, see https://launchpad.net/ubuntu/+source/python-defaults

Manfred Hampl (m-hampl) said : #6

Digging further, I conclude, that we were looking for the wrong package:

/usr/lib/python2.7/tokenize.py is part of the package libpython2.7-minimal, at that had an update recently (June 10, 2019), and now is on version 2.7.15-4ubuntu4~18.04

The diff shows several occurrences of tokenize, including a comment
# Make sure that the tokenizer puts in an implicit NEWLINE
# when the input lacks a trailing new line.

Maybe this is what you are searching for.

https://launchpad.net/ubuntu/+source/python2.7

Best Manfred Hampl (m-hampl) said : #7

The real change seems to be the addition of the following three lines in /usr/lib/python2.7/tokenize.py (lines 447-449)

    # Add an implicit NEWLINE if the input doesn't end in one
    if last_line and last_line[-1] not in '\r\n':
        yield (NEWLINE, '', (lnum - 1, len(last_line)), (lnum - 1, len(last_line) + 1), '')

I have looked for these lines in a few other versions of libpython2.7-minimal in Ubuntu, and they were not present there.

If you deem the addition of these lines wrong, then I suggest that you create a bug report.

Robin Becker (robin-reportlab) said : #8

Awesome, thanks!

On Mon, 29 Jul 2019 at 18:42, Manfred Hampl <
<email address hidden>> wrote:

> Your question #682415 on python-defaults in Ubuntu changed:
>
> https://answers.launchpad.net/ubuntu/+source/python-defaults/+question/682415
>
> Manfred Hampl proposed the following answer:
> The real change seems to be the addition of the following three lines in
> /usr/lib/python2.7/tokenize.py (lines 447-449)
>
> # Add an implicit NEWLINE if the input doesn't end in one
> if last_line and last_line[-1] not in '\r\n':
> yield (NEWLINE, '', (lnum - 1, len(last_line)), (lnum - 1,
> len(last_line) + 1), '')
>
> I have looked for these lines in a few other versions of
> libpython2.7-minimal in Ubuntu, and they were not present there.
>
> If you deem the addition of these lines wrong, then I suggest that you
> create a bug report.
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
>
> https://answers.launchpad.net/ubuntu/+source/python-defaults/+question/682415/+confirm?answer_id=6
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
>
> https://answers.launchpad.net/ubuntu/+source/python-defaults/+question/682415
>
> You received this question notification because you asked the question.
>

--
Robin Becker

Robin Becker (robin-reportlab) said : #9

Thanks for the effort; this code seems to appear in python 3.x for x >= 6

Thanks Manfred Hampl, that solved my question.