PyPI access needed during build

Asked by Chris on 2019-01-20

I'm building a package using dh_virtualenv - it basically packages entire project into a portable virutalenv.

During build process, it downloads all packages from PyPi using pip, but I guess network access is a no go on a build server for security reasons.

Is there an internal PyPI mirror accessible from launchpad build server?

Unfortunately, packaging dependencies is not a practical solution.

Question information

Language:
English Edit question
Status:
Expired
For:
Launchpad itself Edit question
Assignee:
No assignee Edit question
Last query:
2019-02-12
Last reply:
2019-02-27
Colin Watson (cjwatson) said : #1

I'm sorry, but no. The only supported mechanisms are packaging dependencies, vendoring dependencies, or (if applicable) building a snap.

Chris (ezaquarii) said : #2

How about shipping required Python dependencies as a PyPI cache directory? Would that be an option?

Colin Watson (cjwatson) said : #3

That should be possible, yes, though you'd have to work out for yourself how to pass the right virtualenv/pip options through dh_virtualenv.

Chris (ezaquarii) said : #4

I tried this approach with source packages, but the solution is pretty much unmaintainable due how python handles wheel build-time dependencies (ELI5 - it requires network).

How would it work if I ship pre-build python wheels? Some of them contain compiled binary code.

Chris (ezaquarii) said : #5

The technicalities are already solved, but I don't want to wake up in 3 months with my account locked because I violated some terms of use.

William Grant (wgrant) said : #6

Did you try using sdists? Wheels don't really make much sense when you're building for multiple architectures.

Chris (ezaquarii) said : #7

Yes, I tried sdists initially.

The problem is that when pip installs/builds sdist package, it must also pull build-time dependencies. There are few problems:

1) Unfortunately, the mechanism to satisfy those build-time dependencies is broken and requires network trip or customized $HOME

2) newer mechanism - PEP 518 - that solves this problem is not widely used yet, so I'd need to update many dependencies manually

At this stage most of the packages are multiarch with exception of few, so packaging binary wheels for different architectures is definitely more manageable than sdists.

Alternative would be to supply my own PyPI mirror, but that requires some config files in my build-time $HOME to work. This is also doable,but I don't see any way to customize launchpad build environment.

Chris (ezaquarii) said : #8

(please bear in mind that installation-time dependencies are handled differently from build-time deps, which is the source of all problems)

Colin Watson (cjwatson) said : #9

That stuff can be a bit of a pain, but it can be handled: we do that in the build system for Launchpad itself. These bits may be helpful, particularly the trick of writing .pydistutils.cfg at the top of the virtualenv:

  https://bazaar.launchpad.net/+branch/launchpad/view/head:/Makefile#L11
  https://bazaar.launchpad.net/+branch/launchpad/view/head:/Makefile#L237

Launchpad Janitor (janitor) said : #10

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Chris (ezaquarii) said : #11

I'm not sure how your solution works:

(echo '[easy_install]'; \
  echo "allow_hosts = ''"; \
  echo 'find_links = file://$(WD)/download-cache/dist/') \
  >env/.pydistutils.cfg

This puts pydistutils into env. The documentation states that distutils conf are taken from 3 sources:
1) current directory
2) distutils system dir
3) user $HOME

I solved my problem by re-defining $HOME:

override_dh_virtualenv:
 echo $(PWD)
 (echo '[easy_install]'; \
  echo "allow_hosts = ''"; \
  echo 'find_links = file://$(PWD)/pypi/') >$(PWD)/.pydistutils.cfg
 HOME=$(PWD) dh_virtualenv --python /usr/bin/python3 \
               --install-suffix='venv' \
               --builtin-venv \
               --upgrade-pip \
               --extra-pip-arg --no-cache \
               --extra-pip-arg --no-index \
               --extra-pip-arg --find-links=pypi \
               --extra-pip-arg --verbose \
               --requirements=requirements.txt

This way all ./setup.py egg_info pick up my .pydistutils.cfg.

Colin Watson (cjwatson) said : #12

I promise it does work, at least when using virtualenv, although perhaps not with Python 3's built-in venv. The only documentation I've been able to find for it is in the virtualenv changelog:

  https://virtualenv.pypa.io/en/stable/changes/#v1-3-3

Chris (ezaquarii) said : #13

Oh, it looks like the solution from launchpad is indeed virtualenv specific - thank you for the documentation link. :-)

I'll to reach Python developers with a distutils change proposal. I'd be nice to have a way to inject this via environment variable instead of hacking around...