PyPI access needed during build

Asked by Chris on 2019-01-20

I'm building a package using dh_virtualenv - it basically packages entire project into a portable virutalenv.

During build process, it downloads all packages from PyPi using pip, but I guess network access is a no go on a build server for security reasons.

Is there an internal PyPI mirror accessible from launchpad build server?

Unfortunately, packaging dependencies is not a practical solution.

Question information

Language:
English Edit question
Status:
Open
For:
Launchpad itself Edit question
Assignee:
No assignee Edit question
Last query:
2019-02-12
Last reply:
2019-01-22
Colin Watson (cjwatson) said : #1

I'm sorry, but no. The only supported mechanisms are packaging dependencies, vendoring dependencies, or (if applicable) building a snap.

Chris (ezaquarii) said : #2

How about shipping required Python dependencies as a PyPI cache directory? Would that be an option?

Colin Watson (cjwatson) said : #3

That should be possible, yes, though you'd have to work out for yourself how to pass the right virtualenv/pip options through dh_virtualenv.

Chris (ezaquarii) said : #4

I tried this approach with source packages, but the solution is pretty much unmaintainable due how python handles wheel build-time dependencies (ELI5 - it requires network).

How would it work if I ship pre-build python wheels? Some of them contain compiled binary code.

Chris (ezaquarii) said : #5

The technicalities are already solved, but I don't want to wake up in 3 months with my account locked because I violated some terms of use.

William Grant (wgrant) said : #6

Did you try using sdists? Wheels don't really make much sense when you're building for multiple architectures.

Chris (ezaquarii) said : #7

Yes, I tried sdists initially.

The problem is that when pip installs/builds sdist package, it must also pull build-time dependencies. There are few problems:

1) Unfortunately, the mechanism to satisfy those build-time dependencies is broken and requires network trip or customized $HOME

2) newer mechanism - PEP 518 - that solves this problem is not widely used yet, so I'd need to update many dependencies manually

At this stage most of the packages are multiarch with exception of few, so packaging binary wheels for different architectures is definitely more manageable than sdists.

Alternative would be to supply my own PyPI mirror, but that requires some config files in my build-time $HOME to work. This is also doable,but I don't see any way to customize launchpad build environment.

Chris (ezaquarii) said : #8

(please bear in mind that installation-time dependencies are handled differently from build-time deps, which is the source of all problems)

Colin Watson (cjwatson) said : #9

That stuff can be a bit of a pain, but it can be handled: we do that in the build system for Launchpad itself. These bits may be helpful, particularly the trick of writing .pydistutils.cfg at the top of the virtualenv:

  https://bazaar.launchpad.net/+branch/launchpad/view/head:/Makefile#L11
  https://bazaar.launchpad.net/+branch/launchpad/view/head:/Makefile#L237

Can you help with this problem?

Provide an answer of your own, or ask Chris for more information if necessary.

To post a message you must log in.