Launchpad itself

Disable compression and enable direct-SHA1-fetch for cloud-images/+oci/ubuntu-base Git repo

Asked by Tomáš Virtus

Dear Launchpad team,

please

a) disable delta compression for binary files and
b) enable allow-tip-sha1-in-want and allow-reachable-sha1-in-want protocol capabilities

on server side in the following Git repositories:

1. https://git.launchpad.net/cloud-images/+oci/ubuntu-base
2. https://git.launchpad.net/virtustom-cloud-images2/+oci/ubuntu-base

The former repository is used by CPC & ROCKs for passing rootfs tarballs to Launchpad OCI recipes and is going to be used as the source of rootfs tarballs for Docker Hub CI. The latter is a fork of the former used by me for testing.

This repository is used only for storing tarballs and when working with the repo from CI the server takes inordinate amount of time trying to compress already compressed tarball blobs before sending them to the client. It's wasted cycles and time. I've found this link helpful in describing how to disable delta compression on Git server: https://marc.info/?l=git&m=153197844224467

Direct-SHA1-fetch is used by https://github.com/docker-library/official-images (Source of Docker Hub official images) when commit ID is not reachable from given ref or when no ref is specified. We don't store history in this repo, we replace commits with tarballs, and so previous commits are not reachable from the branch. Enabling the aforementioned protocol capabilities would allow client to ask server for commit by SHA1 directly instead of fetching whole branch (where the commit is not anyway). IMHO it's not guaranteed that we will fetch the commit as it can be garbage collected anytime we push to the repository, but that's what https://github.com/tianon/docker-brew-ubuntu-core/ has been relying on, since it also stores only one commit with tarballs. It is currently blocking merges because PR bot cannot fetch previous commit, so enabling it would (perhaps) help.

Question information

Language:
English Edit question
Status:
Needs information
For:
Launchpad itself Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Colin Watson (cjwatson) said : 		#1

This seems like a very peculiar use of git, which is on the whole not a particularly good way to store tarballs, even though it can in principle be made to work. Can you explain what other options you considered here? For example, rootfs tarballs are often better handled via something that looks more like a traditional web server; on cloud-images.ubuntu.com, the CPC team uses simplestreams to index them.

The options here aren't things that at present we can simply switch on; they would involve some non-trivial development work on the Launchpad git server. So, before we commit to anything like that, we should make sure that we've properly considered other options. (Direct SHA1 fetch also has some complex interactions with privacy in the event that somebody were to make a private fork of a repository that has it enabled, so that would additionally involve some webapp-level design and implementation changes.)

Revision history for this message
Tomáš Virtus (virtustom) said (last edit ): 		#2

Unfortunately, this is how build system for Docker Hub works, it requires Git repos for Dockerfiles of images it puts into official namespace. The format these library files have is desccribed https://github.com/docker-library/official-images. It's definitely possible to download rootfs tarballs in the Dockerfile, though that'd require some rework in CPC pipeline. We currently publish to images to https://partner-images.canonical.com/oci/ but there's a delay because it has to go through Jerff. And since we use Git to pass rootfs tarballs to Launchpad OCI recipes, storing the same blobs in the same repo seemed like a good fit.

I think we can get away without direct-SHA1-fetch. To disable compression of binary blobs, this should be set on the server repository:

  echo "*.bin binary -delta" >repo/.git/info/attributes

Can you help with this problem?

Provide an answer of your own, or ask Tomáš Virtus for more information if necessary.

To post a message you must log in.