Build farm taken over daily by single PPA user

Asked by Rob Savoury

Every day the Launchpad build farm is "taken over" by a single PPA user (~mutlaqja) with their nightly builds. For example, below is a snapshot of the build farm status at around 5pm PDT (or midnight GMT) today showing 70 out of 90 amd64 builders in use by the one PPA user (note that one hour later, now 6pm PDT, there are still 60 out of 90 builders in use by this one user!).

Is it normal to have such a large percentage of the Launchpad builders suddenly used by a single user? Is it possible for the daily recipes of this one user to be kicked off in series rather than parallel, such that this one user (building various INDI software, evidently) doesn't cause such a disruption to the Launchpad build farm? Just wondering, as it sure does interrupt the flow of work for everyone else using the Launchpad builders!

---
amd64 i386
lcy01-amd64-001 Building ~mutlaqja/+git/dummy recipe build in ubuntu eoan [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-002 Building ~mutlaqja/+git/dummy recipe build in ubuntu bionic [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-003 Building ~mutlaqja/+git/dummy recipe build in ubuntu xenial [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-004 Building ~mutlaqja/+git/dummy recipe build in ubuntu focal [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-005 Building ~mutlaqja/+git/dummy recipe build in ubuntu xenial [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-006 Cleaning
lcy01-amd64-007 Building ~mutlaqja/+git/dummy recipe build in ubuntu groovy [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-008 Building ~mutlaqja/+git/dummy recipe build in ubuntu focal [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-009 Building amd64 build of linux 5.4.1-42.46custom2 in ubuntu focal RELEASE [~xalt7x/ubuntu/linux-lowlatency-custom]
lcy01-amd64-010 Building ~mutlaqja/+git/dummy recipe build in ubuntu eoan [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-011 Building ~mutlaqja/+git/dummy recipe build in ubuntu bionic [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-012 Building ~mutlaqja/+git/dummy recipe build in ubuntu bionic [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-013 Building ~mutlaqja/+git/dummy recipe build in ubuntu eoan [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-014 Building ~mutlaqja/+git/dummy recipe build in ubuntu bionic [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-015 Building ~mutlaqja/+git/dummy recipe build in ubuntu xenial [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-016 Building ~mutlaqja/+git/dummy recipe build in ubuntu eoan [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-017 Building ~mutlaqja/+git/dummy recipe build in ubuntu focal [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-018 Building ~mutlaqja/+git/dummy recipe build in ubuntu eoan [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-019 Building ~mutlaqja/+git/dummy recipe build in ubuntu bionic [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-020 Building ~mutlaqja/+git/dummy recipe build in ubuntu eoan [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-021 Building ~mutlaqja/+git/dummy recipe build in ubuntu eoan [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-022 Building ~ppsspp/ppsspp/+git/ppsspp recipe build in ubuntu xenial [~ppsspp/ubuntu/testing]
lcy01-amd64-023 Building ~reviczky/texlive/+git/github recipe build in ubuntu eoan [~reviczky/ubuntu/luatex-daily]
lcy01-amd64-024 Building ~mutlaqja/+git/dummy recipe build in ubuntu bionic [~mutlaqja/ubuntu/indinightly]
lcy01-amd64-025 Building ~reviczky/texlive/+git/github recipe build in ubuntu groovy [~reviczky/ubuntu/luatex-daily]
lcy01-amd64-026 Building ~ppsspp/ppsspp/+git/ppsspp recipe build in ubuntu trusty [~ppsspp/ubuntu/testing]
lcy01-amd64-027 Building ~ppsspp/ppsspp/+git/ppsspp recipe build in ubuntu bionic [~ppsspp/ubuntu/testing]
lcy01-amd64-028 Building amd64 build of ffmpeg 7:4.2.4-1sergeyd8.2~16.04.1 in ubuntu xenial RELEASE [~sergey-dryabzhinsky/ubuntu/ffmpeg]
lcy01-amd64-029 Building ~reviczky/texlive/+git/github recipe build in ubuntu focal [~reviczky/ubuntu/luatex-daily]
lcy01-amd64-030 Building ~mutlaqja/+git/dummy recipe build in ubuntu bionic [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-001 Building ~mutlaqja/+git/dummy recipe build in ubuntu bionic [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-002 Building ~reviczky/texlive/+git/github recipe build in ubuntu bionic [~reviczky/ubuntu/luatex-daily]
lgw01-amd64-003 Building amd64 build of 53d5570c233b8b3663242f8d94dc21ad snap package (gitea) in ubuntu bionic-updates
lgw01-amd64-004 Building amd64 build of warzone2100 snap package in ubuntu bionic-updates
lgw01-amd64-005 Building amd64 build of d6e41f8b8ec47424b4530969ecde5163 snap package (statup) in ubuntu xenial-updates
lgw01-amd64-006 Building ~mutlaqja/+git/dummy recipe build in ubuntu bionic [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-007 Building i386 build of e364d0056b593cf13aaa90c0c3687bdd snap package (statping) in ubuntu xenial-updates
lgw01-amd64-008 Building ~mutlaqja/+git/dummy recipe build in ubuntu xenial [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-009 Building ~mutlaqja/+git/dummy recipe build in ubuntu focal [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-010 Building ~mutlaqja/+git/dummy recipe build in ubuntu bionic [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-011 Building ~mutlaqja/+git/dummy recipe build in ubuntu xenial [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-012 Building amd64 build of snapcraft-digikam-74c1d5 snap package in ubuntu bionic-updates
lgw01-amd64-013 Building ~mutlaqja/+git/dummy recipe build in ubuntu focal [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-014 Building ~mutlaqja/+git/dummy recipe build in ubuntu eoan [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-015 Building ~mutlaqja/+git/dummy recipe build in ubuntu xenial [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-016 Building ~mutlaqja/+git/dummy recipe build in ubuntu bionic [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-017 Building ~mutlaqja/+git/dummy recipe build in ubuntu eoan [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-018 Building ~mutlaqja/+git/dummy recipe build in ubuntu xenial [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-019 Building ~mutlaqja/+git/dummy recipe build in ubuntu eoan [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-020 Building ~mutlaqja/+git/dummy recipe build in ubuntu xenial [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-021 Building ~mutlaqja/+git/dummy recipe build in ubuntu xenial [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-022 Building ~mutlaqja/+git/dummy recipe build in ubuntu eoan [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-023 Building ~mutlaqja/+git/dummy recipe build in ubuntu xenial [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-024 Building ~mutlaqja/+git/dummy recipe build in ubuntu groovy [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-025 Building ~mutlaqja/+git/dummy recipe build in ubuntu groovy [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-026 Building ~mutlaqja/+git/dummy recipe build in ubuntu focal [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-027 Building ~mutlaqja/+git/dummy recipe build in ubuntu eoan [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-028 Building ~mutlaqja/+git/dummy recipe build in ubuntu eoan [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-029 Building ~mutlaqja/+git/dummy recipe build in ubuntu focal [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-030 Building ~mutlaqja/+git/dummy recipe build in ubuntu focal [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-031 Building ~mutlaqja/+git/dummy recipe build in ubuntu eoan [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-032 Building ~mutlaqja/+git/dummy recipe build in ubuntu bionic [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-033 Building ~mutlaqja/+git/dummy recipe build in ubuntu bionic [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-034 Building ~mutlaqja/+git/dummy recipe build in ubuntu focal [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-035 Building ~mutlaqja/+git/dummy recipe build in ubuntu eoan [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-036 Building ~mutlaqja/+git/dummy recipe build in ubuntu groovy [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-037 Building ~mutlaqja/+git/dummy recipe build in ubuntu xenial [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-038 Building amd64 build of snapcraft-juju-c52e7e snap package in ubuntu bionic-updates
lgw01-amd64-039 Building i386 build of ffmpeg 7:4.2.4-1sergeyd8.2~16.04.1 in ubuntu xenial RELEASE [~sergey-dryabzhinsky/ubuntu/ffmpeg]
lgw01-amd64-040 Building ~mutlaqja/+git/dummy recipe build in ubuntu groovy [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-041 Building ~mutlaqja/+git/dummy recipe build in ubuntu xenial [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-042 Building ~mutlaqja/+git/dummy recipe build in ubuntu focal [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-043 Building ~mutlaqja/+git/dummy recipe build in ubuntu xenial [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-044 Building ~mutlaqja/+git/dummy recipe build in ubuntu xenial [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-045 Cleaning
lgw01-amd64-046 Building ~mutlaqja/+git/dummy recipe build in ubuntu focal [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-047 Building ~mutlaqja/+git/dummy recipe build in ubuntu xenial [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-048 Building ~mutlaqja/+git/dummy recipe build in ubuntu bionic [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-049 Building ~mutlaqja/+git/dummy recipe build in ubuntu focal [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-050 Building amd64 build of e364d0056b593cf13aaa90c0c3687bdd snap package (statping) in ubuntu xenial-updates
lgw01-amd64-051 Building ~mutlaqja/+git/dummy recipe build in ubuntu bionic [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-052 Building ~mutlaqja/+git/dummy recipe build in ubuntu focal [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-053 Building ~mutlaqja/+git/dummy recipe build in ubuntu focal [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-054 Building ~mutlaqja/+git/dummy recipe build in ubuntu groovy [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-055 Building ~mutlaqja/+git/dummy recipe build in ubuntu focal [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-056 Cleaning
lgw01-amd64-057 Building ~mutlaqja/+git/dummy recipe build in ubuntu bionic [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-058 Building ~mutlaqja/+git/dummy recipe build in ubuntu xenial [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-059 Building ~mutlaqja/+git/dummy recipe build in ubuntu focal [~mutlaqja/ubuntu/indinightly]
lgw01-amd64-060 Building ~mutlaqja/+git/dummy recipe build in ubuntu eoan [~mutlaqja/ubuntu/indinightly]

Question information

Language:
English Edit question
Status:
Solved
For:
Launchpad itself Edit question
Assignee:
No assignee Edit question
Solved by:
Rob Savoury
Solved:
Last query:
Last reply:
Revision history for this message
Manfred Hampl (m-hampl) said :
#1

Adding another item to this topic:
According to https://lists.ubuntu.com/archives/ubuntu-announce/2020-July/000258.html Ubuntu eoan is already EOL.
Why are the builders still accepting build tasks for eoan packages?

Revision history for this message
Kristian Glass (doismellburning) said :
#2

Unfortunately staggering the builds isn't currently something we can currently do particularly easily

What I *have* done though is drop the build score down to -2600 (same as builds for copy archives - https://help.launchpad.net/Packaging/BuildScores) from -2, which should help ensure that other individual builds are prioritised over it - let's see how that goes. Thanks

Revision history for this message
Colin Watson (cjwatson) said :
#3

I'd like to explicitly point out here that the schedule is not under the control of the user who owns the recipes, and the recipes are legitimate (though there are certainly many of them!). They simply have their recipes set to build automatically, and Launchpad decides when to dispatch builds. Until quite recently it was technically infeasible to stagger the builds, because it would have seriously exacerbated some inefficiencies in buildd-manager and probably pushed it over a performance cliff. I made some improvements there recently that should now make it feasible, but it's still a slightly complicated set of changes we'd need to make to Launchpad itself.

Regarding eoan, some aspects of its end-of-life process are as yet incomplete: in particular, the Ubuntu team hasn't marked that series as Obsolete in Launchpad yet.

Revision history for this message
Colin Watson (cjwatson) said :
#4

Oh, one more thing to point out: there was a partial buildd-manager outage last night due to https://bugs.launchpad.net/launchpad/+bug/1889018, which made the recipe build usage look a bit worse than usual because the build farm acquired a slight backlog as a result.

Revision history for this message
Rob Savoury (savoury1) said :
#5

Thank you all for the extra details and explanations, including relative to the capabilities of the Launchpad build system when it comes to staggering of recipe builds. It sounds like it is all working as well as it can be within the known technical limitations (lowering of the build score for the recipes in question does sound like a good way to help other tasks flow through simultaneously), and I do sincerely appreciate the work and effort that you all put into this excellent Launchpad service!