launchpad builders stuck in cleaning

Asked by Manfred Hampl on 2021-01-04

It seems to me that some of the launchpad builders have run into problems. Several of them are - already for an extended period - shown as "Cleaning" or "Cleaning since ..." on

The full history of one of them tells that is has not built a single package since 2020-05-20, and few others like lgw01-amd64-014 did the last build on 2020-11-10.

amd/i386 builders currently are no problem, but the queue for arm packages is quite long (20k jobs for arm64, estimated 2 days), and also some arm builders show strange behavior, e.g. tells for the most recent build "... finished 9 hours ago ...". Can a cleaning action take 9 hours, or why didn't that builder start a new job?

Similar for bos02-arm64-017, bos02-arm64-020, bos02-arm64-022, bos02-arm64-045, bos02-arm64-048, bos02-arm64-051, bos02-arm64-058, bos02-arm64-076, bos02-arm64-078

Isn't this something that should be investigated?

Question information

English Edit question
Launchpad itself Edit question
No assignee Edit question
Solved by:
Manfred Hampl
Last query:
Last reply:
Revision history for this message
Colin Watson (cjwatson) said :

We should indeed look into this, but the repairs can be quite time-consuming so we don't normally worry about a relatively small number of permanently-failing builder resets. (The bos02-arm64-* ones you listed were due to a transient failure that's since been fixed.)

We'll try to sort out the amd64 cases soonish - probably just stuck Neutron ports.

Revision history for this message
Manfred Hampl (m-hampl) said :

Thank you, Colin Watson.
If the issue is already known by the responsible people, then there is no need for keeping this question open.