Launchpad itself

launchpad builders stuck in cleaning

Asked by Manfred Hampl on 2021-01-04

It seems to me that some of the launchpad builders have run into problems. Several of them are - already for an extended period - shown as "Cleaning" or "Cleaning since ..." on https://launchpad.net/builders

The full history of one of them https://launchpad.net/builders/lcy01-amd64-006/+history tells that is has not built a single package since 2020-05-20, and few others like lgw01-amd64-014 did the last build on 2020-11-10.

amd/i386 builders currently are no problem, but the queue for arm packages is quite long (20k jobs for arm64, estimated 2 days), and also some arm builders show strange behavior, e.g. https://launchpad.net/builders/bos02-arm64-015/+history tells for the most recent build "... finished 9 hours ago ...". Can a cleaning action take 9 hours, or why didn't that builder start a new job?

Similar for bos02-arm64-017, bos02-arm64-020, bos02-arm64-022, bos02-arm64-045, bos02-arm64-048, bos02-arm64-051, bos02-arm64-058, bos02-arm64-076, bos02-arm64-078

Isn't this something that should be investigated?

Question information

Language:
English Edit question
Status:
Solved
For:
Launchpad itself Edit question
Assignee:
No assignee Edit question
Solved by:
Manfred Hampl
Solved:
Last query:
Last reply:
Revision history for this message
Colin Watson (cjwatson) said : 		#1

We should indeed look into this, but the repairs can be quite time-consuming so we don't normally worry about a relatively small number of permanently-failing builder resets. (The bos02-arm64-* ones you listed were due to a transient failure that's since been fixed.)

We'll try to sort out the amd64 cases soonish - probably just stuck Neutron ports.

Revision history for this message
Manfred Hampl (m-hampl) said : 		#2

Thank you, Colin Watson.
If the issue is already known by the responsible people, then there is no need for keeping this question open.