recipe builds fails randomly due to GIT fetch

Asked by Jasem Mutlaq on 2018-12-20

My recipes builds are failing randomly for several projects based on GIT.

It fails like 75% of the time.. sometimes when I retrigger the build, it goes through and other times it fails.

This is one example for a recipe:

# git-build-recipe format 0.4 deb-version {debupstream}~{time}
lp:~mutlaqja/+git/dummy master
nest-part src lp:libindi 3rdparty/indi-ffmv indi-ffmv
nest-part cmake lp:libindi cmake_modules indi-ffmv/cmake_modules
nest-part pack lp:libindi debian/indi-ffmv debian

Here are some logs:

and more....

Is there a reason why so many fail?? It appears totally random.

Some of the projects:

Question information

English Edit question
Launchpad itself Edit question
No assignee Edit question
Solved by:
Jasem Mutlaq
Last query:
Last reply:
Colin Watson (cjwatson) said : #1

This is happening because a very large number of your recipe builds are dispatched almost simultaneously, and the effect is to overload the git hosting service about once a day (not entirely due to your recipes but pretty close). Sometimes I notice and cancel the builds, and sometimes I don't and it sends the git service into a load spiral that takes some time to recover from. Either way you'll see somewhat random client-side failures.

We're working on scaling up the git service as a priority, although at this point it may not be done until the new year.

Jasem Mutlaq (mutlaqja) said : #2

Alright thanks, I will try to build one at a time for those that are built-on-request. However, I also have nightlies that I have no control on when they start.

Looking forward to the Git service scaling, thanks!

Colin Watson (cjwatson) said : #3

You should find this to be working rather better for you now. We still have more scaling work in progress, but simply doubling the RAM has helped a fair bit.