Memory exhaustion when linking even with explicit no-parallel

Asked by Rob Savoury

Attempting to build latest Telegram 3.5.2 for Focal and earlier today has resulted in insurmountable failures due memory exhaustion when doing final linking. This is even with re-uploads explicitly specifying "no-parallel" for the builds. For instance in the log of the latest attempted build with Focal [1] is the following pointing directly to memory exhaustion while linking:

collect2: fatal error: ld terminated with signal 9 [Killed]
compilation terminated.

The build is being done with GCC 10.3.0 for all series Focal and earlier. Curiously, the build succeeded for Jammy [2] with parallel=2 and this build was with GCC 11.2.0 as is now the default for Jammy.

Is there anything that can be done on the Launchpad buildd side of things such that the failing builds can succeed? Thanks.

[1] https://launchpad.net/~savoury1/+archive/ubuntu/telegram/+build/23224419
[2] https://launchpad.net/~savoury1/+archive/ubuntu/telegram/+build/23224205

Question information

Language:
English Edit question
Status:
Solved
For:
Launchpad itself Edit question
Assignee:
No assignee Edit question
Solved by:
Colin Watson
Solved:
Last query:
Last reply:
Revision history for this message
Rob Savoury (savoury1) said :
#1

Anyone have any suggestions about this memory issue? Here's some more details of interest:

* Local build system used has 16GB RAM only (these days considered somewhat low for building modern software)
* Even with 16GB only Telegram 3.5.2 can build using full parallelism (8 threads in this instance, based on CPU design)
* Usable RAM after OS usage is actually only ~14GB on the local system, which suggests Launchpad buildd allocates less

As I have asked in a previous question here relative Launchpad but didn't receive an answer, as main question was closed:

* Specifically how much virtual RAM does each Launchpad buildd get allocated for the builds?

It seems that it is clearly < 14GB RAM per buildd which seems low for this era (2022) building modern software.

Can someone please reply to this issue with any suggestions about how I might get these builds to succeed?

Thank you.

Revision history for this message
Best Colin Watson (cjwatson) said :
#2

Builder VMs currently have 8 GiB of RAM and 4 GiB of swap. (This is intended to be consistent across all architectures, but there are currently some weird OpenStack problems that cause arm64/armhf/ppc64el builders to come up with no swap; see https://answers.launchpad.net/launchpad/+question/694428.)

We've discussed increasing the RAM allocation to something larger like 16 or 32 GiB. However, the current build farm design is such that all VMs are homogeneous, so before we can do this we need to make sure that we have enough physical RAM on the relevant compute nodes to do this across the board for all architectures. (If we don't, we'd have to reduce the number of builders, which we'd prefer to avoid as it would decrease overall build farm throughput.)

As a stopgap measure, there are various linker options that can change the linker's trade-off between memory and time, such as -Wl,--reduce-memory-overheads, and it may be worth trying those.

Revision history for this message
Rob Savoury (savoury1) said :
#3

Hi Colin,

Thank you for the detailed answer and suggestion about the linking. As I have not really explored that aspect of building software for Linux-based systems very much I admit my knowledge there is still fairly limited! So the builds now succeeded using the ld --reduce-memory-overheads option as well as making the build use other linking options already in debian/rules that I had not noticed which were being used for i386 but not amd64.

Again, appreciate the response, all the details about the current buildd VM RAM allocation and the issues surrounding possibly increasing that amount, as well as the suggestion for the linker options.

Cheers, Rob