Extremely high memory consumption under heavy workload

After some analysis I believe the very high default value of LogDebugHistory (99999) is to blame for this (attaching massif report from the test case).

With a high number of jobs it's very easy to get hunderds-thousands of messages in LogDebugHistory which in turn will lead to enormous memory consumption, e.g.
40000 jobs * 2000 lines * (16 bytes for cupsd_joblog_t structure + 50-60 chars per message on average) - 5.66 GB.

After setting LogDebugHistory to 0 the memory consumption never exceeded 300 MB in my tests under the same heavy load. That's a very significant difference.

The possible workarounds:
1. Decrease LogDebugHistory to a low value (even to 0, upstream it's set to 200 by default).
2. Change LogLevel to debug (it will slow down cupsd operation significantly, but at the price of lower memory consumption as the LogDebugHistory is unused).
3. Make sure jobs are cleaned periodically by setting e.g. PreserveJobHistory 300.

I am wondering what was the rationale behind increasing LogDebugHistory to such a high value.

Status changed to 'Confirmed' because the bug affects multiple users.

In a recent test with MaxJobs of 40000 I was easily able to reach over 65 GB of memory consumption within 60 mins, while with LogDebugHistory set to 0 the consumption was below 400 MB.

My recommendation is to lower the default LogDebugHistory value back to what is upstream: 200.

I have raised the default LogDebugHistory some years ago as most printing problems happen during the execution of a job and this way one gets a logging with debug verbosity only for jobs and standard verbosity for all the rest. This way one often did not need to ask the bug-reporting users to reproduce their problems after having set logging to debug mode but instead, one could simply start to work with the error_log auto-captured by Apport.

Up to now no one complained that it somehow overloaded system resources.

Please add a line "LogDebugHistory 200" to your /etc/cups/cupsd.conf and restart CUPS.

I will remove this change from the next release of the CUPS package.

OdyX, could you remove the LogDebugHistory=99999 default change from the CUPS package for Debian (remove log-debug-history-nearly-unlimited.patch)? Thanks.

Thank you for the explanation Till.

I believe the lack of previous complaints is because this happens in a pretty unique conditions (very high MaxJobs and large batches of jobs). The test case however was based on a real-life scenario reported to me.

This bug was fixed in the package cups - 2.2.11-0ubuntu1

---------------
cups (2.2.11-0ubuntu1) eoan; urgency=medium

  * New upstream release
  * Removed 10 patches taken from upstream.
  * Refreshed 13 patches with quilt.
  * Removed patch to set "LogDebugHistory 99999" as it causes high
    system load when there are very many jobs (LP: #1831021).
  * Set "DirtyCleanInterval 0" in CUPS configuration to avoid losses of
    jobs when cupsd gets killed (LP: #1830022).

 -- Till Kamppeter <email address hidden> Mon, 05 Aug 2019 11:40:10 +0200