systemd-journald RateLimitBurst is sometimes divided by 4
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| systemd (Ubuntu) |
Fix Released
|
Medium
|
Unassigned | ||
| Xenial |
Fix Released
|
Medium
|
Unassigned | ||
Bug Description
[Impact]
systemd-journald allows you to configure a per-service journal rate limit in /etc/systemd/
code that effectively increases the rate limit when there is a lot of disk space available.
However, all versions of systemd before v232 had a bug which would shrink the rate limit
when there is between 1 and 16 MB available on disk.
If you designed a service to log at a rate R and configured RateLimitBurst to a little above R, this can lead to loss of logs when free disk is between 1 and 16 MB, as your service will be surprisingly rate limited at lower than your configured rate.
This bug was fixed upstream in https:/
It is a straightforward one-line change that makes the code match the comments under it.
[Test Case]
Run a systemd service that prints lots of logs (eg `yes`). Fill your disk to have only 1MB full. Use journalctl to see how many log lines are between "Suppressed" lines. Note that it is 1/4 of what you'd expect. (Admittedly this test case is a little hard to achieve since journald itself is writing to disk. I did run into this in production.)
[Regression Potential]
This does mean that journald can write slightly more to disk than it did before when free disk is between 1 and 16MB, but given that the full burst rate is available below 1MB it seems unlikely that any systems are depending on this change in order to not break.
The fix has been in systemd since v232 (shipped in Zesty). I would like to see it in Xenial.
[Other Info]
I am seeing this on:
ubuntu@
Description: Ubuntu 16.04.3 LTS
Release: 16.04
ubuntu@
systemd:
Installed: 229-4ubuntu19
Candidate: 229-4ubuntu21
Version table:
229-4ubuntu21 500
500 http://
*** 229-4ubuntu19 100
100 /var/lib/
229-4ubuntu10 500
500 http://
229-4ubuntu4 500
500 http://
This bug seems to date back to the original implementation of rate limiting (https:/
| David Glasser (glasser) wrote | #1 |
| David Glasser (glasser) wrote | #2 |
| David Glasser (glasser) wrote | #3 |
| Changed in systemd (Ubuntu): | |
| status: | New → Fix Released |
| David Glasser (glasser) wrote | #4 |
| Changed in systemd (Ubuntu Xenial): | |
| status: | New → In Progress |
| Changed in systemd (Ubuntu): | |
| importance: | Undecided → Medium |
| Changed in systemd (Ubuntu Xenial): | |
| importance: | Undecided → Medium |
| Brian Murray (brian-murray) wrote Please test proposed package | #5 |
| Changed in systemd (Ubuntu Xenial): | |
| status: | In Progress → Fix Committed |
| tags: | added: verification-needed verification-needed-xenial |
| Dimitri John Ledkov (xnox) wrote | #6 |
| tags: |
added: verification-done verification-done-xenial removed: verification-needed verification-needed-xenial |
| Launchpad Janitor (janitor) wrote | #7 |
| Changed in systemd (Ubuntu Xenial): | |
| status: | Fix Committed → Fix Released |
| Łukasz Zemczak (sil2100) wrote Update Released | #8 |
Correction: I believe this occurs when the amount of remaining space until journald hits its allocated limit is between 1 and 16MB, not when the entire filesystem has that little space left. (This makes it much more likely to occur: any system that is using logs enough for them to be rotated due to space will hit this issue whenever it's near rotation time.)