Ubuntu
rsyslog package

How to set rate limit of syslog?

Asked by Xidorn Quan

I noticed an incident that Nautilus for some reason wrote 30GB of log in syslog within minutes, and based on my analysis, it writes >200k lines of log per second.

This raises a question that why there is no rate limit put on syslog, so I tried to setup it myself. But unfortunately I failed to find any way to make rate limit works.

How I'm testing:
* get code of syslog_caller from rsyslog repo at https://github.com/rsyslog/rsyslog/blob/3bee383945c6954f023881141bf7c769b2802639/tests/syslog_caller.c
* remove the `#include <config.h>` then compile it with gcc
* run `./syslog_caller -m 1000` to log 1000 log messages

Methods I have tried but doesn't work:
* put `$SystemLogRateLimitInterval 5` and `$SystemLogRateLimitBurst 200` in rsyslog.conf (rsyslog reports error for this)
* put `SysSock.RateLimit.Interval="5" SysSock.RateLimit.Burst="200"` in `module(load="imuxsock")`
* add a new `input(type="imuxsock" Socket="" RateLimit.Interval="5" RateLimit.Burst="200")` after `module(load="imuxsock")`
* put `$imuxsockRateLimitInterval 5` and `$imuxsockRateLimitBurst 200`

I've also tried adding corresponding severity parameter set to exactly 6 in each of the cases, and I executed `sudo systemctl restart rsyslog.service` each time after I changed the config. But none of them works.

How can I restrict the logging rate limit?

Question information

Language:
English Edit question
Status:
Solved
For:
Ubuntu rsyslog Edit question
Assignee:
No assignee Edit question
Solved by:
actionparsnip
Solved:
Last query:
Last reply:
Revision history for this message
actionparsnip (andrew-woodhead666) said : 		#1

Surely you'd want to catch all logs made whenever they are made. Instead of restricting the log limit I suggest you fix the thing making the logs. The logs tell you there is an issue.

Revision history for this message
Xidorn Quan (upsuper) said : 		#2

I don't. If an application is flooding with 200k lines of logs, the content of logs is very unlikely to be important after the first few.

It's true that the thing should be fixed, but a restriction should also be imposed from the logging system to prevent such accidental log flood, which you can never predict before it happens.

Revision history for this message
actionparsnip (andrew-woodhead666) said : 		#3

I suggest you report a bug. Mark it as a feature request

Revision history for this message
Best actionparsnip (andrew-woodhead666) said : 		#4

This may help
https://www.rsyslog.com/doc/v8-stable/configuration/modules/imuxsock.html

But you should fix the application making the logs. You should also have the storage of the logs monitored so that you can predict it filling and can jump on if used space suddenly spikes.

Revision history for this message
actionparsnip (andrew-woodhead666) said : 		#5

What is the output of:

lsb_release -a; uname -a; apt-cache policy nautilus

and what are the entries in syslog that you are being spammed with?

Revision history for this message
Xidorn Quan (upsuper) said : 		#6

> This may help
> https://www.rsyslog.com/doc/v8-stable/configuration/modules/imuxsock.html

I did try this, and it didn't seem to work before, but as I tried this time, it seems to work now. The reason is probably that it truncate the middle, and reserves burst number of logs at the start and the end, which may make it look not work properly.

Anyway, thanks. That solves my problem. I can create a separate bug report for the nautilus if you think that makes sense.

Revision history for this message
Xidorn Quan (upsuper) said : 		#7

Thanks actionparsnip, that solved my question.

Revision history for this message
Xidorn Quan (upsuper) said : 		#8

The output of that command is

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 19.10
Release: 19.10
Codename: eoan
Linux ##### 5.3.0-29-generic #31-Ubuntu SMP Fri Jan 17 17:27:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
nautilus:
  Installed: 1:3.34.1-1ubuntu1
  Candidate: 1:3.34.1-1ubuntu1
  Version table:
 *** 1:3.34.1-1ubuntu1 500
        500 http://au.archive.ubuntu.com/ubuntu eoan/main amd64 Packages
        100 /var/lib/dpkg/status

That incident happens two weeks ago, not sure whether anything has been updated.

For the log, I think I've since lost the log, but I recalled there are things related to VDPAU not correctly restored after wake up. I'm suspecting it's actually from VLC rather than nautilus since I configured VLC to use VDPAU for video playing.

Revision history for this message
Xidorn Quan (upsuper) said : 		#9

Oh, wait, I recorded some samples elsewhere:
> Jan 11 16:20:34 ########### org.gnome.Nautilus[2392]: [00007fbc7032eb90] vdpau_chroma filter error: video mixer rendering failure: The display was pre-empted, or a fatal error occurred.
> Jan 11 16:20:34 ########### org.gnome.Nautilus[2392]: [00007fbc7032eb90] vdpau_chroma filter error: video mixer features failure: An invalid handle value was provided.
> Jan 11 16:20:34 ########### org.gnome.Nautilus[2392]: [00007fbc7032eb90] vdpau_chroma filter error: video mixer attributes failure: An invalid handle value was provided.
> Jan 11 16:20:34 ########### org.gnome.Nautilus[2392]: [00007fbc7032eb90] vdpau_chroma filter error: video mixer rendering failure: An invalid handle value was provided.
> Jan 11 16:20:34 ########### org.gnome.Nautilus[2392]: [00007fbc7032eb90] vdpau_chroma filter error: video mixer features failure: An invalid handle value was provided.
> Jan 11 16:20:34 ########### org.gnome.Nautilus[2392]: [00007fbc7032eb90] vdpau_chroma filter error: video mixer attributes failure: An invalid handle value was provided.
> Jan 11 16:20:34 ########### org.gnome.Nautilus[2392]: [00007fbc7032eb90] vdpau_chroma filter error: video mixer rendering failure: An invalid handle value was provided.
> Jan 11 16:20:34 ########### org.gnome.Nautilus[2392]: [00007fbc7032eb90] vdpau_chroma filter error: video mixer features failure: An invalid handle value was provided.
> Jan 11 16:20:34 ########### org.gnome.Nautilus[2392]: [00007fbc7032eb90] vdpau_chroma filter error: video mixer attributes failure: An invalid handle value was provided.
> Jan 11 16:20:34 ########### org.gnome.Nautilus[2392]: [00007fbc7032eb90] vdpau_chroma filter error: video mixer rendering failure: An invalid handle value was provided.