systemd 100% cpu usage apport-autoreport.service: Failed with result 'start-limit-hit'

Bug #1891657 reported by Scott Stensland
166
This bug affects 34 people
Affects Status Importance Assigned to Milestone
apport (Ubuntu)
Fix Released
Undecided
Brian Murray
Groovy
Fix Released
Undecided
Brian Murray
systemd (Ubuntu)
Fix Released
High
Balint Reczey
Groovy
Fix Released
High
Balint Reczey

Bug Description

after upgrade from Ubuntu 20.04 to 20.10
seeing systemd use 100% cpu forever

 top

 top - 10:16:19 up 20 min, 1 user, load average: 3.91, 4.28, 3.39
 Tasks: 332 total, 2 running, 330 sleeping, 0 stopped, 0 zombie
 %Cpu(s): 18.7 us, 6.4 sy, 11.9 ni, 62.2 id, 0.0 wa, 0.0 hi, 0.8 si, 0.0 st
 MiB Mem : 15917.6 total, 8553.4 free, 2604.3 used, 4759.9 buff/cache
 MiB Swap: 2048.0 total, 2048.0 free, 0.0 used. 12633.7 avail Mem

     PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
       1 root 20 0 177316 12252 8684 R 100.0 0.1 15:01.25 systemd
    4636 kush 39 19 633796 176152 16360 S 98.3 1.1 17:51.56 tracker-miner-f
     820 message+ 20 0 10740 6452 4212 S 39.8 0.0 6:19.06 dbus-daemon
     316 root 19 -1 535916 322896 320824 S 15.3 2.0 3:16.41 systemd-journal
     844 syslog 20 0 221136 5840 3920 S 13.6 0.0 2:14.52 rsyslogd
     858 root 20 0 83708 74124 7568 S 12.7 0.5 2:06.98 systemd-logind
       5 root 20 0 0 0 0 I 3.4 0.0 0:05.01 kworker/0:0-events
   35566 kush 20 0 3850088 588344 262192 S 2.5 3.6 0:30.48 MainThread
    5626 kush 20 0 5004788 260876 93400 S 1.7 1.6 3:05.19 gnome-shell
    7033 kush 20 0 869792 66440 44272 S 1.7 0.4 0:06.05 gnome-terminal-
   36790 kush 20 0 2525432 156636 98568 S 1.7 1.0 0:04.99 Web Content
     957 root 20 0 1556196 45044 24284 S 0.8 0.3 0:02.48 containerd
    1038 root 20 0 10460 4412 3300 S 0.8 0.0 0:00.16 fancontrol

 sudo tail -n 100 /var/log/syslog
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Start request repeated too quickly.
 Aug 14 10:14:17 lhotse systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
 Aug 14 10:14:17 lhotse systemd[1]: Failed to start Process error reports when automatic reporting is enabled.

ProblemType: Bug
DistroRelease: Ubuntu 20.10
Package: apport 2.20.11-0ubuntu44
ProcVersionSignature: Ubuntu 5.4.0-44.48-generic 5.4.55
Uname: Linux 5.4.0-44-generic x86_64
NonfreeKernelModules: wl
ApportLog:

ApportVersion: 2.20.11-0ubuntu44
Architecture: amd64
CasperMD5CheckResult: skip
CrashReports: 640:0:125:652288:2020-08-13 16:51:21.222003836 -0400:2020-08-13 16:52:08.772174518 -0400:/var/crash/bcmwl-kernel-source.0.crash
Date: Fri Aug 14 10:15:01 2020
InstallationDate: Installed on 2020-04-23 (112 days ago)
InstallationMedia: Ubuntu 20.04 LTS "Focal Fossa" - Alpha amd64 (20200309)
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: apport
UpgradeStatus: Upgraded to groovy on 2020-08-13 (0 days ago)

Revision history for this message
Scott Stensland (scottstensland) wrote :
Revision history for this message
Scott Stensland (scottstensland) wrote :
Revision history for this message
Scott Stensland (scottstensland) wrote :

I am seeing this issue on multiple different laptops
which I upgraded in past 24 hours from Ubuntu 20.04 to 20.10

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apport (Ubuntu):
status: New → Confirmed
Revision history for this message
Ilia (iliaspice) wrote :

CPU usage 100%
/var/log/syslog now takes more than 10GiB and grows...

Revision history for this message
Aaron Lichtman (alichtman) wrote :

I am experiencing this issue on Ubuntu 20.10.

Revision history for this message
Aaron Lichtman (alichtman) wrote :

$ systemctl status appoprt-autoreport.service
● apport-autoreport.service - Process error reports when automatic reporting is enabled
     Loaded: loaded (/lib/systemd/system/apport-autoreport.service; static)
     Active: activating (start) since Sat 2020-08-15 12:43:54 CDT; 6ms ago
TriggeredBy: ● apport-autoreport.path
   Main PID: 8019 (whoopsie-upload)
      Tasks: 1 (limit: 38362)
     Memory: 0B
     CGroup: /system.slice/apport-autoreport.service
             └─8019 /usr/bin/python3 /usr/share/apport/whoopsie-upload-all

Aug 15 12:43:54 arctic systemd[1]: Starting Process error reports when automatic reporting is enabled...
Aug 15 12:43:54 arctic whoopsie-upload-all[8019]: /var/crash/_usr_bin_blueman-assistant.1000.crash already marked for upload, skipping
Aug 15 12:43:54 arctic whoopsie-upload-all[8019]: /var/crash/_usr_share_spotify_spotify.1000.crash already marked for upload, skipping
Aug 15 12:43:54 arctic whoopsie-upload-all[8019]: /var/crash/_usr_share_discord_Discord.1000.crash already marked for upload, skipping
Aug 15 12:43:54 arctic whoopsie-upload-all[8019]: All reports processed
Aug 15 12:43:54 arctic systemd[1]: apport-autoreport.service: Succeeded.
Aug 15 12:43:54 arctic systemd[1]: Finished Process error reports when automatic reporting is enabled.
Aug 15 12:43:54 arctic systemd[1]: Starting Process error reports when automatic reporting is enabled...
Aug 15 12:43:54 arctic whoopsie-upload-all[8029]: /var/crash/_usr_bin_blueman-assistant.1000.crash already marked for upload, skipping
Aug 15 12:43:54 arctic whoopsie-upload-all[8029]: /var/crash/_usr_share_spotify_spotify.1000.crash already marked for upload, skipping
Aug 15 12:43:54 arctic whoopsie-upload-all[8029]: /var/crash/_usr_share_discord_Discord.1000.crash already marked for upload, skipping
Aug 15 12:43:54 arctic whoopsie-upload-all[8029]: All reports processed
Aug 15 12:43:54 arctic systemd[1]: apport-autoreport.service: Succeeded.
Aug 15 12:43:54 arctic systemd[1]: Finished Process error reports when automatic reporting is enabled.
Aug 15 12:43:54 arctic systemd[1]: Starting Process error reports when automatic reporting is enabled...

Revision history for this message
Aaron Lichtman (alichtman) wrote :

This fix, published by the reporter of this bug, worked for me.

https://askubuntu.com/a/1267204

I only needed to follow the steps up to the first reboot.

Revision history for this message
Michael Neuffer (neuffer) wrote :

I also had to purge apport in order to get this under control.

Revision history for this message
Brian Murray (brian-murray) wrote :

Could somebody actively suffering from this problem include the output of 'sudo service whoopsie status'? Thanks in advance.

It's also worth noting that the apport-autoreport.service is provided by the apport-noui package and removing that package would be a better way to work around this problem then removing all of apport.

Revision history for this message
taff (cn65) wrote :

I am suffering from this problem.
Output from 'sudo service whoospie status':

tags: added: rls-gg-incoming
Revision history for this message
Gannet (ken20001) wrote :

The same shit on 20.10. Syslog becomes bigger in a real time, then it eats all free space on my /home and system becomes completely unusable.

$ sudo service whoopsie status
[sudo] пароль до eugene:
● whoopsie.service - crash report submission daemon
     Loaded: loaded (/lib/systemd/system/whoopsie.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-09-10 00:38:34 EEST; 18min ago
   Main PID: 851 (whoopsie)
      Tasks: 3 (limit: 16717)
     Memory: 3.4M
     CGroup: /system.slice/whoopsie.service
             └─851 /usr/bin/whoopsie -f

вер 10 00:38:34 p5q3 whoopsie[851]: [00:38:34] Using lock path: /var/lock/whoopsie/lock
вер 10 00:38:34 p5q3 whoopsie[851]: [00:38:34] offline
вер 10 00:38:34 p5q3 systemd[1]: Started crash report submission daemon.

Revision history for this message
Rocko (rockorequin) wrote :

I ran into this and applied the workaround of deleting all the files in /var/crash to stop the journalctl spam and 100% CPU usage.

Revision history for this message
Brian Murray (brian-murray) wrote :

The output of both of the following commands would help us sort out what is going on.

journalctl -b -u whoopsie

journalctl -b -u apport-autoreport

Thanks! Additionally, if this happening on a release other than Ubuntu 20.10 please let me know.

Changed in apport (Ubuntu):
assignee: nobody → Brian Murray (brian-murray)
Revision history for this message
Nicolas Bock (nicolasbock) wrote :
Download full text (3.8 KiB)

$ journalctl -b -u apport-autoreport
Sep 14 13:01:52 rubberducky systemd[1]: Starting Process error reports when automatic reporting is enabled...
Sep 14 13:02:07 rubberducky whoopsie-upload-all[145261]: Collecting info for /var/crash/_usr_bin_gnome-shell.1000.crash...
Sep 14 13:02:07 rubberducky whoopsie-upload-all[145261]: Marking /var/crash/_usr_bin_gnome-shell.1000.crash for whoopsie upload
Sep 14 13:02:07 rubberducky whoopsie-upload-all[145261]: All reports processed
Sep 14 13:02:07 rubberducky systemd[1]: apport-autoreport.service: Succeeded.
Sep 14 13:02:07 rubberducky systemd[1]: Finished Process error reports when automatic reporting is enabled.
Sep 14 13:02:07 rubberducky systemd[1]: Starting Process error reports when automatic reporting is enabled...
Sep 14 13:02:07 rubberducky whoopsie-upload-all[147307]: /var/crash/_usr_bin_gnome-shell.1000.crash already marked for upload, skipping
Sep 14 13:02:07 rubberducky whoopsie-upload-all[147307]: All reports processed
Sep 14 13:02:07 rubberducky systemd[1]: apport-autoreport.service: Succeeded.
Sep 14 13:02:07 rubberducky systemd[1]: Finished Process error reports when automatic reporting is enabled.
Sep 14 13:02:07 rubberducky systemd[1]: Starting Process error reports when automatic reporting is enabled...
Sep 14 13:02:07 rubberducky whoopsie-upload-all[147310]: /var/crash/_usr_bin_gnome-shell.1000.crash already marked for upload, skipping
Sep 14 13:02:07 rubberducky whoopsie-upload-all[147310]: All reports processed
Sep 14 13:02:07 rubberducky systemd[1]: apport-autoreport.service: Succeeded.
Sep 14 13:02:07 rubberducky systemd[1]: Finished Process error reports when automatic reporting is enabled.
Sep 14 13:02:07 rubberducky systemd[1]: Starting Process error reports when automatic reporting is enabled...
Sep 14 13:02:07 rubberducky whoopsie-upload-all[147313]: /var/crash/_usr_bin_gnome-shell.1000.crash already marked for upload, skipping
Sep 14 13:02:07 rubberducky whoopsie-upload-all[147313]: All reports processed
Sep 14 13:02:07 rubberducky systemd[1]: apport-autoreport.service: Succeeded.
Sep 14 13:02:07 rubberducky systemd[1]: Finished Process error reports when automatic reporting is enabled.
Sep 14 13:02:07 rubberducky systemd[1]: Starting Process error reports when automatic reporting is enabled...
Sep 14 13:02:07 rubberducky whoopsie-upload-all[147315]: /var/crash/_usr_bin_gnome-shell.1000.crash already marked for upload, skipping
Sep 14 13:02:07 rubberducky whoopsie-upload-all[147315]: All reports processed
Sep 14 13:02:07 rubberducky systemd[1]: apport-autoreport.service: Succeeded.
Sep 14 13:02:07 rubberducky systemd[1]: Finished Process error reports when automatic reporting is enabled.
Sep 14 13:02:07 rubberducky systemd[1]: Starting Process error reports when automatic reporting is enabled...
Sep 14 13:02:07 rubberducky whoopsie-upload-all[147317]: /var/crash/_usr_bin_gnome-shell.1000.crash already marked for upload, skipping
Sep 14 13:02:07 rubberducky whoopsie-upload-all[147317]: All reports processed
Sep 14 13:02:07 rubberducky systemd[1]: apport-autoreport.service: Succeeded.
Sep 14 13:02:07 rubberducky systemd[1]: Finished Process error reports when automatic repo...

Read more...

Revision history for this message
Nicolas Bock (nicolasbock) wrote :

The error message (the last 3 lines) keep repeating ad infinitum until I manually remove the crash files in /var/crash as Rocky notes in https://bugs.launchpad.net/ubuntu/+source/apport/+bug/1891657/comments/13.

tags: removed: rls-gg-incoming
Revision history for this message
Brian Murray (brian-murray) wrote :

I actually think this is either a bug with systemd or a change in how it handles the services which apport-autoreport includes. I've tested this by changing /usr/lib/systemd/system/apport-autoreport.service so that it contains the following:

ExecStart=/usr/bin/aplay /usr/share/sounds/sound-icons/canary-long.wav

Then if I touch a .crash file in /var/crash e.g. touch '/var/crash/canary.crash I ended up hearing the canary sound 5 times then a pause and then the canary sound multiple times again. On Ubuntu 20.04 the same test case results in the canary sound being heard only one time.

Revision history for this message
Rocko (rockorequin) wrote :

Doesn't systemd normally stop trying to start a service if it fails too often? That's clearly not happening here, though.

Balint Reczey (rbalint)
Changed in systemd (Ubuntu Groovy):
assignee: nobody → Balint Reczey (rbalint)
status: New → In Progress
Balint Reczey (rbalint)
Changed in systemd (Ubuntu Groovy):
status: In Progress → Confirmed
Revision history for this message
Balint Reczey (rbalint) wrote :

Apport keeps reshooting apport-autoreport.service as long as there are files like /var/crash/*.crash:

# cat /lib/systemd/system/apport-autoreport.path
[Unit]
Description=Process error reports when automatic reporting is enabled (file watch)
ConditionPathExists=/var/lib/apport/autoreport

[Path]
PathExistsGlob=/var/crash/*.crash

[Install]
WantedBy=paths.target

# cat /lib/systemd/system/apport-autoreport.service
[Unit]
Description=Process error reports when automatic reporting is enabled
ConditionPathExists=/var/lib/apport/autoreport
Wants=whoopsie.service
After=whoopsie.service

[Service]
Type=oneshot
ExecStart=/usr/share/apport/whoopsie-upload-all

If the service succeeds there is no reason for systemd to not restart it.

IMO apport should check the files once after the boot finised and monitor /var/crash/ for _changes_ only while the system is running.

I still check if something could be done on systemd side.

Revision history for this message
Nicolas Bock (nicolasbock) wrote :

When I run the whoopsie upload script I get:

$ sudo /usr/share/apport/whoopsie-upload-all
/var/crash/_usr_bin_emacs-gtk.1000.crash already marked for upload, skipping
All reports processed

Why is it skipping the crash?

Revision history for this message
Nicolas Bock (nicolasbock) wrote :

After deleting the files *.upload{,ed} I can run

$ sudo /usr/share/apport/whoopsie-upload-all
/var/crash/_usr_bin_emacs-gtk.1000.crash already has info collected
Marking /var/crash/_usr_bin_emacs-gtk.1000.crash for whoopsie upload
All reports processed

And get the two files back. The content of the uploaded file is

$ sudo cat _usr_bin_emacs-gtk.1000.uploaded
1c9b8196-f9f4-11ea-b6f1-fa163ee63de6

Is this procedure supposed to delete the crash file at some point?

Revision history for this message
Brian Murray (brian-murray) wrote : Re: [Bug 1891657] Re: systemd 100% cpu usage apport-autoreport.service: Failed with result 'start-limit-hit'

On Fri, Sep 18, 2020 at 08:56:29PM -0000, Nicolas Bock wrote:
> When I run the whoopsie upload script I get:
>
> $ sudo /usr/share/apport/whoopsie-upload-all
> /var/crash/_usr_bin_emacs-gtk.1000.crash already marked for upload, skipping
> All reports processed
>
> Why is it skipping the crash?

whoopsie-upload-all checks for a corresponding .uploaded file for the
crash report and if it exists that means the crash has already been
reported. So if every .crash has a .uploaded file then
whoopsie-upload-all does nothing.

--
Brian Murray

Revision history for this message
Brian Murray (brian-murray) wrote :

On Fri, Sep 18, 2020 at 09:18:25PM -0000, Nicolas Bock wrote:
> After deleting the files *.upload{,ed} I can run
>
> $ sudo /usr/share/apport/whoopsie-upload-all
> /var/crash/_usr_bin_emacs-gtk.1000.crash already has info collected
> Marking /var/crash/_usr_bin_emacs-gtk.1000.crash for whoopsie upload
> All reports processed
>
> And get the two files back. The content of the uploaded file is
>
> $ sudo cat _usr_bin_emacs-gtk.1000.uploaded
> 1c9b8196-f9f4-11ea-b6f1-fa163ee63de6

There is an apport cronjob, /etc/cron.daily/apport, which removes
crashes which are older than a week. If I recall correctly this predates
sending crash reports to the Ubuntu Error Tracker and
whoopsie-upload-all. So it'd make sense to have whoopsie-upload-all
remove the .crash and .upload files after the upload succeeded.

However, this is still something goofy going on with systemd.

--
Brian Murray

Revision history for this message
ToxicDragon (christian-ohrfandl-f) wrote :

Deleting the files in /var/crash (https://bugs.launchpad.net/ubuntu/+source/apport/+bug/1891657/comments/13) solved the issue for me aswell.

Revision history for this message
Gannet (ken20001) wrote :

>Deleting the files in /var/crash (https://bugs.launchpad.net/ubuntu/+source/apport/+bug/1891657/comments/13) solved the issue for me aswell.

Yeah, until rebooting the machine.

Revision history for this message
Balint Reczey (rbalint) wrote :

Will be fixed in next systemd upload, in 246.6-1ubuntu1, at least for the systemd part.
https://github.com/systemd/systemd/issues/16669

Changed in systemd (Ubuntu Groovy):
status: Confirmed → In Progress
importance: Undecided → High
Revision history for this message
Scott Stensland (scottstensland) wrote :

here is output of running

    journalctl -b -u whoopsie

    journalctl -b -u apport-autoreport

on Ubuntu 20.10

which has same 100% used by systemd

Revision history for this message
Masroor (mmasroorali) wrote :

This is growing my syslog file like crazy. Often running out of space in /var partition (yes, I have a different partition for this).

Deleting the files /var/crash/ stops this. Restarts again after a reboot.

Revision history for this message
Rocko (rockorequin) wrote :

I changed

[Path]
PathChangedGlob=/var/crash/*.crash

Revision history for this message
Rocko (rockorequin) wrote :

Trying again... After reading the linked systemd bug report, I changed /usr/lib/systemd/system/apport-autoreport.path from having PathExistsGlob to PathChangedGlob and ran sudo systemctl daemon-reload, and this seemed to fix the problem:

[Unit]
Description=Process error reports when automatic reporting is enabled (file watch)
ConditionPathExists=/var/lib/apport/autoreport

[Path]
PathChangedGlob=/var/crash/*.crash

[Install]
WantedBy=paths.target

Now journalctl -e shows:

Sep 23 18:00:35 xps15 whoopsie-upload-all[12306]: /var/crash/_opt_wine-devel_bin_wine-preloader.1000.cr>
Sep 23 18:00:35 xps15 whoopsie-upload-all[12306]: All reports processed
Sep 23 18:00:35 xps15 systemd[1]: apport-autoreport.service: Succeeded.
Sep 23 18:00:35 xps15 systemd[1]: Finished Process error reports when automatic reporting is enabled.

Revision history for this message
Rocko (rockorequin) wrote :

Also, if you want to reduce the massive size of the journal logs that this bug creates, do something like this:

sudo journalctl --rotate
sudo journalctl --vacuum-size=100M

That second command reduces it to 100 MB, but you can specify larger or smaller sizes.

Revision history for this message
Nicolas Bock (nicolasbock) wrote :

On systemd-246.4-1ubuntu1 the setting PathChangedGlob leads to an error. But I tried the following which seems to work correctly:

[Path]
# PathExistsGlob=/var/crash/*.crash
PathChanged=/var/crash

Thanks @rockrequin for the hint!

Revision history for this message
Nicolas Bock (nicolasbock) wrote :

@balint I was wondering why we don't use `PathChanged` in the systemd path file. This checks for any changes in the path and will pick up new crash files... But after reading the systemd docs again I guess the reason is that we want the service to activate if the crash directory already contains a crash file on startup. Am I understanding this correctly?

Revision history for this message
Brian Murray (brian-murray) wrote :

On Fri, Sep 25, 2020 at 03:42:00PM -0000, Nicolas Bock wrote:
> @balint I was wondering why we don't use `PathChanged` in the systemd
> path file. This checks for any changes in the path and will pick up new
> crash files... But after reading the systemd docs again I guess the
> reason is that we want the service to activate if the crash directory
> already contains a crash file on startup. Am I understanding this
> correctly?

The issue with PathChanged is that /var/crash is used for writing three
types of files. The .crash file, the .upload file (which signifies the
crash is ready to be uploaded), and the .uploaded file (which contains
the UUID of the crash in the error tracker and indicates the crash has
been uploaded). So if PathChanged is used then whoopsie-upload-all will
be called for .upload and .uploaded files. Additionally, there is no
PathChangedGlob so that we could watch for just .crash files.

To help with the issue I'm working on a change to whoopsie-upload-all to
remove the .crash file after a .uploaded file exists.

--
Brian Murray

Revision history for this message
Nicolas Bock (nicolasbock) wrote :

On Fri, Sep 25 2020, Brian Murray wrote:

> The issue with PathChanged is that /var/crash is used for writing three
> types of files. The .crash file, the .upload file (which signifies the
> crash is ready to be uploaded), and the .uploaded file (which contains
> the UUID of the crash in the error tracker and indicates the crash has
> been uploaded). So if PathChanged is used then whoopsie-upload-all will
> be called for .upload and .uploaded files. Additionally, there is no
> PathChangedGlob so that we could watch for just .crash files.

Thanks, that makes sense. Although re-reading the
systemd.path manpage over again I still don't understand how
this ever worked. It says:

"PathExists= may be used to watch the mere existence of a
file or directory. If the file specified exists, the
configured unit is activated."

I read this as saying that the current behavior on Groovy is
actually correct. Or am I missing something?

> To help with the issue I'm working on a change to whoopsie-upload-all to
> remove the .crash file after a .uploaded file exists.

Thanks! I think this certainly should help.

Revision history for this message
Brian Murray (brian-murray) wrote :

On Fri, Sep 25, 2020 at 07:13:52PM -0000, Nicolas Bock wrote:
> On Fri, Sep 25 2020, Brian Murray wrote:
>
> > The issue with PathChanged is that /var/crash is used for writing three
> > types of files. The .crash file, the .upload file (which signifies the
> > crash is ready to be uploaded), and the .uploaded file (which contains
> > the UUID of the crash in the error tracker and indicates the crash has
> > been uploaded). So if PathChanged is used then whoopsie-upload-all will
> > be called for .upload and .uploaded files. Additionally, there is no
> > PathChangedGlob so that we could watch for just .crash files.
>
> Thanks, that makes sense. Although re-reading the
> systemd.path manpage over again I still don't understand how
> this ever worked. It says:
>
> "PathExists= may be used to watch the mere existence of a
> file or directory. If the file specified exists, the
> configured unit is activated."
>
> I read this as saying that the current behavior on Groovy is
> actually correct. Or am I missing something?

A bug was fixed in systemd that resolved a discrepancy between what the
documentation said (which you've quoted) and the actual behavior of
systemd. This is documented upstream in the following issue:

https://github.com/systemd/systemd/issues/16669

--
Brian Murray

Revision history for this message
Fritz Hudnut (este-el-paz) wrote :

@Brian M:

As you suggest that my or my problem reported likely by Dan van Vugt with my install of Ubuntu-MATE 20.10 had this similar problem of long strings of "Failed to start " items in the TTY and very long boot times . . . wasn't exactly "fixed" by trashing the /var/crash directory as he suggested there . . . . Then he said, "if that doesn't fix it you should do a fresh install"

https://bugs.launchpad.net/ubuntu/+source/apport/+bug/1893952

I'll have to read through some of the suggestions here when I have more time to spend on it. As mentioned in my bug report, I also have a Lubuntu 20.10 and that doesn't have these issues with long boot times. In my U-MATE edition when the GUI log in window finally loads there is like a "clang-a-lang" error sound . . . unique to that one install out of maybe 6 linux installs I have on "bare metal" . . . .

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.20.11-0ubuntu48

---------------
apport (2.20.11-0ubuntu48) groovy; urgency=medium

  * data/whoopsie-upload-all: When processing reports if a .crash file already
    has a corresponding .uploaded file which is newer than the .crash file
    remove the .crash file. This reduces the number of times the
    apport-autoreport.service runs. (LP: #1891657)

 -- Brian Murray <email address hidden> Fri, 25 Sep 2020 14:49:27 -0700

Changed in apport (Ubuntu Groovy):
status: Confirmed → Fix Released
Revision history for this message
Rocko (rockorequin) wrote :

I'm still seeing this bug with apport 2.20.11-0ubuntu48 and systemd 246.5-1ubuntu1. Do we need systemd 246.6 before it is fixed?

Revision history for this message
Brian Murray (brian-murray) wrote :

On Mon, Sep 28, 2020 at 04:51:27AM -0000, Rocko wrote:
> I'm still seeing this bug with apport 2.20.11-0ubuntu48 and systemd
> 246.5-1ubuntu1. Do we need systemd 246.6 before it is fixed?

I ran 'sudo apt-get dist-upgrade' on a system running groovy and still
saw this issue as whoopsie was not able to contact the server to upload
the crash report and create the .uploaded file. You could see if this is
happening to you by running 'sudo systemctl status whoopsie'. I rebooted
and then whoopsie was able to contact the server and the .uploaded file
was created and then the .crash file was removed.

--
Brian Murray

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 246.6-1ubuntu1

---------------
systemd (246.6-1ubuntu1) groovy; urgency=medium

  [ Dan Streetman ]
  * Fix resolved.conf Cache= default to match default (LP: #1895418)
    Files:
    - debian/patches/lp1895418.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=dd4d93d94ebe2cf416f6b5a5eb59a16432cbc47b

  [ Balint Reczey ]
  * Bugfix-only upload
  * Merge from Debian unstable
    - core: propagate unit start limit hit state to triggering path unit (LP: #1891657)
  * Skip test_rsyslog in s390x containers.
    The test is failing almost all the time. (LP: #1895576)
    File: debian/tests/boot-and-services
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ff8d286de52daa8b0856213ae52e66b81474fb57
  * debian/tests/tests-in-lxd: Don't create the lxd test image twice
    File: debian/tests/tests-in-lxd
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=e858e7c41b84bc4c34e8af79d6e4b6114fe23952

systemd (246.6-1) unstable; urgency=medium

  * New upstream version 246.6
  * Rebase patches

 -- Balint Reczey <email address hidden> Thu, 24 Sep 2020 21:27:11 +0200

Changed in systemd (Ubuntu Groovy):
status: In Progress → Fix Released
Revision history for this message
Rocko (rockorequin) wrote :

systemd 246.6-1ubuntu1 and apport 2.20.11-0ubuntu48 fix this issue for me.

Revision history for this message
Nicolas Bock (nicolasbock) wrote :

Still failing for me. I see now:

Sep 29 06:32:27 rubberducky systemd[1]: Starting Process error reports when automatic reporting is enabled...
Sep 29 06:32:27 rubberducky systemd[1]: apport-autoreport.service: Succeeded.
Sep 29 06:32:27 rubberducky systemd[1]: Finished Process error reports when automatic reporting is enabled.
Sep 29 06:32:27 rubberducky systemd[1]: apport-autoreport.service: Start request repeated too quickly.
Sep 29 06:32:27 rubberducky systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
Sep 29 06:32:27 rubberducky systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
Sep 29 06:32:27 rubberducky systemd[1]: apport-autoreport.path: Failed with result 'unit-start-limit-hit'.

I have 2 crash files in /var/crash. Note that those two files were in /var/crash already on boot, I didn't add anything to this directory.

Revision history for this message
Brian Murray (brian-murray) wrote :

On Tue, Sep 29, 2020 at 12:38:38PM -0000, Nicolas Bock wrote:
> Still failing for me. I see now:
>
> Sep 29 06:32:27 rubberducky systemd[1]: Starting Process error reports when automatic reporting is enabled...
> Sep 29 06:32:27 rubberducky systemd[1]: apport-autoreport.service: Succeeded.
> Sep 29 06:32:27 rubberducky systemd[1]: Finished Process error reports when automatic reporting is enabled.
> Sep 29 06:32:27 rubberducky systemd[1]: apport-autoreport.service: Start request repeated too quickly.
> Sep 29 06:32:27 rubberducky systemd[1]: apport-autoreport.service: Failed with result 'start-limit-hit'.
> Sep 29 06:32:27 rubberducky systemd[1]: Failed to start Process error reports when automatic reporting is enabled.
> Sep 29 06:32:27 rubberducky systemd[1]: apport-autoreport.path: Failed with result 'unit-start-limit-hit'.
>
> I have 2 crash files in /var/crash. Note that those two files were in
> /var/crash already on boot, I didn't add anything to this directory.

Could you please run 'systemctl status whoopsie' and include the output
of the command here?

--
Brian Murray

Revision history for this message
Nicolas Bock (nicolasbock) wrote :

On Tue, Sep 29 2020, Brian Murray wrote:

> On Tue, Sep 29, 2020 at 12:38:38PM -0000, Nicolas Bock wrote:
> Could you please run 'systemctl status whoopsie' and
> include the output of the command here?

$ sudo systemctl status whoopsie
● whoopsie.service - crash report submission daemon
     Loaded: loaded (/lib/systemd/system/whoopsie.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2020-09-29 06:27:06 MDT; 10h ago
   Main PID: 2686 (whoopsie)
      Tasks: 3 (limit: 47889)
     Memory: 5.0M
     CGroup: /system.slice/whoopsie.service
             └─2686 /usr/bin/whoopsie -f

Sep 29 08:18:55 rubberducky whoopsie[2686]: [08:18:55] offline
Sep 29 08:18:56 rubberducky whoopsie[2686]: [08:18:56] The default IPv4 route is: /org/freedesktop/NetworkManager/ActiveConnection/4
Sep 29 08:18:56 rubberducky whoopsie[2686]: [08:18:56] Not a paid data plan: /org/freedesktop/NetworkManager/ActiveConnection/4
Sep 29 08:18:56 rubberducky whoopsie[2686]: [08:18:56] Found usable connection: /org/freedesktop/NetworkManager/ActiveConnection/4
Sep 29 08:18:56 rubberducky whoopsie[2686]: [08:18:56] online
Sep 29 08:59:15 rubberducky whoopsie[2686]: [08:59:15] offline
Sep 29 08:59:16 rubberducky whoopsie[2686]: [08:59:16] The default IPv4 route is: /org/freedesktop/NetworkManager/ActiveConnection/4
Sep 29 08:59:16 rubberducky whoopsie[2686]: [08:59:16] Not a paid data plan: /org/freedesktop/NetworkManager/ActiveConnection/4
Sep 29 08:59:16 rubberducky whoopsie[2686]: [08:59:16] Found usable connection: /org/freedesktop/NetworkManager/ActiveConnection/4
Sep 29 08:59:16 rubberducky whoopsie[2686]: [08:59:16] online

Revision history for this message
Fritz Hudnut (este-el-paz) wrote :

On Tue, Sep 29, 2020 at 4:10 PM Nicolas Bock <email address hidden>
wrote:

> On Tue, Sep 29 2020, Brian Murray wrote:
>
> > On Tue, Sep 29, 2020 at 12:38:38PM -0000, Nicolas Bock wrote:
> > Could you please run 'systemctl status whoopsie' and
> > include the output of the command here?
>
> $ sudo systemctl status whoopsie
>
>
I had a few minutes to boot my U-MATE 20.10 distro that is so slow to boot
. . . . I checked "ls var/crash" and it showed three files there, when
one of the last times I booted it up I trashed the whole directory, as was
suggested to try to improve boot times . . . at first it seemed to but
today wasn't very good as far as comparing ubuntu to my other options.

I then ran the requested "whoopsie" command and got at least ten blocks of
lines claiming to be "19 lines" spaced out with blackness between them . .
. I copied one of the blocks, looks fairly similar to the previous posting??

[CODE] whoopsie.service - crash report submission daemon
     Loaded: loaded (/lib/systemd/system/whoopsie.service; enabled; vendor
preset: enabled)
     Active: active (running) since Wed 2020-09-30 11:26:52 PDT; 12min ago
   Main PID: 1902 (whoopsie)
      Tasks: 3 (limit: 19101)
     Memory: 7.9M
     CGroup: /system.slice/whoopsie.service
             └─1902 /usr/bin/whoopsie -f

Sep 30 11:26:52 non1-MacPro systemd[1]: Started crash report submission
daemon.
Sep 30 11:26:52 non1-MacPro whoopsie[1902]: [11:26:52] Using lock path:
/var/lock/whoopsie/lock
Sep 30 11:26:53 non1-MacPro whoopsie[1902]: [11:26:53] The default IPv4
route is: /org/freedesktop/>
Sep 30 11:26:53 non1-MacPro whoopsie[1902]: [11:26:53] Not a paid data
plan: /org/freedesktop/Netwo>
Sep 30 11:26:53 non1-MacPro whoopsie[1902]: [11:26:53] Found usable
connection: /org/freedesktop/Ne>
Sep 30 11:32:32 non1-MacPro whoopsie[1902]: [11:32:32] Parsing
/var/crash/_usr_libexec_ayatana-indi>
Sep 30 11:32:32 non1-MacPro whoopsie[1902]: [11:32:32] Uploading
/var/crash/_usr_libexec_ayatana-in>
Sep 30 11:32:33 non1-MacPro whoopsie[1902]: [11:32:33] Sent; server replied
with: No error
Sep 30 11:32:33 non1-MacPro whoopsie[1902]: [11:32:33] Response code: 200
Sep 30 11:32:33 non1-MacPro whoopsie[1902]: [11:32:33] Reported OOPS ID
4e2fc590-034b-11eb-bd50-fa1>
~ [CODE/]

tags: added: fr-691
Changed in apport (Ubuntu):
status: Fix Released → Incomplete
Changed in systemd (Ubuntu):
status: Fix Released → Incomplete
Changed in apport (Ubuntu Groovy):
status: Fix Released → Incomplete
Changed in systemd (Ubuntu Groovy):
status: Fix Released → Incomplete
tags: added: connection-refused
Changed in apport (Ubuntu):
status: Incomplete → Fix Released
Changed in apport (Ubuntu Groovy):
status: Incomplete → Fix Released
Changed in systemd (Ubuntu Groovy):
status: Incomplete → Fix Released
Changed in systemd (Ubuntu):
status: Incomplete → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.