Ubuntu

Ubuntu dev Host rebooting frequently

Asked by Dilip B R

Ubuntu dev Host rebooting frequently:
No cronjobs
No hint in /var/log/syslog
No hint in /var/log/auth.log

Mar 21 02:45:01 test-dev CRON[24830]: pam_unix(cron:session): session closed for user root
Mar 21 02:55:00 test-dev systemd-logind[1325]: Creating /run/nologin, blocking further logins...
Mar 21 02:55:01 test-dev CRON[29640]: pam_unix(cron:session): session opened for user root by (uid=0)
Mar 21 02:55:01 test-dev CRON[29640]: pam_unix(cron:session): session closed for user root
Mar 21 03:00:00 test-dev systemd-logind[1325]: System is rebooting.
Mar 21 03:00:25 test-dev systemd-logind[1229]: New seat seat0.
Mar 21 03:00:25 test-dev systemd-logind[1229]: Watching system buttons on /dev/input/event0 (Power Button)
Mar 21 03:00:25 test-dev systemd-logind[1229]: Watching system buttons on /dev/input/event1 (AT Translated Set 2 keyboard)
Mar 21 03:00:25 test-dev sshd[1623]: Server listening on 0.0.0.0 port 22.
Mar 21 03:00:25 test-dev sshd[1623]: Server listening on :: port 22.

Mar 21 02:52:58 test-dev nrpe[28645]: Error: (use_ssl == true): Request packet version was invalid!
Mar 21 02:52:58 test-dev nrpe[28645]: Could not read request from client 10.32.238.50, bailing out...
Mar 21 02:52:58 test-dev nrpe[28645]: INFO: SSL Socket Shutdown.
Mar 21 02:55:01 test-dev CRON[29641]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Mar 21 02:57:43 test-dev nrpe[30862]: Error: (use_ssl == true): Request packet version was invalid!
Mar 21 02:57:43 test-dev nrpe[30862]: Could not read request from client 10.32.238.50, bailing out...
Mar 21 02:57:43 test-dev nrpe[30862]: INFO: SSL Socket Shutdown.
Mar 21 02:57:57 test-dev nrpe[31022]: Error: (use_ssl == true): Request packet version was invalid!
Mar 21 02:57:57 test-dev nrpe[31022]: Could not read request from client 10.32.238.50, bailing out...
Mar 21 02:57:57 test-dev nrpe[31022]: INFO: SSL Socket Shutdown.
Mar 21 03:00:00 test-dev systemd[1]: Stopped Unattended Upgrades Shutdown.
Mar 21 03:00:00 test-dev systemd[1]: Stopping Authorization Manager...
Mar 21 03:00:00 test-dev dockerd[15764]: time="2021-03-21T03:00:00.065940884+01:00" level=info msg="Processing signal 'terminated'"
Mar 21 03:00:00 test-dev systemd[1]: Stopped target Timers.
Mar 21 03:00:00 test-dev systemd[1]: Stopped Discard unused blocks once a week.
Mar 21 03:00:00 test-dev systemd[1]: Stopping LVM2 PV scan on device 8:3...
Mar 21 03:00:00 test-dev systemd[1]: Stopped target Graphical Interface.
Mar 21 03:00:00 test-dev systemd[1]: Stopping Accounts Service...
Mar 21 03:00:00 test-dev systemd[1]: Stopping Docker Application Container Engine...
Mar 21 03:00:00 test-dev systemd[1]: Stopped target RPC Port Mapper.
Mar 21 03:00:00 test-dev systemd[1]: Stopped Daily apt upgrade and clean activities.
Mar 21 03:00:00 test-dev systemd[1]: Stopped target Cloud-init target.
Mar 21 03:00:00 test-dev systemd[1]: Stopping Availability of block devices...
Mar 21 03:00:00 test-dev systemd[1]: Closed Load/Save RF Kill Switch Status /dev/rfkill Watch.
Mar 21 03:00:00 test-dev systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
Mar 21 03:00:25 test-dev systemd-modules-load[542]: Inserted module 'iscsi_tcp'

Question information

Language:
English Edit question
Status:
Expired
For:
Ubuntu Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
actionparsnip (andrew-woodhead666) said : 		#1

Is it a virtual machine? If so, check for scheduled jobs in vCentre and on the ESX host in crontab there too

Revision history for this message
Dilip B R (dilip-br) said : 		#2

Found no cronjobs or any scheduled reboot at OS level and VM level
Multiple docker containers are running on this host, suspect rebooting/shutting down the container also rebooting the parent host.
Any suggestions or is this known issue?

Revision history for this message
Manfred Hampl (m-hampl) said : 		#3

Is this the same system as the one in https://answers.launchpad.net/ubuntu/+question/696111 ?
If yes, why did you create a second question instead of continuing the old one?

Does it happen at 03:00:00 every day or once a week or on random days?

Revision history for this message
Dilip B R (dilip-br) said : 		#4

Yes, didnt get any response for old request hence created new request.

Does it happen at 03:00:00 every day or once a week or on random days?
Yes its happening random days but only at 3:00am

# last reboot
reboot system boot 4.15.0-140-gener Sat Mar 27 03:00 still running
reboot system boot 4.15.0-139-gener Sun Mar 21 03:00 - 12:20 (4+09:20)
reboot system boot 4.15.0-137-gener Wed Mar 17 03:00 - 03:00 (3+23:59)

Revision history for this message
Manfred Hampl (m-hampl) said : 		#5

The "last reboot" command shows a new version of the kernel with each reboot. Maybe a hint, maybe a coincidence.

Kernel version 4.15.0-137.141 was published on 2021-03-15
Kernel version 4.15.0-139.143 was published on 2021-03-19
Kernel version 4.15.0-140.144 was published on 2021-03-24 or 2021-03-25

Is there a setting in place that automatically installs new versions of packages and/or reboots if a reboot is required to activate them (e.g. for kernel packages)?

Is there any cron job that runs at 03:00:00?
see https://phoenixnap.com/kb/how-to-list-display-view-all-cron-jobs-linux

Revision history for this message
Dilip B R (dilip-br) said : 		#6

Yes,
I can notify on every reboot kernel package showing updated version

Found kernel is set to install manually:
linux-image-generic is already the newest version (4.15.0.140.127).
linux-image-generic set to manually installed.

Revision history for this message
Manfred Hampl (m-hampl) said : 		#7

"linux-image-generic set to manually installed."

This is a response to the command "apt install linux-image-generic" and tells that now you have set that this package is marked as "manually installed", and that it is no more installed because of dependencies (and it is a dependency of linux-image which in turn usually is defined as manually installed, see "apt-mark showmanual")
This does not have any relationship to the reboot problem.

Revision history for this message
Dilip B R (dilip-br) said : 		#8

No cronjobs/scheduled reboot on host
No VM level scheduled reboot
But still host getting reboot on random days only at 03:00 AM
Strange still didnt get RC, could you help on this to get RCA.

Revision history for this message
Manfred Hampl (m-hampl) said : 		#9

Is there ANY cron job or at job that runs at 03:00?

Revision history for this message
Dilip B R (dilip-br) said : 		#10

Found no cron jobs on this host.

# crontab -l
no crontab for root

Revision history for this message
Manfred Hampl (m-hampl) said : 		#11

Not only for root, but also for other users, see https://phoenixnap.com/kb/how-to-list-display-view-all-cron-jobs-linux

Revision history for this message
actionparsnip (andrew-woodhead666) said : 		#12

ALL users have a crontab...not just root. You can see them all with:

echo "***************"; for i in `cat /etc/passwd | cut -d":" -f 1`; do echo $i; sudo crontab -l -u $i; echo "***************"; done

Do ANY users have a crontab to reboot?

Revision history for this message
Dilip B R (dilip-br) said : 		#13

none of the user having cron jobs on this host

***************
root
no crontab for root
***************
daemon
no crontab for daemon
***************
bin
no crontab for bin
***************
sys
no crontab for sys
***************
sync
no crontab for sync
***************
games
no crontab for games
***************
man
no crontab for man
***************
lp
no crontab for lp
***************
mail
no crontab for mail
***************
news
no crontab for news
***************
uucp
no crontab for uucp
***************
proxy
no crontab for proxy
***************
www-data
no crontab for www-data
***************
backup
no crontab for backup
***************
list
no crontab for list
***************
irc
no crontab for irc
***************
gnats
no crontab for gnats
***************
nobody
no crontab for nobody
***************
systemd-network
no crontab for systemd-network
***************
systemd-resolve
no crontab for systemd-resolve
***************
syslog
no crontab for syslog
***************
messagebus
no crontab for messagebus
***************
_apt
no crontab for _apt
***************
lxd
no crontab for lxd
***************
uuidd
no crontab for uuidd
***************
dnsmasq
no crontab for dnsmasq
***************
landscape
no crontab for landscape
***************
pollinate
no crontab for pollinate
***************
sshd
no crontab for sshd
***************
uroot
no crontab for uroot
***************
easy_50
no crontab for easy_50
***************
nagios
no crontab for nagios
***************
statd
no crontab for statd
***************
_grueter_50
no crontab for _grueter_50
***************

Revision history for this message
Manfred Hampl (m-hampl) said : 		#14

Is there really nothing in cron.daily etc.?

Revision history for this message
Launchpad Janitor (janitor) said : 		#15

This question was expired because it remained in the 'Needs information' state without activity for the last 15 days.