crontab tasks are accumilating

What is the output of:

ps -ef | grep cron; lsb_release -a; uname -a

Thanks

What is the output of the commands

systemctl status cron
sudo ls -la /var/spool/cron/crontabs

(You are free to obfuscate details that you want to keep secret, as long as the gist stays readable.)

Remark:
I am a bit astonished to see a quite old version of the kernel (4.4.0-31). Current version is 4.4.0-200

Thanks for the update, will look into kernel version.

Please find the details below,

root@bd-server1:~# ps -ef | grep cron; lsb_release -a; uname -a
root 4396 1 0 Jan06 ? 00:00:04 /usr/sbin/cron -f
root 19845 19829 0 12:20 pts/27 00:00:00 grep --color=auto cron
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial
Linux bd-server1.com 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

root@bd-server1:~# systemctl status cron
● cron.service - Regular background program processing daemon
   Loaded: loaded (/lib/systemd/system/cron.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-01-06 00:34:08 IST; 1 weeks 6 days ago
     Docs: man:cron(8)
 Main PID: 4396 (cron)
    Tasks: 167
   Memory: 44.4M
      CPU: 1h 53min 50.866s
   CGroup: /system.slice/cron.service
           ├─ 356 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
           ├─ 431 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
           ├─ 580 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
           ├─ 582 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
           ├─ 771 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
           ├─ 967 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
           ├─ 1200 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
           ├─ 1653 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
.
.
.
.
          ├─31552 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
           ├─31889 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
           ├─32577 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
           └─32641 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh

Jan 19 12:17:01 bd-server1.com CRON[19643]: (barman) CMD ([ -x /usr/bin/barman ] && /usr/bin/barman -q cron)
Jan 19 12:17:01 bd-server1.com CRON[19640]: pam_unix(cron:session): session closed for user barman
Jan 19 12:18:01 bd-server1.com CRON[19665]: pam_unix(cron:session): session opened for user barman by (uid=0)
Jan 19 12:18:01 bd-server1.com CRON[19666]: (barman) CMD ([ -x /usr/bin/barman ] && /usr/bin/barman -q cron)
Jan 19 12:19:01 bd-server1.com CRON[19685]: pam_unix(cron:session): session opened for user barman by (uid=0)
Jan 19 12:19:01 bd-server1.com CRON[19686]: (barman) CMD ([ -x /usr/bin/barman ] && /usr/bin/barman -q cron)
Jan 19 12:19:02 bd-server1.com CRON[19685]: pam_unix(cron:session): session closed for user barman
Jan 19 12:20:01 bd-server1.com CRON[19758]: pam_unix(cron:session): session opened for user barman by (uid=0)
Jan 19 12:20:01 bd-server1.com CRON[19759]: (barman) CMD ([ -x /usr/bin/barman ] && /usr/bin/barman -q cron)
Jan 19 12:20:01 bd-server1.com CRON[19758]: pam_unix(cron:session): session closed for user barman

root@bd-server1:~# sudo ls -la /var/spool/cron/crontabs
total 12
drwx-wx--T 2 root crontab 4096 Jan 6 00:36 .
drwxr-xr-x 5 root root 4096 Dec 24 2018 ..
-rw------- 1 oracle crontab 2727 Jan 6 00:36 oracle

Are those backup jobs still running?
e.g. do you see them with
ps ax | grep "/bin/sh /home"

Eventually there is a problem with the backup jobs that keeps them running forever instead of ending.

for your question, yes.

Initially we stopped all schedules for couple of days. then after again enabled.

ps ax | grep "/bin/sh /home"
  356 ? D 0:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
  431 ? D 0:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
  580 ? D 0:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
  582 ? D 0:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
  771 ? D 0:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
  967 ? D 0:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
 1200 ? D 0:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh

yes, your are right. But same script is running fine in other machines.

Is there any way to clear all crontab tasks and restarting the cron service.

Stopping cron service and staring not clearing the Queue of TASKS.

Could kill the PIDs. I suggest you add lines to the script to echo text to a file on the system so that you can read it and see where it stops and hangs

Killing PIDs not worked.

Backups are running now without any issues. But couldn't clear the past cron TASK's queue.

Any other ways, I can get here.

What happens if you issue the command

kill 356 431 580

or (if this gives a "not authorized")

sudo kill 356 431 580

or eventually

sudo kill -SIGHUP 356 431 580

etc.

We got no issues while killing those PID's, Please find the below details. kill with SIGHUP also didn't work.(details added below)

oracle@bd-server1:~$ kill 356 431 580
oracle@bd-server1:~$ ps -ef|grep 356
oracle 356 1 0 2020 ? 00:00:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
oracle 8289 8241 0 10:10 pts/26 00:00:00 grep --color=auto 356
oracle 23562 1 0 2020 ? 00:00:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
oracle@bd-server1:~$ ps -ef|grep 431
oracle 431 1 0 2020 ? 00:00:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
oracle 4314 1 0 2020 ? 00:00:00 /lib/systemd/systemd --user
oracle 4315 4314 0 2020 ? 00:00:00 (sd-pam)
oracle 8292 8241 0 10:10 pts/26 00:00:00 grep --color=auto 431
oracle@bd-server1:~$ ps -ef|grep 580
oracle 580 1 0 2020 ? 00:00:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
oracle 4580 1 0 2020 ? 00:02:25 ora_p005_BDAPX20
oracle 8295 8241 0 10:10 pts/26 00:00:00 grep --color=auto 580

----------------------------------------------------------------------------------------------------------------------------------

oracle@bd-server1:~$ kill -SIGHUP 356 431 580
oracle@bd-server1:~$
oracle@bd-server1:~$ ps -ef|grep 356
oracle 356 1 0 2020 ? 00:00:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
oracle 8315 8241 0 10:11 pts/26 00:00:00 grep --color=auto 356
oracle 23562 1 0 2020 ? 00:00:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
oracle@bd-server1:~$ ps -ef|grep 431
oracle 431 1 0 2020 ? 00:00:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
oracle 4314 1 0 2020 ? 00:00:00 /lib/systemd/systemd --user
oracle 4315 4314 0 2020 ? 00:00:00 (sd-pam)
oracle 8325 8241 0 10:11 pts/26 00:00:00 grep --color=auto 431
oracle@bd-server1:~$ ps -ef|grep 580
oracle 580 1 0 2020 ? 00:00:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
oracle 4580 1 0 2020 ? 00:02:25 ora_p005_BDAPX20
oracle 8327 8241 0 10:11 pts/26 00:00:00 grep --color=auto 580
oracle@bd-server1:~$ kill 356 431 580
oracle@bd-server1:~$ ps -ef|grep 356
oracle 356 1 0 2020 ? 00:00:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
oracle 8289 8241 0 10:10 pts/26 00:00:00 grep --color=auto 356
oracle 23562 1 0 2020 ? 00:00:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
oracle@bd-server1:~$ ps -ef|grep 431
oracle 431 1 0 2020 ? 00:00:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
oracle 4314 1 0 2020 ? 00:00:00 /lib/systemd/systemd --user
oracle 4315 4314 0 2020 ? 00:00:00 (sd-pam)
oracle 8292 8241 0 10:10 pts/26 00:00:00 grep --color=auto 431
oracle@bd-server1:~$ ps -ef|grep 580
oracle 580 1 0 2020 ? 00:00:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh
oracle 4580 1 0 2020 ? 00:02:25 ora_p005_BDAPX20
oracle 8295 8241 0 10:10 pts/26 00:00:00 grep --color=auto 580

----------------------------------------------------------------------------------------------------------------------------------

If the "peaceful" signals SIGHUP and SIGTERM do not work, then try killing the processes with SIGKILL

kill -SIGKILL 356 431 580

Thanks for the update.

even SIGKILL not worked for me. Plz find the details below,

oracle@bd-server1:~$ kill -SIGKILL 356 431 580
oracle@bd-server1:~$
oracle@bd-server1:~$
oracle@bd-server1:~$ ps -ef|grep 356
oracle 356 1 0 2020 ? 00:00:00 /bin/sh /home/oracle/bd-server1/scripts/backup_db_Archives.sh

What is the output of

which kill

Pleaser find the out below,

root@bd-server1:~# which kill
/bin/kill
root@bd-server1:~#

If the command
sudo /bin/kill -SIGKILL 356
does not work, then I am running out of ideas.
A reboot will most probably help, but I do not know whether this is feasible.

Thankyou "Manfred Hampl".

yes, Will check other alternatives and finally will go with reboot if required. will keep updated this if issue is fixed wthout reboot.