I have a 24.04.3 LTS server running KVM/LibVirt hosting VMs.
I set this up several months ago and it has been running sweetly until this past week.
In the period of 3 days I have had the VMs crash twice. Troubleshooting I find they have lost their disks. All on a mounted disk on the host over an iSCSI connection to a NAS.
And looking into the host /var/log/syslog (grepping iscsi) I see the following sequence (note, I send audit to syslog so they get sent to a central syslog server)
2026-02-04T06:16:37.229079+00:00 host iscsid: semop down failed 22
2026-02-04T06:16:37.229279+00:00 host systemd[1]: Stopping iscsid.service - iSCSI initiator daemon (iscsid)...
2026-02-04T06:16:37.232226+00:00 host systemd[1]: iscsid.service: Deactivated successfully.
2026-02-04T06:16:37.232601+00:00 host systemd[1]: Stopped iscsid.service - iSCSI initiator daemon (iscsid).
2026-02-04T06:16:37.232720+00:00 host systemd[1]: iscsid.service: Consumed 10.335s CPU time, 6.1M memory peak, 0B memory swap peak.
2026-02-04T06:16:37.232819+00:00 host kernel: audit: type=1131 audit(1770185797.230:6076): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=iscsid comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? add
r=? terminal=? res=success'
2026-02-04T06:16:37.236237+00:00 host systemd[1]: Starting iscsid.service - iSCSI initiator daemon (iscsid)...
2026-02-04T06:16:37.263530+00:00 host iscsid: iSCSI logger with pid=116600 started!
2026-02-04T06:16:37.275400+00:00 host systemd[1]: Started iscsid.service - iSCSI initiator daemon (iscsid).
2026-02-04T06:16:37.275849+00:00 host audisp-syslog: node=host type=SERVICE_START msg=audit(1770185797.273:6085): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=iscsid comm="systemd" exe="/usr/lib/syst
emd/systemd" hostname=? addr=? terminal=? res=success'
2026-02-04T06:16:37.568450+00:00 host systemd-networkd[116731]: iscsi01: Link UP
2026-02-04T06:16:37.568582+00:00 host systemd-networkd[116731]: iscsi01: Gained carrier
2026-02-04T06:16:37.576650+00:00 host systemd-networkd[116731]: iscsi01: Gained IPv6LL
2026-02-04T06:16:37.598887+00:00 host systemd-networkd[116731]: iscsi01: Configuring with /run/systemd/network/10-netplan-eno4.network.
2026-02-04T06:16:38.263894+00:00 host iscsid: iSCSI daemon with pid=116603 started!
2026-02-04T06:16:40.264158+00:00 host iscsid: Login failed to authenticate with target iqn.2xxtargetxxsi.target-0.90774d
2026-02-04T06:16:40.264362+00:00 host iscsid: session 1 login rejected: Initiator failed authentication with target
2026-02-04T06:16:42.264397+00:00 host iscsid: Login failed to authenticate with target iqn.2xxtargetxxsi.target-0.90774d
2026-02-04T06:16:42.264573+00:00 host iscsid: session 1 login rejected: Initiator failed authentication with target
2026-02-04T06:16:44.264797+00:00 host iscsid: Login failed to authenticate with target iqn.2xxtargetxxsi.target-0.90774d
2026-02-04T06:16:44.265117+00:00 host iscsid: session 1 login rejected: Initiator failed authentication with target
2026-02-04T06:16:46.265080+00:00 host iscsid: Login failed to authenticate with target iqn.2xxtargetxxsi.target-0.90774d
2026-02-04T06:16:46.265270+00:00 host iscsid: session 1 login rejected: Initiator failed authentication with target
2026-02-04T06:16:48.265378+00:00 host iscsid: Login failed to authenticate with target iqn.2xxtargetxxsi.target-0.90774d
.
.
.
.
To try and get things going again I have a done a systemctl restart iscsid.service and this does not report any issues.
So then I try a iscsiadm -m node --login but it tells me:
iscsiadm: eno4: 1 session requested, but 1 already present.
So I reboot the host (iscsi initiator) and it boots up and reconnects AOK and I can restart the VMs and they replay their journal entries (ext4) and away we go.
Hardware is a HPE ProLiant DL380 Gen9: 48core/384 GB Mem
The eno4 is one of the onboard 1Gbit Ethernets directly connected to the NAS storage, a QNAP ts-473a
# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.3 LTS"
# uname -a
Linux host 6.8.0-90-generic #91-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 18 14:14:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
I have just rebooted to get latest kernel installed... but the version above is what the latest crash was running on.
Any thoughts on what is causing the semop down failures?
Cheers
Jim