iscsid semop down failed 22 - ubuntu 24.04.3 qnap NAS

Asked by Jim Pye

I have a 24.04.3 LTS server running KVM/LibVirt hosting VMs.

I set this up several months ago and it has been running sweetly until this past week.

In the period of 3 days I have had the VMs crash twice. Troubleshooting I find they have lost their disks. All on a mounted disk on the host over an iSCSI connection to a NAS.

And looking into the host /var/log/syslog (grepping iscsi) I see the following sequence (note, I send audit to syslog so they get sent to a central syslog server)

2026-02-04T06:16:37.229079+00:00 host iscsid: semop down failed 22
2026-02-04T06:16:37.229279+00:00 host systemd[1]: Stopping iscsid.service - iSCSI initiator daemon (iscsid)...
2026-02-04T06:16:37.232226+00:00 host systemd[1]: iscsid.service: Deactivated successfully.
2026-02-04T06:16:37.232601+00:00 host systemd[1]: Stopped iscsid.service - iSCSI initiator daemon (iscsid).
2026-02-04T06:16:37.232720+00:00 host systemd[1]: iscsid.service: Consumed 10.335s CPU time, 6.1M memory peak, 0B memory swap peak.
2026-02-04T06:16:37.232819+00:00 host kernel: audit: type=1131 audit(1770185797.230:6076): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=iscsid comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? add
r=? terminal=? res=success'
2026-02-04T06:16:37.236237+00:00 host systemd[1]: Starting iscsid.service - iSCSI initiator daemon (iscsid)...
2026-02-04T06:16:37.263530+00:00 host iscsid: iSCSI logger with pid=116600 started!
2026-02-04T06:16:37.275400+00:00 host systemd[1]: Started iscsid.service - iSCSI initiator daemon (iscsid).
2026-02-04T06:16:37.275849+00:00 host audisp-syslog: node=host type=SERVICE_START msg=audit(1770185797.273:6085): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=iscsid comm="systemd" exe="/usr/lib/syst
emd/systemd" hostname=? addr=? terminal=? res=success'
2026-02-04T06:16:37.568450+00:00 host systemd-networkd[116731]: iscsi01: Link UP
2026-02-04T06:16:37.568582+00:00 host systemd-networkd[116731]: iscsi01: Gained carrier
2026-02-04T06:16:37.576650+00:00 host systemd-networkd[116731]: iscsi01: Gained IPv6LL
2026-02-04T06:16:37.598887+00:00 host systemd-networkd[116731]: iscsi01: Configuring with /run/systemd/network/10-netplan-eno4.network.
2026-02-04T06:16:38.263894+00:00 host iscsid: iSCSI daemon with pid=116603 started!
2026-02-04T06:16:40.264158+00:00 host iscsid: Login failed to authenticate with target iqn.2xxtargetxxsi.target-0.90774d
2026-02-04T06:16:40.264362+00:00 host iscsid: session 1 login rejected: Initiator failed authentication with target
2026-02-04T06:16:42.264397+00:00 host iscsid: Login failed to authenticate with target iqn.2xxtargetxxsi.target-0.90774d
2026-02-04T06:16:42.264573+00:00 host iscsid: session 1 login rejected: Initiator failed authentication with target
2026-02-04T06:16:44.264797+00:00 host iscsid: Login failed to authenticate with target iqn.2xxtargetxxsi.target-0.90774d
2026-02-04T06:16:44.265117+00:00 host iscsid: session 1 login rejected: Initiator failed authentication with target
2026-02-04T06:16:46.265080+00:00 host iscsid: Login failed to authenticate with target iqn.2xxtargetxxsi.target-0.90774d
2026-02-04T06:16:46.265270+00:00 host iscsid: session 1 login rejected: Initiator failed authentication with target
2026-02-04T06:16:48.265378+00:00 host iscsid: Login failed to authenticate with target iqn.2xxtargetxxsi.target-0.90774d
.
.
.
.

To try and get things going again I have a done a systemctl restart iscsid.service and this does not report any issues.

So then I try a iscsiadm -m node --login but it tells me:
iscsiadm: eno4: 1 session requested, but 1 already present.

So I reboot the host (iscsi initiator) and it boots up and reconnects AOK and I can restart the VMs and they replay their journal entries (ext4) and away we go.

Hardware is a HPE ProLiant DL380 Gen9: 48core/384 GB Mem

The eno4 is one of the onboard 1Gbit Ethernets directly connected to the NAS storage, a QNAP ts-473a

# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.3 LTS"

# uname -a
Linux host 6.8.0-90-generic #91-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 18 14:14:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

I have just rebooted to get latest kernel installed... but the version above is what the latest crash was running on.

Any thoughts on what is causing the semop down failures?

Cheers
Jim

Question information

Language:
English Edit question
Status:
Answered
For:
Ubuntu Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Jim Pye (jimpye) said :
#1

Oh, and:

# iscsiadm --version
iscsiadm version 2.1.9

Revision history for this message
actionparsnip (andrew-woodhead666) said :
#2

Have you tried rebooting the NAS?
Do you have the latest updates in the NAS?

Revision history for this message
Alex Phill (alexphill3928) said (last edit ):
#3
Revision history for this message
Alex Phill (alexphill3928) said :
#4

Wow, this looks frustrating. Flipgrid https://myflipgridapp.com/ makes sharing quick updates way easier, wish there was a way to post these logs there and get real-time tips from folks who’ve handled iSCSI issues.

Revision history for this message
Jim Pye (jimpye) said :
#5

@actionparsnip I did not reboot NAS but I did reboot the Linux to apply the latest Kernel and so far it has not faulted for 5 days now. Touch wood.

I did a dive into the source code of the open-iscsi and annoyingly the only mention of semop is in the log.c which is only checking for a semaphore issue and printing to the log the initial error 22 message I am seeing. So some further digging if it reappears.

And thanks for the spam @Alex Phil :(

Cheers
Jim

Can you help with this problem?

Provide an answer of your own, or ask Jim Pye for more information if necessary.

To post a message you must log in.