Openvswitch-ipsec daemon fails to start

https://packages.ubuntu.com/search?keywords=openvswitch&searchon=names&suite=focal&section=all

Its in the repos. Why are you downloading anything manually?

because

a.
it's not there. the Package openvswitch-ipsec is not part of this list.
infact, it was removed on 2.8.1 and reintroduce on 2.11.0 but still not found easily.
look here: https://answers.launchpad.net/ubuntu/+source/openvswitch/+question/695108

b.
I would like to use the latest version.

also:
After building it using openvswitch-2.14.1.tar.gz it still don't load.
I think it's only a matter of missing component or explation here:
https://docs.openvswitch.org/en/latest/tutorials/ipsec

Would you be so kind and point out for complete working installation guide for testing openvswitch+ipsec?

Whats in the new version that you absolutely need?

latest 2.14.1
if it includes ipsec support

If you have problems building software from outside Ubuntu, then you better ask at the source from where you take the source.

fair enough, that was done in the meantime, while searching for the installation package.

However, can you point out where is the Ubuntu latest version of openvswitch-ipsec?

There currently is no "Ubuntu latest version of openvswitch-ipsec", because that package has been removed at a certain point in time and has not yet been re-added to the current Ubuntu releases, see Bug #1912610

What you could try for further diagnostics:
Dig into the script files to check what they do for "systemctl restart openvswitch-ipsec" and check which single step fails.
E.g. something like
sudo sh -x /etc/init.d/openvswitch-ipsec start

already try that. got the same results:

+ echo -n Starting openvswitch-ipsec (via systemctl): openvswitch-ipsec.service
Starting openvswitch-ipsec (via systemctl): openvswitch-ipsec.service+ log_daemon_msg_post Starting openvswitch-ipsec (via systemctl) openvswitch-ipsec.service
+ :
+ /bin/systemctl --no-pager start openvswitch-ipsec.service
Warning: The unit file, source configuration file or drop-ins of openvswitch-ipsec.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job for openvswitch-ipsec.service failed because the control process exited with error code.
See "systemctl status openvswitch-ipsec.service" and "journalctl -xe" for details.

the bug that you mentioned was open by me. I know it. but docs also say that openvswitch-ipsec reintroduce on 2.11.0
so 2.14.1 (current) should include it. however I don't see its running when running "systemctl status openvswitch-ipsec"

where can I find support on that component?
is it removed?

Is there a logfile in /var/log/openvswitch that can help?

ovs-ctl.log ovs-pki.log ovs-vswitchd.log ovsdb-server.log say nothing special.
openvswitch-ipsec does not even mention.
it fails during 'start' although it looks like it fail install.

dpkg.log say:
2021-02-02 10:22:02 configure openvswitch-ipsec:arm64 2.14.1-1 2.14.1-1
2021-02-02 10:22:02 status half-configured openvswitch-ipsec:arm64 2.14.1-1

The failure of the installation is caused by a failed start of the daemon. On of the last steps of installing is starting the daemon, and if the demon fails to start, this is reported as incomplete (half-configured) installation.

What's in ovs-ctl.log?

As far as I can see the real start of the command is
/usr/share/openvswitch/ovs-ctl --ike-daemon=strongswan start-ovs-ipsec

If you run this with "sh -x", does this give hints about the cause of the error?

it fails inside the nested script called: lib/lsb/init-functions
and it show what I copy above after:
+ /bin/systemctl --no-pager start openvswitch-ipsec.service

do you think it calls strongswan when it is loaded?

anyhow strongswan is installed:
$ ipsec --version
Linux strongSwan U5.8.2/K5.4.0-26-generic

If you want out help, then please provide the full output and not your interpretation.

glad. here is the full dump.

root@c-236-148-60-063:/ws# sudo sh -x /etc/init.d/openvswitch-ipsec start
+ PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
+ DAEMON=/usr/share/openvswitch/scripts/ovs-monitor-ipsec
+ NAME=ovs-monitor-ipsec
+ LOGDIR=/var/log/openvswitch
+ DATADIR=/usr/share/openvswitch
+ PIDFILE=/var/run/openvswitch/ovs-monitor-ipsec.pid
+ test -x /usr/share/openvswitch/scripts/ovs-monitor-ipsec
+ . /lib/lsb/init-functions
+ run-parts --lsbsysinit --list /lib/lsb/init-functions.d
+ [ -r /lib/lsb/init-functions.d/00-verbose ]
+ . /lib/lsb/init-functions.d/00-verbose
+ [ -r /lib/lsb/init-functions.d/40-systemd ]
+ . /lib/lsb/init-functions.d/40-systemd
+ _use_systemctl=0
+ [ -d /run/systemd/system ]
+ [ -n ]
+ [ openvswitch-ipsec = init-d-script ]
+ [ openvswitch-ipsec = start ]
+ executable=/etc/init.d/openvswitch-ipsec
+ argument=start
+ prog=openvswitch-ipsec
+ service=openvswitch-ipsec.service
+ systemctl -p LoadState --value show openvswitch-ipsec.service
+ state=loaded
+ [ loaded = masked ]
+ [ 86131 -ne 1 ]
+ [ -z ]
+ readlink -f /etc/init.d/openvswitch-ipsec
+ [ loaded != not-found ]
+ _use_systemctl=1
+ systemctl -p CanReload --value show openvswitch-ipsec.service
+ [ no = no ]
+ [ start = reload ]
+ [ 1 = 1 ]
+ set +e
+ set +u
+ systemctl_redirect /etc/init.d/openvswitch-ipsec start
+ local s
+ local rc
+ local prog=openvswitch-ipsec
+ local command=start
+ s=Starting openvswitch-ipsec (via systemctl)
+ service=openvswitch-ipsec.service
+ systemctl is-system-running
+ OUT=degraded
+ [ degraded != degraded ]
+ [ start = status ]
+ log_daemon_msg Starting openvswitch-ipsec (via systemctl) openvswitch-ipsec.service
+ [ -z Starting openvswitch-ipsec (via systemctl) ]
+ log_daemon_msg_pre Starting openvswitch-ipsec (via systemctl) openvswitch-ipsec.service
+ :
+ [ -z openvswitch-ipsec.service ]
+ echo -n Starting openvswitch-ipsec (via systemctl): openvswitch-ipsec.service
Starting openvswitch-ipsec (via systemctl): openvswitch-ipsec.service+ log_daemon_msg_post Starting openvswitch-ipsec (via systemctl) openvswitch-ipsec.service
+ :
+ /bin/systemctl --no-pager start openvswitch-ipsec.service
Job for openvswitch-ipsec.service failed because the control process exited with error code.
See "systemctl status openvswitch-ipsec.service" and "journalctl -xe" for details.
+ rc=1
+ [ start = status ]
+ log_end_msg 1
+ [ -z 1 ]
+ local retval
+ retval=1
+ log_end_msg_pre 1
+ :
+ log_use_fancy_output
+ TPUT=/usr/bin/tput
+ EXPR=/usr/bin/expr
+ [ -t 1 ]
+ [ xxterm-256color != x ]
+ [ xxterm-256color != xdumb ]
+ [ -x /usr/bin/tput ]
+ [ -x /usr/bin/expr ]
+ /usr/bin/tput hpa 60
+ /usr/bin/tput setaf 1
+ [ -z ]
+ FANCYTTY=1
+ true
+ /usr/bin/tput setaf 1
+ RED=
+ /usr/bin/tput setaf 3
+ YELLOW=
+ /usr/bin/tput op
+ NORMAL=
+ [ 1 -eq 0 ]
+ [ 1 -eq 255 ]
+ /bin/echo -e failed!
 failed!
+ log_end_msg_post 1
+ :
+ return 1
+ return 1
+ exit 1
root@c-236-148-60-063:

Sorry, a misunderstanding. In comment #11 I also asked for the output of

sudo sh -x /usr/share/openvswitch/ovs-ctl --ike-daemon=strongswan start-ovs-ipsec

ok. here goes.

ovs-ctl found under scripts, so I change the command a bit.
"sudo sh -x /usr/share/openvswitch/scripts/ovs-ctl --ike-daemon=strongswan start-ovs-ipsec"
it seems that it cannot find vswitch.ovsschema'
is that say something to you?

here is the output:

# sudo sh -x /usr/share/openvswitch/scripts/ovs-ctl --ike-daemon=strongswan start-ovs-ipsec
+ echo /usr/share/openvswitch/scripts/ovs-ctl
+ sed s,/[^/]*$,,
+ dir0=/usr/share/openvswitch/scripts
+ . /usr/share/openvswitch/scripts/ovs-lib
+ logdir=/var/log/openvswitch
+ rundir=/var/run/openvswitch
+ sysconfdir=/etc
+ etcdir=/etc/openvswitch
+ datadir=/usr/share/openvswitch
+ bindir=/usr/bin
+ sbindir=/usr/sbin
+ test X != X
+ test X != X
+ dbdir=/etc/openvswitch
+ VERSION=2.14.1
+ DAEMON_CWD=/
+ LC_ALL=C
+ export LC_ALL
+ test -e /etc/init.d/functions
+ test -e /etc/rc.d/init.d/functions
+ test -e /lib/lsb/init-functions
+ . /lib/lsb/init-functions
+ run-parts --lsbsysinit --list /lib/lsb/init-functions.d
+ [ -r /lib/lsb/init-functions.d/00-verbose ]
+ . /lib/lsb/init-functions.d/00-verbose
+ [ -r /lib/lsb/init-functions.d/40-systemd ]
+ . /lib/lsb/init-functions.d/40-systemd
+ _use_systemctl=0
+ [ -d /run/systemd/system ]
+ [ -n ]
+ [ ovs-ctl = init-d-script ]
+ [ ovs-ctl = --ike-daemon=strongswan ]
+ executable=/usr/share/openvswitch/scripts/ovs-ctl
+ argument=--ike-daemon=strongswan
+ prog=ovs-ctl
+ service=ovs-ctl.service
+ systemctl -p LoadState --value show ovs-ctl.service
+ state=not-found
+ [ not-found = masked ]
+ [ 117308 -ne 1 ]
+ [ -z ]
+ readlink -f /usr/share/openvswitch/scripts/ovs-ctl
+ [ 0 = 1 ]
+ [ -r /lib/lsb/init-functions.d/50-ubuntu-logging ]
+ . /lib/lsb/init-functions.d/50-ubuntu-logging
+ LOG_DAEMON_MSG=
+ [ -r /lib/lsb/init-functions.d/99-plymouth ]
+ . /lib/lsb/init-functions.d/99-plymouth
+ plymouth --ping
+ return
+ FANCYTTY=
+ [ -e /etc/lsb-base-logging.sh ]
+ true
+ type log_success_msg
+ :
+ type log_failure_msg
+ :
+ type log_warning_msg
+ :
+ type action
+ set_defaults
+ SYSTEM_ID=
+ FULL_HOSTNAME=yes
+ DELETE_BRIDGES=no
+ DELETE_TRANSIENT_PORTS=no
+ DAEMON_CWD=/
+ FORCE_COREFILES=yes
+ MLOCKALL=yes
+ SELF_CONFINEMENT=yes
+ MONITOR=yes
+ OVS_USER=
+ OVSDB_SERVER=yes
+ OVS_VSWITCHD=yes
+ OVSDB_SERVER_PRIORITY=-10
+ OVS_VSWITCHD_PRIORITY=-10
+ OVSDB_SERVER_WRAPPER=
+ OVS_VSWITCHD_WRAPPER=
+ OVSDB_SERVER_OPTIONS=
+ OVS_VSWITCHD_OPTIONS=
+ DB_FILE=/etc/openvswitch/conf.db
+ DB_SOCK=/var/run/openvswitch/db.sock
+ DB_SCHEMA=/usr/share/openvswitch/vswitch.ovsschema
+ EXTRA_DBS=
+ PROTOCOL=gre
+ DPORT=
+ SPORT=
+ IKE_DAEMON=
+ RESTART_IKE_DAEMON=yes
+ type_file=/etc/openvswitch/system-type.conf
+ version_file=/etc/openvswitch/system-version.conf
+ test -e /etc/openvswitch/system-type.conf
+ test -e /etc/os-release
+ . /etc/os-release
+ NAME=Ubuntu
+ VERSION=20.04 LTS (Focal Fossa)
+ ID=ubuntu
+ ID_LIKE=debian
+ PRETTY_NAME=Ubuntu 20.04 LTS
+ VERSION_ID=20.04
+ HOME_URL=https://www.ubuntu.com/
+ SUPPORT_URL=https://help.ubuntu.com/
+ BUG_REPORT_URL=https://bugs.launchpad.net/ubuntu/
+ PRIVACY_POLICY_URL=https://www.ubuntu.com/legal/terms-and-policies/privacy-policy
+ VERSION_CODENAME=focal
+ UBUNTU_CODENAME=focal
+ echo ubuntu
+ SYSTEM_TYPE=ubuntu
+ . /etc/os-release
+ NAME=Ubuntu
+ VERSION=20.04 LTS (Focal Fossa)
+ ID=ubuntu
+ ID_LIKE=debian
+ PRETTY_NAME=Ubuntu 20.04 LTS
+ VERSION_ID=20.04
+ HOME_URL=https://www.ubuntu.com/
+ SUPPORT_URL=https://help.ubuntu.com/
+ BUG_REPORT_URL=https://bugs.launchpad.net/ubuntu/
+ PRIVACY_POLICY_URL=https://www.ubuntu.com/legal/terms-and-policies/privacy-policy
+ VERSION_CODENAME=focal
+ UBUNTU_CODENAME=focal
+ echo 20.04
+ SYSTEM_VERSION=20.04
+ extra_ids=
+ command=
+ expr X--ike-daemon=strongswan : X--\([^=]*\)
+ option=ike-daemon
+ expr X--ike-daemon=strongswan : X[^=]*=\(.*\)
+ value=strongswan
+ type=string
+ set_option
+ echo ike-daemon
+ tr abcdefghijklmnopqrstuvwxyz- ABCDEFGHIJKLMNOPQRSTUVWXYZ_
+ var=IKE_DAEMON
+ eval set=${IKE_DAEMON+yes}
+ set=yes
+ eval old_value=$IKE_DAEMON
+ old_value=
+ test Xyes = X
+ test string = bool
+ eval IKE_DAEMON=$value
+ IKE_DAEMON=strongswan
+ test X = X
+ command=start-ovs-ipsec
+ start_ovs_ipsec
+ test Xyes = Xno
+ /usr/share/openvswitch/scripts/ovs-monitor-ipsec --pidfile=/var/run/openvswitch/ovs-monitor-ipsec.pid --ike-daemon=strongswan --log-file --detach --monitor unix:/var/run/openvswitch/db.sock
2021-02-03T06:15:07Z | 0 | ovs-monitor-ipsec | INFO | Restarting IKE daemon
2021-02-03T06:15:07Z | 1 | ovs-monitor-ipsec | INFO | Restarting StrongSwan
Stopping strongSwan IPsec...
Starting strongSwan 5.8.2 IPsec [starter]...
2021-02-03T06:15:09Z | 2 | ovs-monitor-ipsec | ERR | traceback
Traceback (most recent call last):
  File "/usr/share/openvswitch/scripts/ovs-monitor-ipsec", line 1260, in <module>
    main()
  File "/usr/share/openvswitch/scripts/ovs-monitor-ipsec", line 1209, in main
    schema_helper = ovs.db.idl.SchemaHelper()
  File "/usr/local/lib/python3.8/dist-packages/ovs/db/idl.py", line 1959, in __init__
    schema_json = ovs.json.from_file(location)
  File "/usr/local/lib/python3.8/dist-packages/ovs/json.py", line 61, in from_file
    stream = open(name, "r")
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/share/openvswitch/vswitch.ovsschema'
+ return 1

I would expect that the file vswitch.ovsschema is in directory /usr/share/openvswitch/ and not in /usr/local/share/openvswitch/
And idl.py should be /usr/lib/python3/dist-packages/ovs/db/idl.py instead of /usr/local/lib/python3.8/dist-packages/ovs/db/idl.py

It seems to me that you have additional python installations on your system and this now interferes with starting openvswitch.
Or maybe there are remnants of the Ubuntu version of openvswitch that are in conflict with the openvswitch packages that you have built locally.

Do you know anything specific about this?

Eventually the commands

which python
update-alternatives --list python
find /usr -name idl.py -ls
ls -ld /usr/share/pyth*
ls -ld /usr/lib/pyth*
ls -ld /usr/lib/pyth*/dist*

show something unusual.

Thanks a lot. you r right. below at end the output.
yes, I do have prev installation of ovs which is the same 2.14.1.
That's why I looked for the .deb files of that specific version.

Well, I did the following and woalla.
# export PATH=$PATH:/usr/share/openvswitch/scripts
# export OVS_PKGDATADIR=/usr/share/openvswitch

After those settings, openvswitch-ipsec.service loads with no errors ... :-)

$ systemctl status openvswitch-ipsec.service
● openvswitch-ipsec.service - LSB: Open vSwitch GRE-over-IPsec daemon
     Loaded: loaded (/etc/init.d/openvswitch-ipsec; generated)
     Active: active (exited) since Wed 2021-02-03 09:31:51 UTC; 4min 30s ago
       Docs: man:systemd-sysv-generator(8)
    Process: 430613 ExecStart=/etc/init.d/openvswitch-ipsec start (code=exited, status=0/SUCCESS)
Feb 03 09:31:51 lab-bf-sw-6-bf systemd[1]: Starting LSB: Open vSwitch GRE-over-IPsec daemon...
Feb 03 09:31:51 lab-bf-sw-6-bf openvswitch-ipsec[430613]: * Starting ovs-monitor-ipsec
Feb 03 09:31:51 lab-bf-sw-6-bf openvswitch-ipsec[430613]: ...done.
Feb 03 09:31:51 lab-bf-sw-6-bf systemd[1]: Started LSB: Open vSwitch GRE-over-IPsec daemon.

I think ovs-monitor-ipsec simply search the 'vswitch.ovsschema' in wrong place.
thanks for your tips.
BTW, still need to report somewhere that there is no updated public installation for those packages.

LOG of search
===========
which python
/usr/bin/python
(BFL)[root@lab-bf-sw-6-bf ws]$ update-alternatives --list python
update-alternatives: error: no alternatives for python
(BFL)[root@lab-bf-sw-6-bf ws]$ find /usr -name idl.py -ls
   165383 84 -rw-r--r-- 1 root root 84806 Jan 13 16:26 /usr/lib/python3.7/dist-packages/ovs/db/idl.py
   535126 16 -rw-r--r-- 1 root root 14986 Dec 7 2018 /usr/lib/python3/dist-packages/pygments/lexers/idl.py
   808457 84 -rw-r--r-- 1 root staff 84832 Feb 2 10:04 /usr/local/lib/python3.8/dist-packages/ovs/db/idl.py
   564907 84 -rw-r--r-- 1 root root 84806 Jan 13 16:26 /usr/src/openvswitch-2.14.1/python/ovs/db/idl.py
(BFL)[root@lab-bf-sw-6-bf ws]$ ls -ld /usr/share/pyth*
drwxr-xr-x 6 root root 4096 Jan 28 18:47 /usr/share/python
drwxr-xr-x 3 root root 4096 Dec 1 15:25 /usr/share/python-apt
drwxr-xr-x 3 root root 4096 Jan 28 18:47 /usr/share/python-babel-localedata
drwxr-xr-x 2 root root 4096 Jan 28 20:11 /usr/share/python-wheels
drwxr-xr-x 7 root root 4096 Dec 1 15:26 /usr/share/python3
(BFL)[root@lab-bf-sw-6-bf ws]$ ls -ld /usr/lib/pyth*
drwxr-xr-x 27 root root 20480 Jan 28 20:25 /usr/lib/python2.7
drwxr-xr-x 3 root root 4096 Dec 1 15:22 /usr/lib/python3
drwxr-xr-x 3 root root 4096 Feb 2 10:07 /usr/lib/python3.7
drwxr-xr-x 30 root root 4096 Dec 1 15:26 /usr/lib/python3.8
drwxr-xr-x 5 root root 4096 Dec 1 15:26 /usr/lib/python3.9
(BFL)[root@lab-bf-sw-6-bf ws]$ ls -ld /usr/lib/pyth*/dist*
drwxr-xr-x 24 root root 4096 Jan 28 20:11 /usr/lib/python2.7/dist-packages
drwxr-xr-x 3 root root 4096 Dec 6 04:45 /usr/lib/python2.7/distutils
drwxr-xr-x 4 root root 4096 Feb 2 10:08 /usr/lib/python3.7/dist-packages
drwxr-xr-x 4 root root 4096 Dec 1 15:26 /usr/lib/python3.8/distutils
drwxr-xr-x 3 root root 4096 Dec 1 15:26 /usr/lib/python3.9/distutils
drwxr-xr-x 202 root root 12288 Jan 28 20:15 /usr/lib/python3/dist-packages

Ok, now that the openvswitch-ipsec.service loads with no errors you should be able to complete the installation (and make the package management system happy again):

sudo dpkg --configure -a
sudo apt install -f

The question now is, what will happen with openvswitch-ipsec.service after a reboot? I guess that it will fail again, because it will again search on the wrong place. You should consider doing a clean-up with the unused version in the wrong directory.

thanks.
Its already updated as both commands end with no changes
I will add that export to the system environ.
Tx