TLS bridge connections suddenly rejected after several days of success

Asked by Andy Murdoch

Hi, I'm experiencing a weird issue and not sure where to start debugging.

I have a mosquitto server that accepts incoming connections over TLSv1.2 with a client certificate. I have a whole bunch of devices that set up a bridge to that server and communicate quite happily. Client certificates are unique and used to identify individual devices. So far so good.

From time to time, devices lose connection due to Wi-Fi or power issues but reconnect automatically and successfully. Still good so far.

Then, after the mosquitto server has been running uninterrupted for about 30-40 days, every single device that attempts to reconnect gets its connection rejected with the following in the server log:

    Jun 3 06:33:04 www-01 mosquitto[13267]: OpenSSL Error: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned
    Jun 3 06:33:04 www-01 mosquitto[13267]: Socket error on client <unknown>, disconnecting.

There is not another successful connection via TLS until I restart the server. After restart, everything is good again so the client certs have not expired.

Has anyone experienced anything similar? I see nothing unusual in any other logs, the system time is sane, I am at a loss as to where I should look next.

Thanks.

Question information

Language:
English Edit question
Status:
Solved
For:
mosquitto Edit question
Assignee:
No assignee Edit question
Solved by:
Andy Murdoch
Solved:
Last query:
Last reply:
Revision history for this message
Roger Light (roger.light) said :
#1

What version of mosquitto and what platform? Can you post your config?

Revision history for this message
Andy Murdoch (an7y) said :
#2

Thanks for the response. Here's the info:

On the devices:
=============

mosquitto version 1.4.5

Config File:

pid_file /var/run/mosquitto.pid
bind_address localhost

# =================================================================
# Bridges
# =================================================================

# Experimental support for connecting multiple MQTT brokers
# together.
# Specify multiple sets of connection, address and topic
# configurations.
# Each connection must have a unique name.
# Only a single address per configuration is currently supported,
# unlike in rsmb.
# The direction that the topic will be shared can be chosen by
# specifying out, in or both, where the default value is out.
# The QoS level of the bridged communication can be specified with the final
# topic option. The default QoS level is 2, to change the QoS the topic
# direction must also be given.
# Multiple topics can be specified per connection, but be careful
# not to create any loops.
connection ipmbridge
address mqtt.emberenergy.com.au:8883
#bridge_cafile /etc/ssl/certs/IPM_CA_cert.pem
bridge_capath /etc/ssl/certs
bridge_certfile /var/data/cert/spkac-cert.pem
bridge_keyfile /var/data/cert/spkac-key-nopass.pem

# Installation-specific bridge configuration---

# Set the client id for this bridge connection. If not defined,
# this defaults to 'name.hostname' where name is the connection
# name and hostname is the hostname of this computer.
clientid D429170AE956C8DB.hpc.beatenergy.com

# Define a topic pattern to be shared between the two brokers.
# Any topics matching the pattern (which may include wildcards) are shared.
topic /to_hpc/D429170AE956C8DB.hpc.beatenergy.com/# in 0 "" ""
topic /frm_hpc/D429170AE956C8DB.hpc.beatenergy.com/# out 0 "" ""

# Set the clean session variable for this bridge.
# When set to true, when the bridge disconnects for any reason, all
# messages and subscriptions will be cleaned up on the remote
# broker. Note that with cleansession set to true, there may be a
# significant amount of retained messages sent when the bridge
# reconnects after losing its connection.
# When set to false, the subscriptions and messages are kept on the
# remote broker, and delivered when the bridge reconnects.
cleansession true

And on the server:
===============

mosquitto version 1.4.8

Config File:

# Config file for mosquitto
#
# See mosquitto.conf(5) for more information.
#
# Default values are shown, uncomment to change.
#
# Use the # character to indicate a comment, but only if it is the
# very first character on the line.

# =================================================================
# General configuration
# =================================================================

# Time in seconds to wait before resending an outgoing QoS=1 or
# QoS=2 message.
#retry_interval 20

# Time in seconds between updates of the $SYS tree.
#sys_interval 10

# Time in seconds between cleaning the internal message store of
# unreferenced messages. Lower values will result in lower memory
# usage but more processor time, higher values will have the
# opposite effect.
# Setting a value of 0 means the unreferenced messages will be
# disposed of as quickly as possible.
#store_clean_interval 10

# Write process id to a file. Default is a blank string which means
# a pid file shouldn't be written.
# This should be set to /var/run/mosquitto.pid if mosquitto is
# being run automatically on boot with an init script and
# start-stop-daemon or similar.
pid_file /var/run/mosquitto.pid

# When run as root, drop privileges to this user and its primary
# group.
# Leave blank to stay as root, but this is not recommended.
# If run as a non-root user, this setting has no effect.
# Note that on Windows this has no effect and so mosquitto should
# be started by the user you wish it to run as.
user mosquitto

# The maximum number of QoS 1 and 2 messages currently inflight per
# client.
# This includes messages that are partway through handshakes and
# those that are being retried. Defaults to 10. Set to 0 for no
# maximum. Setting to 1 will guarantee in-order delivery of QoS 1
# and 2 messages.
#max_inflight_messages 10

# The maximum number of QoS 1 and 2 messages to hold in a queue
# above those that are currently in-flight. Defaults to 100. Set
# to 0 for no maximum (not recommended).
#max_queued_messages 100

# =================================================================
# Default listener
# =================================================================

# IP address/hostname to bind the default listener to. If not
# given, the default listener will not be bound to a specific
# address and so will be accessible to all network interfaces.
# bind_address ip-address/host name
#bind_address

# Port to use for the default listener.
port 1883

# The maximum number of client connections to allow. This is
# a per listener setting.
# Default is -1, which means unlimited connections.
# Note that other process limits mean that unlimited connections
# are not really possible. Typically the default maximum number of
# connections possible is around 1024.
#max_connections -1

# =================================================================
# Extra listeners
# =================================================================

# Listen on a port/ip address combination. By using this variable
# multiple times, mosquitto can listen on more than one port. If
# this variable is used and neither bind_address nor port given,
# then the default listener will not be started.
# The port number to listen on must be given. Optionally, an ip
# address or host name may be supplied as a second argument. In
# this case, mosquitto will attempt to bind the listener to that
# address and so restrict access to the associated network and
# interface. By default, mosquitto will listen on all interfaces.
# listener port-number [ip address/host name]
listener 8883
cafile /etc/ssl/certs/IPM_CA_cert.pem
#capath /etc/ssl/certs
certfile /etc/ssl/certs/mqtt.emberenergy.com.au.chained.cert
keyfile /etc/mosquitto/keys/mqtt.emberenergy.com.au.key
tls_version tlsv1.2

require_certificate true
crlfile /var/www/ipm/crl/ipm.crl.pem
use_identity_as_username true
allow_anonymous false

# The maximum number of client connections to allow. This is
# a per listener setting.
# Default is -1, which means unlimited connections.
# Note that other process limits mean that unlimited connections
# are not really possible. Typically the default maximum number of
# connections possible is around 1024.
#max_connections -1

# The listener can be restricted to operating within a topic hierarchy using
# the mount_point option. This is achieved be prefixing the mount_point string
# to all topics for any clients connected to this listener. This prefixing only
# happens internally to the broker; the client will not see the prefix.
#mount_point

# =================================================================
# Persistence
# =================================================================

# If persistence is enabled, save the in-memory database to disk
# every autosave_interval seconds. If set to 0, the persistence
# database will only be written when mosquitto exits.
# Note that writing of the persistence database can be forced by
# sending mosquitto a SIGUSR1 signal.
#autosave_interval 1800

# Save persistent message data to disk (true/false).
# This saves information about all messages, including
# subscriptions, currently in-flight messages and retained
# messages.
# retained_persistence is a synonym for this option.
persistence false

# The filename to use for the persistent database, not including
# the path.
#persistence_file mosquitto.db

# Location for persistent database. Must include trailing /
# Default is an empty string (current directory).
# Set to /var/lib/mosquitto/ if running as a proper service.
persistence_location /var/lib/mosquitto/

# =================================================================
# Logging
# =================================================================

# Places to log to. Use multiple log_dest lines for multiple
# logging destinations.
# Possible destinations are: stdout stderr syslog topic
# stdout and stderr log to the console on the named output.
# syslog uses the userspace syslog facility which usually ends up
# in /var/log/messages or similar.
# topic logs to the broker topic '$SYS/broker/log/<severity>',
# where severity is one of D, E, W, N, I which are debug, error,
# warning, notice and information.
# Use "log_dest none" if you wish to disable logging.
log_dest syslog

# Types of messages to log. Use multiple log_type lines for logging
# multiple types of messages.
# Possible types are: debug, error, warning, notice, information,
# none.
# Note that debug type messages are for decoding the incoming
# network packets.
# They are not logged in syslog.
log_type error
log_type warning
log_type notice
log_type information

# If set to true, client connection and disconnection messages will be included
# in the log.
connection_messages true

# If set to true, add a timestamp value to each log message.
log_timestamp true

# =================================================================
# Security
# =================================================================

# If set, only clients that have a matching prefix on their
# clientid will be allowed to connect to the broker. By default,
# all clients may connect.
# For example, setting "secure-" here would mean a client "secure-
# client" could connect but another with clientid "mqtt" couldn't.
#clientid_prefixes

# Boolean value that determines whether clients that connect
# without providing a username are allowed to connect. If set to
# false then a password file should be created (see the
# password_file option) to control authenticated client access.
# Defaults to true.
allow_anonymous false

# Control access to the broker using a password file. The file is a
# text file # of lines in the format:
# username:password
# The password (and colon) may be omitted if desired, although this
# offers very little in the way of security.
password_file /etc/mosquitto/passwd

# Control access to topics on the broker using an access control list
# file. If this parameter is defined then only the topics listed will
# have access.
# Topic access is added with lines of the format:
#
# topic [read|write] <topic>
#
# The access type is controlled using "read" or "write". This parameter
# is optional - if not given then the access is read/write.
# <topic> can contain the + or # wildcards as in subscriptions.
#
# The first set of topics are applied to anonymous clients, assuming
# allow_anonymous is true. User specific topic ACLs are added after a
# user line as follows:
#
# user <username>
#
# The username referred to here is the same as in password_file. It is
# not the clientid.
#
#
# If is also possible to define ACLs based on pattern substitution within the
# topic. The patterns available for substition are:
#
# %c to match the client id of the client
# %u to match the username of the client
#
# The substitution pattern must be the only text for that level of hierarchy.
#
# The form is the same as for the topic keyword, but using pattern as the
# keyword.
# Pattern ACLs apply to all users even if the "user" keyword has previously
# been given.
#
# pattern [read|write] <topic>
#
# Example:
#
# pattern write sensor/%u/data
#
acl_file /etc/mosquitto/acl.conf

# =================================================================
# Bridges
# =================================================================

# Experimental support for connecting multiple MQTT brokers
# together.
# Specify multiple sets of connection, address and topic
# configurations.
# Each connection must have a unique name.
# Only a single address per configuration is currently supported,
# unlike in rsmb.
# The direction that the topic will be shared can be chosen by
# specifying out, in or both, where the default value is out.
# The QoS level of the bridged communication can be specified with the final
# topic option. The default QoS level is 2, to change the QoS the topic
# direction must also be given.
# Multiple topics can be specified per connection, but be careful
# not to create any loops.
#connection testbridge
#address 192.168.14.68
#topic <topic> [[out | in | both] qos-level]
#topic /test/# in 2 /ipm ""
#topic /test/# out 2 "" /hpc/%u

# Set the client id for this bridge connection. If not defined,
# this defaults to 'name.hostname' where name is the connection
# name and hostname is the hostname of this computer.
#clientid

# Set the clean session variable for this bridge.
# When set to true, when the bridge disconnects for any reason, all
# messages and subscriptions will be cleaned up on the remote
# broker. Note that with cleansession set to true, there may be a
# significant amount of retained messages sent when the bridge
# reconnects after losing its connection.
# When set to false, the subscriptions and messages are kept on the
# remote broker, and delivered when the bridge reconnects.
#cleansession true

# If set to true, publish notification messages to the local and remote brokers
# giving information about the state of the bridge connection. Retained
# messages are published to the topic $SYS/bridge/connection/<clientid>/state.
# If the message is 1 then the connection is active, or 0 if the connection has
# failed.
#notifications true

# Set the keepalive interval for this bridge connection, in
# seconds.
#keepalive_interval 60

# Set the start type of the bridge. This controls how the bridge starts and
# can be one of three types: automatic, lazy and once. Note that RSMB provides
# a fourth start type "manual" which isn't currently supported by mosquitto.
#
# "automatic" is the default start type and means that the bridge connection
# will be started automatically when the broker starts and also restarted
# after a short delay (30 seconds) if the connection fails.
#
# Bridges using the "lazy" start type will be started automatically when the
# number of queued messages exceeds the number set with the "threshold"
# parameter. It will be stopped automatically after the time set by the
# "idle_timeout" parameter. Use this start type if you wish the connection to
# only be active when it is needed.
#
# A bridge using the "once" start type will be started automatically when the
# broker starts but will not be restarted if the connection fails.
#start_type automatic

# Set the amount of time a bridge using the lazy start type must be idle before
# it will be stopped. Defaults to 60 seconds.
#idle_timeout 60

# Set the number of messages that need to be queued for a bridge with lazy
# start type to be restarted. Defaults to 10 messages.
# Must be less than max_queued_messages.
#threshold 10

# Set the username to use when connecting to an MQTT v3.1 broker
# that requires authentication.
#username

# Set the password to use when connecting to an MQTT v3.1 broker
# that requires authentication. This option is only used if
# username is also set.
#password

# =================================================================
# External security checks
# =================================================================

# This section defines options for potential use with external
# databases.
# They are intended to make it easier to add external security
# checks along with WITH_EXT_SECURITY_CHECKS. If you haven't written
# code to support another database and compiled support in, you are
# unlikely to need to change anything here.

# Database hostname.
#db_host

# Database port.
#db_port

# Database name.
#db_name

# Database username.
#db_username

# Database password.
#db_password

# =================================================================
# Unsupported rsmb options - for the future
# =================================================================

#autosave_on_changes
#addresses
#notification_topic
#round_robin

# =================================================================
# rsmb options - unlikely to ever be supported
# =================================================================

#ffdc_output
#max_log_entries
#trace_level
#trace_output
#try_private

Revision history for this message
Roger Light (roger.light) said :
#3

This is just removing the commented lines to make it easier to read

On the devices:
=============

mosquitto version 1.4.5

Config File:

pid_file /var/run/mosquitto.pid
bind_address localhost

connection ipmbridge
address mqtt.emberenergy.com.au:8883

bridge_capath /etc/ssl/certs
bridge_certfile /var/data/cert/spkac-cert.pem
bridge_keyfile /var/data/cert/spkac-key-nopass.pem

clientid D429170AE956C8DB.hpc.beatenergy.com

topic /to_hpc/D429170AE956C8DB.hpc.beatenergy.com/# in 0 "" ""
topic /frm_hpc/D429170AE956C8DB.hpc.beatenergy.com/# out 0 "" ""

cleansession true

And on the server:
===============

mosquitto version 1.4.8

Config File:

pid_file /var/run/mosquitto.pid

user mosquitto

port 1883

listener 8883
cafile /etc/ssl/certs/IPM_CA_cert.pem
certfile /etc/ssl/certs/mqtt.emberenergy.com.au.chained.cert
keyfile /etc/mosquitto/keys/mqtt.emberenergy.com.au.key
tls_version tlsv1.2

require_certificate true
crlfile /var/www/ipm/crl/ipm.crl.pem
use_identity_as_username true
allow_anonymous false

persistence false

persistence_location /var/lib/mosquitto/

log_dest syslog

log_type error
log_type warning
log_type notice
log_type information

connection_messages true

log_timestamp true

allow_anonymous false

password_file /etc/mosquitto/passwd

acl_file /etc/mosquitto/acl.conf

Revision history for this message
Launchpad Janitor (janitor) said :
#4

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Revision history for this message
Andy Murdoch (an7y) said :
#5

I think I have figured it out - the CRL file is only ever valid for 30 days but we regularly re-generate it. However if mosquitto runs without restarting for 30 days, it will never re-read the CRL so its in-memory copy will expire and new connections cannot be validated.

Restarting mosquitto causes it to read the updated version and work as before. Is there a way to cause mosquitto to re-read the CRL (or the whole config) without dropping all existing client connections?

Revision history for this message
Andy Murdoch (an7y) said :
#6

I should add that I've checked the mosquitto.conf documentation but sadly none of the "Certificate based SSL/TLS Support" items state whether or not they are reloaded on SIGHUP.

I've had a look through the source and I think that they are not. A restart seems to be the only way.