Ubuntu
systemd package

systemd-tmpfiles: fchownat() of /run/.. failed: Invalid argument

Asked by Rafael on 2018-11-18

Hello everyone,
I'm running 16.04 LTS on a virtual server which, I think, uses OpenVz. After a recent reboot I found most of my services to be in a failed state. The reason for that, I guess, are these log entries:

Nov 17 04:47:42 h2118376 systemd-tmpfiles[165]: fchownat() of /run/elasticsearch failed: Invalid argument
Nov 17 04:47:42 h2118376 systemd-tmpfiles[165]: fchownat() of /run/kopano failed: Invalid argument
Nov 17 04:47:42 h2118376 systemd-tmpfiles[165]: fchownat() of /run/kopano failed: Invalid argument
Nov 17 04:47:42 h2118376 systemd-tmpfiles[165]: fchownat() of /run/php failed: Invalid argument
Nov 17 04:47:42 h2118376 systemd-tmpfiles[165]: fchownat() of /run/postgresql failed: Invalid argument
Nov 17 04:47:42 h2118376 systemd-tmpfiles[165]: fchownat() of /run/redis failed: Invalid argument
Nov 17 04:47:42 h2118376 systemd-tmpfiles[165]: fchownat() of /run/screen failed: Invalid argument
Nov 17 04:47:42 h2118376 systemd-tmpfiles[165]: fchownat() of /run/utmp failed: Invalid argument
Nov 17 04:47:42 h2118376 systemd-tmpfiles[165]: fchownat() of /run/systemd/netif failed: Invalid argument
Nov 17 04:47:42 h2118376 systemd-tmpfiles[165]: fchownat() of /run/systemd/netif/links failed: Invalid argument
Nov 17 04:47:42 h2118376 systemd-tmpfiles[165]: fchownat() of /run/systemd/netif/leases failed: Invalid argument
Nov 17 04:47:42 h2118376 systemd-tmpfiles[165]: fchownat() of /run/log/journal failed: Invalid argument
Nov 17 04:47:42 h2118376 systemd-tmpfiles[165]: fchownat() of /run/log/journal/bbad3a438f4b4fb49e5d0700bd5981e8 failed: Invalid argument
Nov 17 04:47:42 h2118376 systemd-tmpfiles[165]: fchownat() of /run/log/journal/bbad3a438f4b4fb49e5d0700bd5981e8/system.journal failed: Invalid argument

To verify I tried this:

/usr/lib/tmpfiles.d# SYSTEMD_LOG_LEVEL=debug systemd-tmpfiles --create elasticsearch.conf
Reading config file "elasticsearch.conf".
Running create action for entry d /var/run/elasticsearch
Found existing directory "/var/run/elasticsearch".
"/run/elasticsearch" has right mode 40755
chown "/run/elasticsearch" to 120.128
fchownat() of /run/elasticsearch failed: Invalid argument

I can manually chown the directories, e.g. "chown elasticsearch:elasticsearch /var/run/elasticsearch" and restart the service successfully. My suspicion is, this is related to an upgrade of systemd to 229-4ubuntu21.8.

At this point I don't know what to do.

I'm also confused about the version I have installed, which I thought is systemd-229. Howver, I looked at https://github.com/systemd/systemd/blob/v229/src/tmpfiles/tmpfiles.c and found that fchownat() is only used from version 238+:
Tag v237 (and earlier, including 229):
/.../
                        if (chown(fn,
                                  i->uid_set ? i->uid : UID_INVALID,
                                  i->gid_set ? i->gid : GID_INVALID) < 0)
                                return log_error_errno(errno, "chown(%s) failed: %m", path);
}
/.../

Tag v238

/.../
    if (fchownat(fd,
                             "",
                             i->uid_set ? i->uid : UID_INVALID,
                             i->gid_set ? i->gid : GID_INVALID,
                             AT_EMPTY_PATH) < 0)
return log_error_errno(errno, "fchownat() of %s failed: %m", path);
/.../

Any help fixing this problem would be highly appreciated.
Many thanks,
Rafael

Question information

Language:
English Edit question
Status:
Solved
For:
Ubuntu systemd Edit question
Assignee:
No assignee Edit question
Solved by:
Rafael
Solved:
2018-11-28
Last query:
2018-11-28
Last reply:
Rafael (riffraff99) said : #1

a little update:
I did apt-get source for
   - systemd=229-4ubuntu21.8
   - systemd=229-4ubuntu4 (which was my previously installed version)

and compared the two tmpfiles.c. I can confirm that the latest version replaced 'chown' with 'fchownat'. The packages are hosted by my server provider.

So my question is, why is fchownat not working on my system? Is there a chance I misconfigured anything? Can it be related to the virtualization?

Peter Enns (nelway) said : #2

I am having the same issue.

I upgraded my VPS with 'sudo apt-get upgrade' a few days ago which broke Redis on my server. After a little investigation, the cause was that the installation failed with the error "fchownat() of /run/redis failed: Invalid argument".

I tried a clean install of the 16.04 OS, with 2 different VPS providers, and the exact same result both times. If I install the updates, the following installation of several different packages fail with the error message "fchownat() of /run/redis failed: Invalid argument"

Steps to reproduce:
1. Clean install of Ubuntu 16.04 (not sure exactly what each of the VPS providers use as the starting image, both are slightly different)
2. sudo apt update && sudo apt upgrade
3. sudo apt install lamp-server^
4. sudo apt install redis-server

Results in:

Setting up redis-server (2:3.0.6-1) ...
fchownat() of /run/redis failed: Invalid argument
Job for redis-server.service failed because a timeout was exceeded. See "systemctl status redis-server.service" and "journalctl -xe" for details.
invoke-rc.d: initscript redis-server, action "start" failed.

I have not found a solution yet (except not installing the latest version).

Chris E (cbz) said : #3

Raised and linked the following bug to this problem:

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1804847

Rafael (riffraff99) said : #4

Thanks all for taking this up!