#StackBounty: #systemd #systemd-journald #yocto #systemctl #journalctl systemd-journald persistent logs do not work with bind mount /va…

Bounty: 200

I am using Yocto to produce a custom image for a small embedded Linux system with SystemD Version 241. The root file system is Read-Only. I am using bind mounts and overlayfs to make the /var/log/journal directory exist on a seperate Read/Write partition. I have a problem where systemd-journald gets “Amnesia” and does not remember previous boot logs, even though they are on the persistent Read/Write filesystem. This means journal cannot access or clean older log files from previous boots, even though the log files are present and valid on the filesystem.

Yocto volatile binds

# Setup overlayfs binds for various RW files
VOLATILE_BINDS_append = " 
    /persistent-storage/var/log /var/logn
"

The path /var/log exists:

root@me:/var/log# cd /var/log/
root@me:/var/log# ls -lrt
total 9
drwxr-xr-x 2 root root            1024 Jun  3 01:50 nginx
-rw-r--r-- 1 root root            5260 Jun  9 17:56 messages
drwxr-sr-x 5 root systemd-journal 1024 Jun  9 18:00 journal
root@me:/var/log# ls -lrt journal/
total 3
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 17:56 5f6085cd81114e8688cf23b3bb91933e
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 17:57 de59603d1ea24e7582ed7d7ed3ac8fb0
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 18:00 0c34cc794e6c4241a75774bbb8324102

I have a journald config file fragment in /lib/systemd/journald.conf.d/10-persistent-journal.conf that looks like this:

# By default the maximum use limit (SystemMaxUse) is 10% of the filesystem, and the minimum
# free space (SystemKeepFree) value is 15% - though they are both capped at 4G.
# The journals should be rotated automatically when they reach the SystemMaxFileSize value,
# and the number of journals is controlled by SystemMaxFiles. If you prefer time based
# rotation you can set a MaxFileSec to set the maximum time entries are stored in a single journal.
[Journal]
Storage=persistent
SystemMaxFileSize=128M
SystemMaxFiles=10
SystemMaxUse=256M
SystemKeepFree=256M
SyncIntervalSec=30

The problem is that even though I reboot several times, and journald successfully finds and writes logs to /var/log/journal, it can never find previous logs and has no knowledge about previous boot logs. This means I cannot vacuum previous logs and my partition runs out of space even though journald should maintain 50% of the partition free.

root@me:/# journalctl --list-boots
0 82fef865e29e481aae27bd247c10e591 Tue 2020-06-09 18:00:12 UTC—Tue 2020-06-09
 18:15:23 UTC

Even though:

root@me:/# ls -lrt /var/log/journal/
total 3
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 17:56 5f6085cd81114e8688cf23b3bb91933e
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 17:57 de59603d1ea24e7582ed7d7ed3ac8fb0
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 18:00 0c34cc794e6c4241a75774bbb8324102

Also, the following commands work:

root@me:/# journalctl -b 0
<information>
root@me:/# journalctl -b 1
<information>

root@me:/# journalctl -b 2
Data from the specified boot (+2) is not available: No such boot ID in journal

I read this post: Can be the journal path on a filesystem other than /?. And I tried the following mount file, however I see exactly the same behavior:

[Unit]
Description=Persistent Journal Storage Bind

[Mount]
What=/anotherfs/journal
Where=/var/log/journal
Type=none
Options=bind

[Install]
WantedBy=local-fs.target

What am I doing wrong and how can I get journald to work with persistent logs on a bind mount system?


Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.