#StackBounty: #gunicorn #systemctl Enabling systemctl service does not load at boot

Bounty: 50

I have set up gunicorn to server two Django sites, as per this setup. This works well. If I run:

# systemctl start gunicorn@my_website.service

Then all is well and my website is being served as expected. So I move to enable this on boot:

# systemctl enable gunicorn@my_website.service

Let’s check it’s enabled:

# systemctl is-enabled gunicorn@my_website.service
enabled

Looks good. Now, let’s reboot…

Turns out my site is not up. nginx is working, it just seems that gunicorn isn’t doing it’s thing. Let’s investigate:

# systemctl status gunicorn@my_website.service
 ● gunicorn@my_website.service - gunicorn daemon
   Loaded: loaded (/etc/systemd/system/gunicorn@.service; indirect; vendor preset: enabled)
   Active: inactive (dead)

Hmmm… that looks weird. Checking out the log since boot:

# journalctl -u gunicorn@my_website.service -b
-- Logs begin at Mon 2019-04-08 06:04:03 UTC, end at Thu 2020-10-15 13:23:30 UTC. --
-- No entries --

Very puzzling… not even a log entry! When I start the service manually we’re back in operations:

# systemctl start gunicorn@my_website.service
# systemctl status gunicorn@my_website.service
● gunicorn@my_website.service - gunicorn daemon
   Loaded: loaded (/etc/systemd/system/gunicorn@.service; indirect; vendor preset: enabled)
   Active: active (running) since Thu 2020-10-15 13:25:29 UTC; 30s ago
 Main PID: 1272 (gunicorn)
    Tasks: 4 (limit: 1151)
   CGroup: /system.slice/system-gunicorn.slice/gunicorn@my_website.service
           ├─1272 /usr/bin/python3 /usr/local/bin/gunicorn --access-logfile - --workers 3 --bind unix:/home/my_website/gunicorn.sock my_website.wsgi:application
           ├─1294 /usr/bin/python3 /usr/local/bin/gunicorn --access-logfile - --workers 3 --bind unix:/home/my_website/gunicorn.sock my_website.wsgi:application
           ├─1297 /usr/bin/python3 /usr/local/bin/gunicorn --access-logfile - --workers 3 --bind unix:/home/my_website/gunicorn.sock my_website.wsgi:application
           └─1298 /usr/bin/python3 /usr/local/bin/gunicorn --access-logfile - --workers 3 --bind unix:/home/my_website/gunicorn.sock my_website.wsgi:application

Oct 15 13:25:29 web systemd[1]: Started gunicorn daemon.
Oct 15 13:25:29 web gunicorn[1272]: [2020-10-15 13:25:29 +0000] [1272] [INFO] Starting gunicorn 19.9.0
Oct 15 13:25:29 web gunicorn[1272]: [2020-10-15 13:25:29 +0000] [1272] [INFO] Listening at: unix:/home/my_website/gunicorn.sock (1272)
Oct 15 13:25:29 web gunicorn[1272]: [2020-10-15 13:25:29 +0000] [1272] [INFO] Using worker: sync
Oct 15 13:25:29 web gunicorn[1272]: [2020-10-15 13:25:29 +0000] [1294] [INFO] Booting worker with pid: 1294
Oct 15 13:25:29 web gunicorn[1272]: [2020-10-15 13:25:29 +0000] [1297] [INFO] Booting worker with pid: 1297
Oct 15 13:25:29 web gunicorn[1272]: [2020-10-15 13:25:29 +0000] [1298] [INFO] Booting worker with pid: 1298

And indeed, my website works now! But why is this seemingly not even run on boot when it’s enabled? How to debug this?

Running Ubuntu 18.04.

As requested, the content of the gunicorn@.service file below:

# cat /etc/systemd/system/gunicorn@.service
[Unit]
Description=gunicorn daemon
After=network.target
PartOf=gunicorn.target
# Since systemd 235 reloading target can pass through
ReloadPropagatedFrom=gunicorn.target

[Service]
User=ubuntu
Group=www-data
WorkingDirectory=/home/%i/
ExecStart=/usr/local/bin/gunicorn 
          --access-logfile - 
          --workers 3 
          --bind unix:/home/%i/gunicorn.sock 
          %i.wsgi:application

[Install]
WantedBy=gunicorn.target


Get this bounty!!!

#StackBounty: #linux #memory #apache-http-server #systemctl Why does systemctl show high memory usage when top and htop show much less?

Bounty: 50

I’m running an apache2.4 web server, and noticed that when I run systemctl status apache2 or run service apache2 status, it says it is using 4.2GiB of RAM, which isn’t a problem for the amount of RAM I have, but I am confused, since top and htop both show only 500MiB of RAM being used, much less that systemctl reports apache2 using alone. Why might this be, and which is right?

OS Info, if needed:

OS: Kali GNU/Linux Rolling x86_64
Kernel: 4.19.0-kali3-amd64


Get this bounty!!!

#StackBounty: #systemd #systemd-journald #yocto #systemctl #journalctl systemd-journald persistent logs do not work with bind mount /va…

Bounty: 200

I am using Yocto to produce a custom image for a small embedded Linux system with SystemD Version 241. The root file system is Read-Only. I am using bind mounts and overlayfs to make the /var/log/journal directory exist on a seperate Read/Write partition. I have a problem where systemd-journald gets “Amnesia” and does not remember previous boot logs, even though they are on the persistent Read/Write filesystem. This means journal cannot access or clean older log files from previous boots, even though the log files are present and valid on the filesystem.

Yocto volatile binds

# Setup overlayfs binds for various RW files
VOLATILE_BINDS_append = " 
    /persistent-storage/var/log /var/logn
"

The path /var/log exists:

root@me:/var/log# cd /var/log/
root@me:/var/log# ls -lrt
total 9
drwxr-xr-x 2 root root            1024 Jun  3 01:50 nginx
-rw-r--r-- 1 root root            5260 Jun  9 17:56 messages
drwxr-sr-x 5 root systemd-journal 1024 Jun  9 18:00 journal
root@me:/var/log# ls -lrt journal/
total 3
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 17:56 5f6085cd81114e8688cf23b3bb91933e
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 17:57 de59603d1ea24e7582ed7d7ed3ac8fb0
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 18:00 0c34cc794e6c4241a75774bbb8324102

I have a journald config file fragment in /lib/systemd/journald.conf.d/10-persistent-journal.conf that looks like this:

# By default the maximum use limit (SystemMaxUse) is 10% of the filesystem, and the minimum
# free space (SystemKeepFree) value is 15% - though they are both capped at 4G.
# The journals should be rotated automatically when they reach the SystemMaxFileSize value,
# and the number of journals is controlled by SystemMaxFiles. If you prefer time based
# rotation you can set a MaxFileSec to set the maximum time entries are stored in a single journal.
[Journal]
Storage=persistent
SystemMaxFileSize=128M
SystemMaxFiles=10
SystemMaxUse=256M
SystemKeepFree=256M
SyncIntervalSec=30

The problem is that even though I reboot several times, and journald successfully finds and writes logs to /var/log/journal, it can never find previous logs and has no knowledge about previous boot logs. This means I cannot vacuum previous logs and my partition runs out of space even though journald should maintain 50% of the partition free.

root@me:/# journalctl --list-boots
0 82fef865e29e481aae27bd247c10e591 Tue 2020-06-09 18:00:12 UTC—Tue 2020-06-09
 18:15:23 UTC

Even though:

root@me:/# ls -lrt /var/log/journal/
total 3
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 17:56 5f6085cd81114e8688cf23b3bb91933e
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 17:57 de59603d1ea24e7582ed7d7ed3ac8fb0
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 18:00 0c34cc794e6c4241a75774bbb8324102

Also, the following commands work:

root@me:/# journalctl -b 0
<information>
root@me:/# journalctl -b 1
<information>

root@me:/# journalctl -b 2
Data from the specified boot (+2) is not available: No such boot ID in journal

I read this post: Can be the journal path on a filesystem other than /?. And I tried the following mount file, however I see exactly the same behavior:

[Unit]
Description=Persistent Journal Storage Bind

[Mount]
What=/anotherfs/journal
Where=/var/log/journal
Type=none
Options=bind

[Install]
WantedBy=local-fs.target

What am I doing wrong and how can I get journald to work with persistent logs on a bind mount system?


Get this bounty!!!

#StackBounty: #debian #systemd #services #systemctl systemd: Can't unmask root mount (-.mount)

Bounty: 50

(Dist: Debian 10)

I have a reoccurring error message that mainly pops up when using systemctl (also when installing a package, and occasionally in a few other places that escape me),

Unit -.mount is masked.

Sometimes (depending on what command called the error message) it is more verbose, such as

Error: GDBus.Error:org.freedesktop.systemd1.UnitMasked: Unit -.mount is masked.

This error doesn’t impede installing packages or any systemd services which are enabled already (and as such are loaded at boot), but using systemctl or service to restart, start or stop a service fails. This means I have to reboot the whole server to restart a service, which can be a little annoying.

Trying to unmask the root mount with systemctl unmask -- -.mount appears to work (nothing is returned), but systemctl status -- -.mount still outputs the following after:

● -.mount - Root Mount
   Loaded: masked (Reason: Unit -.mount is masked.)
   Active: active (mounted) since Mon 2019-08-05 15:03:38 AEST; 4h 8min ago
    Where: /
     What: /dev/sde1
    Tasks: 0 (limit: 4915)
   Memory: 0B
   CGroup: /system.slice/-.mount

Any ideas? I’m don’t want to start from a fresh install for this server, so either I find a fix or just deal with having to restart if I need to reload a service.


Get this bounty!!!

#StackBounty: #debian #systemd #services #systemctl systemd: Can't unmask root mount (-.mount)

Bounty: 50

(Dist: Debian 10)

I have a reoccurring error message that mainly pops up when using systemctl (also when installing a package, and occasionally in a few other places that escape me),

Unit -.mount is masked.

Sometimes (depending on what command called the error message) it is more verbose, such as

Error: GDBus.Error:org.freedesktop.systemd1.UnitMasked: Unit -.mount is masked.

This error doesn’t impede installing packages or any systemd services which are enabled already (and as such are loaded at boot), but using systemctl or service to restart, start or stop a service fails. This means I have to reboot the whole server to restart a service, which can be a little annoying.

Trying to unmask the root mount with systemctl unmask -- -.mount appears to work (nothing is returned), but systemctl status -- -.mount still outputs the following after:

● -.mount - Root Mount
   Loaded: masked (Reason: Unit -.mount is masked.)
   Active: active (mounted) since Mon 2019-08-05 15:03:38 AEST; 4h 8min ago
    Where: /
     What: /dev/sde1
    Tasks: 0 (limit: 4915)
   Memory: 0B
   CGroup: /system.slice/-.mount

Any ideas? I’m don’t want to start from a fresh install for this server, so either I find a fix or just deal with having to restart if I need to reload a service.


Get this bounty!!!

#StackBounty: #debian #systemd #services #systemctl systemd: Can't unmask root mount (-.mount)

Bounty: 50

(Dist: Debian 10)

I have a reoccurring error message that mainly pops up when using systemctl (also when installing a package, and occasionally in a few other places that escape me),

Unit -.mount is masked.

Sometimes (depending on what command called the error message) it is more verbose, such as

Error: GDBus.Error:org.freedesktop.systemd1.UnitMasked: Unit -.mount is masked.

This error doesn’t impede installing packages or any systemd services which are enabled already (and as such are loaded at boot), but using systemctl or service to restart, start or stop a service fails. This means I have to reboot the whole server to restart a service, which can be a little annoying.

Trying to unmask the root mount with systemctl unmask -- -.mount appears to work (nothing is returned), but systemctl status -- -.mount still outputs the following after:

● -.mount - Root Mount
   Loaded: masked (Reason: Unit -.mount is masked.)
   Active: active (mounted) since Mon 2019-08-05 15:03:38 AEST; 4h 8min ago
    Where: /
     What: /dev/sde1
    Tasks: 0 (limit: 4915)
   Memory: 0B
   CGroup: /system.slice/-.mount

Any ideas? I’m don’t want to start from a fresh install for this server, so either I find a fix or just deal with having to restart if I need to reload a service.


Get this bounty!!!

#StackBounty: #debian #systemd #services #systemctl systemd: Can't unmask root mount (-.mount)

Bounty: 50

(Dist: Debian 10)

I have a reoccurring error message that mainly pops up when using systemctl (also when installing a package, and occasionally in a few other places that escape me),

Unit -.mount is masked.

Sometimes (depending on what command called the error message) it is more verbose, such as

Error: GDBus.Error:org.freedesktop.systemd1.UnitMasked: Unit -.mount is masked.

This error doesn’t impede installing packages or any systemd services which are enabled already (and as such are loaded at boot), but using systemctl or service to restart, start or stop a service fails. This means I have to reboot the whole server to restart a service, which can be a little annoying.

Trying to unmask the root mount with systemctl unmask -- -.mount appears to work (nothing is returned), but systemctl status -- -.mount still outputs the following after:

● -.mount - Root Mount
   Loaded: masked (Reason: Unit -.mount is masked.)
   Active: active (mounted) since Mon 2019-08-05 15:03:38 AEST; 4h 8min ago
    Where: /
     What: /dev/sde1
    Tasks: 0 (limit: 4915)
   Memory: 0B
   CGroup: /system.slice/-.mount

Any ideas? I’m don’t want to start from a fresh install for this server, so either I find a fix or just deal with having to restart if I need to reload a service.


Get this bounty!!!

#StackBounty: #debian #systemd #services #systemctl systemd: Can't unmask root mount (-.mount)

Bounty: 50

(Dist: Debian 10)

I have a reoccurring error message that mainly pops up when using systemctl (also when installing a package, and occasionally in a few other places that escape me),

Unit -.mount is masked.

Sometimes (depending on what command called the error message) it is more verbose, such as

Error: GDBus.Error:org.freedesktop.systemd1.UnitMasked: Unit -.mount is masked.

This error doesn’t impede installing packages or any systemd services which are enabled already (and as such are loaded at boot), but using systemctl or service to restart, start or stop a service fails. This means I have to reboot the whole server to restart a service, which can be a little annoying.

Trying to unmask the root mount with systemctl unmask -- -.mount appears to work (nothing is returned), but systemctl status -- -.mount still outputs the following after:

● -.mount - Root Mount
   Loaded: masked (Reason: Unit -.mount is masked.)
   Active: active (mounted) since Mon 2019-08-05 15:03:38 AEST; 4h 8min ago
    Where: /
     What: /dev/sde1
    Tasks: 0 (limit: 4915)
   Memory: 0B
   CGroup: /system.slice/-.mount

Any ideas? I’m don’t want to start from a fresh install for this server, so either I find a fix or just deal with having to restart if I need to reload a service.


Get this bounty!!!

#StackBounty: #debian #systemd #services #systemctl systemd: Can't unmask root mount (-.mount)

Bounty: 50

(Dist: Debian 10)

I have a reoccurring error message that mainly pops up when using systemctl (also when installing a package, and occasionally in a few other places that escape me),

Unit -.mount is masked.

Sometimes (depending on what command called the error message) it is more verbose, such as

Error: GDBus.Error:org.freedesktop.systemd1.UnitMasked: Unit -.mount is masked.

This error doesn’t impede installing packages or any systemd services which are enabled already (and as such are loaded at boot), but using systemctl or service to restart, start or stop a service fails. This means I have to reboot the whole server to restart a service, which can be a little annoying.

Trying to unmask the root mount with systemctl unmask -- -.mount appears to work (nothing is returned), but systemctl status -- -.mount still outputs the following after:

● -.mount - Root Mount
   Loaded: masked (Reason: Unit -.mount is masked.)
   Active: active (mounted) since Mon 2019-08-05 15:03:38 AEST; 4h 8min ago
    Where: /
     What: /dev/sde1
    Tasks: 0 (limit: 4915)
   Memory: 0B
   CGroup: /system.slice/-.mount

Any ideas? I’m don’t want to start from a fresh install for this server, so either I find a fix or just deal with having to restart if I need to reload a service.


Get this bounty!!!

#StackBounty: #debian #systemd #services #systemctl systemd: Can't unmask root mount (-.mount)

Bounty: 50

(Dist: Debian 10)

I have a reoccurring error message that mainly pops up when using systemctl (also when installing a package, and occasionally in a few other places that escape me),

Unit -.mount is masked.

Sometimes (depending on what command called the error message) it is more verbose, such as

Error: GDBus.Error:org.freedesktop.systemd1.UnitMasked: Unit -.mount is masked.

This error doesn’t impede installing packages or any systemd services which are enabled already (and as such are loaded at boot), but using systemctl or service to restart, start or stop a service fails. This means I have to reboot the whole server to restart a service, which can be a little annoying.

Trying to unmask the root mount with systemctl unmask -- -.mount appears to work (nothing is returned), but systemctl status -- -.mount still outputs the following after:

● -.mount - Root Mount
   Loaded: masked (Reason: Unit -.mount is masked.)
   Active: active (mounted) since Mon 2019-08-05 15:03:38 AEST; 4h 8min ago
    Where: /
     What: /dev/sde1
    Tasks: 0 (limit: 4915)
   Memory: 0B
   CGroup: /system.slice/-.mount

Any ideas? I’m don’t want to start from a fresh install for this server, so either I find a fix or just deal with having to restart if I need to reload a service.


Get this bounty!!!