#StackBounty: #20.04 #zfs Ubuntu 20.04 with root on ZFS errors after 'apt dist-upgrade'

Bounty: 100

I’ve been running 20.04 w/ root on ZFS for a couple of months now, with no major problems.

This morning, I ran an apt dist-upgrade, and I got the following errors:

mjb@feynmann:~$ sudo apt dist-upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
3 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] 
Setting up zfsutils-linux (0.8.3-1ubuntu12.1) ...
zfs-import-scan.service is a disabled or a static unit, not starting it.
zfs-import-scan.service is a disabled or a static unit, not starting it.
Job for zfs-mount.service failed because a fatal signal was delivered to the control process.
See "systemctl status zfs-mount.service" and "journalctl -xe" for details.
invoke-rc.d: initscript zfs-mount, action "start" failed.
● zfs-mount.service - Mount ZFS filesystems
     Loaded: loaded (/lib/systemd/system/zfs-mount.service; enabled; vendor preset: enabled)
     Active: failed (Result: signal) since Wed 2020-06-03 09:38:01 EDT; 4ms ago
       Docs: man:zfs(8)
    Process: 760032 ExecStart=/sbin/zfs mount -a (code=killed, signal=ABRT)
   Main PID: 760032 (code=killed, signal=ABRT)

Jun 03 09:38:00 feynmann systemd[1]: Starting Mount ZFS filesystems...
Jun 03 09:38:00 feynmann zfs[760032]: cannot mount '/root': directory is not empty
Jun 03 09:38:00 feynmann zfs[760032]: cannot mount '/var/cache': directory is not empty
Jun 03 09:38:00 feynmann zfs[760032]: cannot mount '/var/lib/AccountsService': directory is not empty
Jun 03 09:38:00 feynmann zfs[760032]: free(): double free detected in tcache 2
Jun 03 09:38:01 feynmann systemd[1]: zfs-mount.service: Main process exited, code=killed, status=6/ABRT
Jun 03 09:38:01 feynmann systemd[1]: zfs-mount.service: Failed with result 'signal'.
Jun 03 09:38:01 feynmann systemd[1]: Failed to start Mount ZFS filesystems.
dpkg: error processing package zfsutils-linux (--configure):
 installed zfsutils-linux package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of zfs-initramfs:
 zfs-initramfs depends on zfsutils-linux (>= 0.8.3-1ubuntu12.1); however:
  Package zfsutils-linux is not configured yet.

dpkg: error processing package zfs-initramfs (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of zfs-zed:
 zfs-zed depends on zfsutils-linux (>= 0.8.3-1ubuntu12.1); however:
  Package zfsutils-linux is not configured yet.

dpkg: error processing package zfs-zed (--configure):
 dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          No apport report written because the error message i
ndicates its a followup error from a previous failure.
                                                      Errors were encountered while processing:
 zfsutils-linux
 zfs-initramfs
 zfs-zed
E: Sub-process /usr/bin/dpkg returned an error code (1)

More info:

mjb@feynmann:~$ sudo systemctl status zfs-mount.service
● zfs-mount.service - Mount ZFS filesystems
     Loaded: loaded (/lib/systemd/system/zfs-mount.service; enabled; vendor preset: enabled)
     Active: failed (Result: signal) since Wed 2020-06-03 09:38:01 EDT; 1min 9s ago
       Docs: man:zfs(8)
    Process: 760032 ExecStart=/sbin/zfs mount -a (code=killed, signal=ABRT)
   Main PID: 760032 (code=killed, signal=ABRT)

Jun 03 09:38:00 feynmann systemd[1]: Starting Mount ZFS filesystems...
Jun 03 09:38:00 feynmann zfs[760032]: cannot mount '/root': directory is not empty
Jun 03 09:38:00 feynmann zfs[760032]: cannot mount '/var/cache': directory is not empty
Jun 03 09:38:00 feynmann zfs[760032]: cannot mount '/var/lib/AccountsService': directory is not empty
Jun 03 09:38:00 feynmann zfs[760032]: free(): double free detected in tcache 2
Jun 03 09:38:01 feynmann systemd[1]: zfs-mount.service: Main process exited, code=killed, status=6/ABRT
Jun 03 09:38:01 feynmann systemd[1]: zfs-mount.service: Failed with result 'signal'.
Jun 03 09:38:01 feynmann systemd[1]: Failed to start Mount ZFS filesystems.
mjb@feynmann:~$ sudo journalctl -xe
-- The process' exit code is 'killed' and its exit status is 6.
Jun 03 09:38:01 feynmann systemd[1]: zfs-mount.service: Failed with result 'signal'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit zfs-mount.service has entered the 'failed' state with result 'signal'.
Jun 03 09:38:01 feynmann systemd[1]: Failed to start Mount ZFS filesystems.
-- Subject: A start job for unit zfs-mount.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit zfs-mount.service has finished with a failure.
-- 
-- The job identifier is 16641 and the job result is failed.
Jun 03 09:38:01 feynmann dbus-daemon[2075]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by>
Jun 03 09:38:01 feynmann systemd[1]: Starting PackageKit Daemon...
-- Subject: A start job for unit packagekit.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit packagekit.service has begun execution.
-- 
-- The job identifier is 16649.
Jun 03 09:38:01 feynmann PackageKit[760066]: daemon start
Jun 03 09:38:01 feynmann dbus-daemon[2075]: [system] Successfully activated service 'org.freedesktop.PackageKit'
Jun 03 09:38:01 feynmann systemd[1]: Started PackageKit Daemon.
-- Subject: A start job for unit packagekit.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit packagekit.service has finished successfully.
-- 
-- The job identifier is 16649.
Jun 03 09:38:01 feynmann sudo[759659]: pam_unix(sudo:session): session closed for user root
Jun 03 09:39:10 feynmann sudo[760629]:      mjb : TTY=pts/4 ; PWD=/home/mjb ; USER=root ; COMMAND=/usr/bin/systemctl status zfs-mount.service
Jun 03 09:39:10 feynmann sudo[760629]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 03 09:39:10 feynmann sudo[760629]: pam_unix(sudo:session): session closed for user root
Jun 03 09:39:15 feynmann sudo[760671]:      mjb : TTY=pts/4 ; PWD=/home/mjb ; USER=root ; COMMAND=/usr/bin/journalctl -xe
Jun 03 09:39:15 feynmann sudo[760671]: pam_unix(sudo:session): session opened for user root by (uid=0)

Even more info:

It appears that the problem is with apt install zfsuitls-linux, which gets:

mjb@feynmann:~$ sudo apt install zfsutils-linux
Reading package lists... Done
Building dependency tree       
Reading state information... Done
zfsutils-linux is already the newest version (0.8.3-1ubuntu12.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
3 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] 
Setting up zfsutils-linux (0.8.3-1ubuntu12.1) ...
zfs-import-scan.service is a disabled or a static unit, not starting it.
zfs-import-scan.service is a disabled or a static unit, not starting it.
Job for zfs-mount.service failed because a fatal signal was delivered to the control process.
See "systemctl status zfs-mount.service" and "journalctl -xe" for details.
invoke-rc.d: initscript zfs-mount, action "start" failed.
● zfs-mount.service - Mount ZFS filesystems
     Loaded: loaded (/lib/systemd/system/zfs-mount.service; enabled; vendor preset: enabled)
     Active: failed (Result: signal) since Thu 2020-06-04 05:49:45 EDT; 4ms ago
       Docs: man:zfs(8)
    Process: 1176564 ExecStart=/sbin/zfs mount -a (code=killed, signal=ABRT)
   Main PID: 1176564 (code=killed, signal=ABRT)

Jun 04 05:49:45 feynmann systemd[1]: Starting Mount ZFS filesystems...
Jun 04 05:49:45 feynmann zfs[1176564]: cannot mount '/root': directory is not empty
Jun 04 05:49:45 feynmann zfs[1176564]: cannot mount '/var/cache': directory is not empty
Jun 04 05:49:45 feynmann zfs[1176564]: cannot mount '/var/lib/AccountsService': directory is not empty
Jun 04 05:49:45 feynmann zfs[1176564]: free(): double free detected in tcache 2
Jun 04 05:49:45 feynmann systemd[1]: zfs-mount.service: Main process exited, code=killed, status=6/ABRT
Jun 04 05:49:45 feynmann systemd[1]: zfs-mount.service: Failed with result 'signal'.
Jun 04 05:49:45 feynmann systemd[1]: Failed to start Mount ZFS filesystems.
dpkg: error processing package zfsutils-linux (--configure):
 installed zfsutils-linux package post-installation script subprocess returned error exit status 1
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          No apport report written because the error message indicates its a followup error from a previous failure.
                                                      dpkg: dependency problems prevent configuration of zfs-initramfs:
 zfs-initramfs depends on zfsutils-linux (>= 0.8.3-1ubuntu12.1); however:
  Package zfsutils-linux is not configured yet.

dpkg: error processing package zfs-initramfs (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of zfs-zed:
 zfs-zed depends on zfsutils-linux (>= 0.8.3-1ubuntu12.1); however:
  Package zfsutils-linux is not configured yet.

dpkg: error processing package zfs-zed (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 zfsutils-linux
 zfs-initramfs
 zfs-zed
E: Sub-process /usr/bin/dpkg returned an error code (1)

Upon closer examination, I see:

Job for zfs-mount.service failed because a fatal signal was delivered to the control process.
    See "systemctl status zfs-mount.service" and "journalctl -xe" for details.

So, I tried:

mjb@feynmann:~$ systemctl status zfs-mount.service
● zfs-mount.service - Mount ZFS filesystems
     Loaded: loaded (/lib/systemd/system/zfs-mount.service; enabled; vendor preset: enabled)
     Active: failed (Result: signal) since Thu 2020-06-04 05:49:45 EDT; 3min 1s ago
       Docs: man:zfs(8)
    Process: 1176564 ExecStart=/sbin/zfs mount -a (code=killed, signal=ABRT)
   Main PID: 1176564 (code=killed, signal=ABRT)

And finally:

mjb@feynmann:~$ sudo /sbin/zfs mount -a
cannot mount '/root': directory is not empty
cannot mount '/var/cache': directory is not empty
cannot mount '/var/lib/AccountsService': directory is not empty
Segmentation fault

At which point, I’m lost.

Still more info:

mjb@feynmann:~$ cat /etc/fstab 
# UNCONFIGURED FSTAB FOR BASE SYSTEM
PARTUUID=84ba431b-1974-486b-b859-41f7417c5b1e /boot/efi vfat nofail,x-systemd.device-timeout=1 0 1
bpool/BOOT/ubuntu /boot zfs nodev,relatime,x-systemd.requires=zfs-import-bpool.service 0 0
rpool/var/log /var/log zfs nodev,relatime 0 0
rpool/var/spool /var/spool zfs nodev,relatime 0 0
rpool/var/tmp /var/tmp zfs nodev,relatime 0 0
/dev/zvol/rpool/swap none swap discard 0 0

Why would my fstab be unconfigured?

Help?


Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.