Bounty: 100
Problem:
I have a git repository mounted via sshfs and cannot commit changes with the following error message:
fatal: cannot update the ref 'HEAD': unable to append to '.git/logs/HEAD': Permission denied
Note that I can
cp -a .git/logs/HEAD .git/logs/HEAD.bu
printf foo > .git/logs/HEAD
mv .git/logs/HEAD.bu .git/logs/HEAD
without a problem, but
printf foo >> .git/logs/HEAD
gives me the ‘Permission denied’ as well.
Question:
What do I need to change about my configuration to be able to commit from my local machine to the remote repository?
What I tried:
Given the above symptoms, I assume the issue lies with appending to a file.
I found Git repository on SSHFS: unable to append to '.git/logs/HEAD': Invalid argument which refers to https://github.com/libfuse/sshfs/issues/82 suggesting the issue (note the slighty differerent error message) could be solved by mounting the remote file system with writeback_cache=no
. The latter source quotes the man
page referencing the following caveat/workaround:
CAVEATS / WORKAROUNDS
[...]
O_APPEND
When writeback caching is enabled, SSHFS cannot reliably support the O_APPEND open
flag and thus signals an error on open. To enable support for unreliable O_APPEND
(which may overwrite data if the file changes on the server at a bad time), mount
the file system with -o unreliable_append.
However, this section is not in my man page:
sshfs -V
SSHFS version 3.7.0
FUSE library version 3.9.1
using FUSE kernel interface version 7.31
fusermount3 version: 3.9.1
I found that the writeback-cache feature that I tried to disable, was actually removed (after being disabled and re-enabled more than once before).
So I guess I should be good but clearly there (still) is a problem.
A further complication that I should probably mention, is that my user name and ID on the remote system do not match the local one, so I need to used the idmap
feature.
Here is the corresponding fstab
entry:
<remote-user>@<remote-machine>: /mnt/ssh/<remote-machine> sshfs _netdev,user,idmap=user,allow_other 0 0
Also, my /etc/fuse.conf
contains
user_allow_other
Background:
To avoid answer just telling me not to do this:
- I know how git works.
- I know I can clone the repository locally, commit there, and push to the remote one over ssh.
Why I don’t do it? – Because I track code that can only be tested on the remote machine and I do want to test it before committing.
So to some extent this is ‘just’ a convenience issue to avoid having to:
- Edit code on the local copy.
- Commit the changes to the local copy.
- Push to the remote copy.
- SSH to the remote machine (or switch terminals).
- Test the code on the remote machine.
- Checkout another branch (to allow force-pushing).
- End the SSH sessions (or switch [back] terminals).
- Edit the code.
- Amend the previous commit on the local copy.
- Force-push to the remote copy.
- SSH to the remote machine (or switch terminals).
- Checkout the force-pushed branch.
- Repeat steps 5 – 11 (seven steps!) until I’m happy.
Instead, I want to:
- SSH to the remote machine (or switch terminals).
- Edit code on the remote copy from the remote machine.
- Test the code on the remote machine.
- Edit code on the remote copy from the remote machine.
- Repeat steps 3 – 4 (two steps!) until I’m happy.
- End the SSH sessions (or switch [back] terminals).
- Commit the changes to the remote copy from the local machine.
Why don’t I simply commit from the remote machine? – Because I want to sign my commits but can’t entrust the remote machine with the private key.
So the best alternative I could come up with is:
- SSH to the remote machine (or switch terminals).
- Edit code on the remote copy from the remote machine.
- Test the code on the remote machine.
- Edit code on the remote copy from the remote machine.
- Repeat steps 3 – 4 (two steps!) until I’m happy.
- Commit the changes to the remote copy from the remote machine.
- Checkout another branch (to allow force-pushing).
- End the SSH sessions (or switch [back] terminals).
- Pull from the remote copy.
- Amend (sign) the previous commit on the local copy.
- Force-push to the remote copy.
- SSH to the remote machine (or switch terminals).
- Checkout the force-pushed branch.
So on the one hand, I’d like to get rid of these extra steps (things get more complicated when adding feature branches as those need to be properly checked out on both copies and configured for proper tracking), on the other I want to understand why it doesn’t ‘just work'(tm).
Update:
Following up on a comment by @tukan, I reproduced the error with debug output:
- Mount remote with debug output:
mount -o sshfs_debug MOUNTPOINT
SSHFS version 3.7.0
executing <ssh> <-x> <-a> <-oClearAllForwardings=yes> <-2> <USER@SERVER> <-s> <sftp>
USER@SERVER's password:
Server version: 3
Extension: versions <2,3,4,5,6>
Extension: fsync@openssh.com <1>
Extension: posix-rename@openssh.com <1>
Extension: statvfs@openssh.com <2>
Extension: fstatvfs@openssh.com <2>
Extension: hardlink@openssh.com <1>
remote_uid = 0
- In a different terminal, access the mounted share:
cd MOUNTPOINT/DIR_WITH_WRITE_PERMISSIONS
[00002] LSTAT
[00002] ATTRS 45bytes (188ms)
- Verify regular writing works:
echo foo > foobar
[00003] LSTAT
[00003] STATUS 38bytes (46ms)
[00004] LSTAT
[00004] STATUS 38bytes (32ms)
[00005] LSTAT
[00005] ATTRS 45bytes (242ms)
[00006] OPENDIR
[00006] HANDLE 29bytes (31ms)
[00007] READDIR
[00008] READDIR
[00007] NAME 668bytes (58ms)
[00009] READDIR
[00010] READDIR
[00008] NAME 483bytes (65ms)
[00011] READDIR
[00012] READDIR
[00009] STATUS 37bytes (27ms)
[00010] STATUS 37bytes (27ms)
[00013] CLOSE
[00014] LSTAT
[00011] STATUS 37bytes (27ms)
[00012] STATUS 37bytes (27ms)
[00013] STATUS 28bytes (26ms)
[00014] STATUS 38bytes (31ms)
[00015] OPEN
[00016] LSTAT
[00015] HANDLE 29bytes (153ms)
[00016] ATTRS 45bytes (158ms)
[00017] FSTAT
[00017] ATTRS 45bytes (29ms)
[00018] WRITE
[00018] STATUS 28bytes (28ms)
[00019] CLOSE
[00019] STATUS 28bytes (28ms)
- Trigger error by attempting to append:
echo bar >> foobar
[00020] LSTAT
[00020] STATUS 38bytes (74ms)
[00021] LSTAT
[00021] STATUS 38bytes (57ms)
[00022] LSTAT
[00022] ATTRS 45bytes (52ms)
[00023] OPENDIR
[00023] HANDLE 29bytes (53ms)
[00024] READDIR
[00025] READDIR
[00024] NAME 668bytes (68ms)
[00026] READDIR
[00027] READDIR
[00025] NAME 597bytes (77ms)
[00028] READDIR
[00029] READDIR
[00026] STATUS 37bytes (47ms)
[00030] CLOSE
[00027] STATUS 37bytes (47ms)
[00031] OPEN
[00032] LSTAT
[00028] STATUS 37bytes (47ms)
[00029] STATUS 37bytes (47ms)
[00030] STATUS 28bytes (26ms)
[00031] STATUS 43bytes (28ms)
[00032] ATTRS 45bytes (29ms)
zsh: permission denied: foobar
Hope this helps to find the root cause of my problem.
Get this bounty!!!