I’ve got a raspi (in my robot project) connected to the internet over a 3G connection using a dongle with a simcard. I then use ngrok to expose ssh to a static address so that I can always ssh into it using:
ssh -p 29xxx email@example.com
That has always worked great for me. Today I wanted to do another test with my robot, but I can’t ssh into the machine through ngrok anymore. It just gives me
kex_exchange_identification: Connection closed by remote host
So this is what I did to debug it:
- I can ssh into it over the local network fine, so I guess sshd on the pi itself is not the problem.
- I checked the ngrok website to see if the pi connected to the ngrok network. It lists the connection saying it was established a couple minutes ago.
- I ssh’ed into the pi over the local network, stopped the tunnel and then manually started it to see the logs. It shows me this:
pi@myrobot:~$ ngrok tcp -remote-addr=1.tcp.ngrok.io:29xxx --log=stdout 22 INFO[09-11|09:39:33] no configuration paths supplied INFO[09-11|09:39:33] using configuration at default config path path=/home/pi/.ngrok2/ngrok.yml INFO[09-11|09:39:33] open config file path=/home/pi/.ngrok2/ngrok.yml err=nil t=2020-09-11T09:39:33+0000 lvl=info msg="starting web service" obj=web addr=127.0.0.1:4040 t=2020-09-11T09:39:34+0000 lvl=info msg="tunnel session started" obj=tunnels.session t=2020-09-11T09:39:34+0000 lvl=info msg="client session established" obj=csess id=1b6463ec0724 t=2020-09-11T09:39:34+0000 lvl=info msg="started tunnel" obj=tunnels name=command_line addr=//localhost:22 url=tcp://1.tcp.ngrok.io:29xxx t=2020-09-11T09:39:38+0000 lvl=warn msg="failed to check for update" obj=updater err="Post https://update.equinox.io/check: context deadline exceeded"
- I then tried ssh’ing into the pi over the internet again using verbose output (
ssh -v -p 29xxx firstname.lastname@example.org), which gives me the following output:
$ ssh -v -p 29xxx email@example.com OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f 31 Mar 2020 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files debug1: /etc/ssh/ssh_config line 21: Applying options for * debug1: Connecting to 1.tcp.ngrok.io [3.13.191.xxx] port 29xxx. debug1: Connection established. debug1: identity file /home/kramer65/.ssh/id_rsa type 0 debug1: identity file /home/kramer65/.ssh/id_rsa-cert type -1 debug1: identity file /home/kramer65/.ssh/id_dsa type -1 debug1: identity file /home/kramer65/.ssh/id_dsa-cert type -1 debug1: identity file /home/kramer65/.ssh/id_ecdsa type -1 debug1: identity file /home/kramer65/.ssh/id_ecdsa-cert type -1 debug1: identity file /home/kramer65/.ssh/id_ecdsa_sk type -1 debug1: identity file /home/kramer65/.ssh/id_ecdsa_sk-cert type -1 debug1: identity file /home/kramer65/.ssh/id_ed25519 type -1 debug1: identity file /home/kramer65/.ssh/id_ed25519-cert type -1 debug1: identity file /home/kramer65/.ssh/id_ed25519_sk type -1 debug1: identity file /home/kramer65/.ssh/id_ed25519_sk-cert type -1 debug1: identity file /home/kramer65/.ssh/id_xmss type -1 debug1: identity file /home/kramer65/.ssh/id_xmss-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 kex_exchange_identification: Connection closed by remote host
In the terminal on the pi I see nothing happening. But the output above also suggests that it never actually reaches the pi.
Could it be that ngrok doesn’t pass on the connection properly? Is the problem in the pi? or on my laptop locally? All tips are welcome!
After some more debugging I found the problem has to be with the mobile connection. When I remove the 3G-dongle and connect the pi to the internet over wifi I can perfectly ssh into it using the ngrok address. But when I connect over 3G I cannot. I checked whether the internet over 3G works by ssh’ing into the pi over the wifi network and using
curl ip.me to check whether the public ip changes when I connect over 3G (plus, a ping to
22.214.171.124 increases from 10ms to about 40ms).
I also checked the syslog and that doesn’t say the anything about the incoming message (I would expect a
Started Session c7 of user pi). Furthermore, when I start the ngrok tunnel over 3G it adds the line below to the output. The rest (including
"client session established") is the same though
lvl=warn msg="failed to check for update" obj=updater err="Post https://update.equinox.io/check: context deadline exceeded"
So why would the tunnel fail over 3G? Could it be that my telco closes all kinds of ports or blocks traffic? Any way of debugging this further?