#StackBounty: #linux #ssh #raspberry-pi #tunnel #ngrok ngrok kex_exchange_identification: read: Connection reset by peer

Bounty: 250

I’ve got a raspi (in my robot project) connected to the internet over a 3G connection using a dongle with a simcard. I then use ngrok to expose ssh to a static address so that I can always ssh into it using:

ssh -p 29xxx pi@1.tcp.ngrok.io

That has always worked great for me. Today I wanted to do another test with my robot, but I can’t ssh into the machine through ngrok anymore. It just gives me kex_exchange_identification: Connection closed by remote host

So this is what I did to debug it:

  1. I can ssh into it over the local network fine, so I guess sshd on the pi itself is not the problem.
  2. I checked the ngrok website to see if the pi connected to the ngrok network. It lists the connection saying it was established a couple minutes ago.
  3. I ssh’ed into the pi over the local network, stopped the tunnel and then manually started it to see the logs. It shows me this:
pi@myrobot:~$ ngrok tcp -remote-addr=1.tcp.ngrok.io:29xxx --log=stdout 22
INFO[09-11|09:39:33] no configuration paths supplied
INFO[09-11|09:39:33] using configuration at default config path path=/home/pi/.ngrok2/ngrok.yml                                                                             
INFO[09-11|09:39:33] open config file                         path=/home/pi/.ngrok2/ngrok.yml err=nil                                                                       
t=2020-09-11T09:39:33+0000 lvl=info msg="starting web service" obj=web addr=127.0.0.1:4040                                                                                  
t=2020-09-11T09:39:34+0000 lvl=info msg="tunnel session started" obj=tunnels.session
t=2020-09-11T09:39:34+0000 lvl=info msg="client session established" obj=csess id=1b6463ec0724                                                                              
t=2020-09-11T09:39:34+0000 lvl=info msg="started tunnel" obj=tunnels name=command_line addr=//localhost:22 url=tcp://1.tcp.ngrok.io:29xxx                                   
t=2020-09-11T09:39:38+0000 lvl=warn msg="failed to check for update" obj=updater err="Post https://update.equinox.io/check: context deadline exceeded"                      
  1. I then tried ssh’ing into the pi over the internet again using verbose output (ssh -v -p 29xxx pi@1.tcp.ngrok.io), which gives me the following output:
$ ssh -v -p 29xxx pi@1.tcp.ngrok.io
OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 1.tcp.ngrok.io [3.13.191.xxx] port 29xxx.
debug1: Connection established.
debug1: identity file /home/kramer65/.ssh/id_rsa type 0
debug1: identity file /home/kramer65/.ssh/id_rsa-cert type -1
debug1: identity file /home/kramer65/.ssh/id_dsa type -1
debug1: identity file /home/kramer65/.ssh/id_dsa-cert type -1
debug1: identity file /home/kramer65/.ssh/id_ecdsa type -1
debug1: identity file /home/kramer65/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/kramer65/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/kramer65/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/kramer65/.ssh/id_ed25519 type -1
debug1: identity file /home/kramer65/.ssh/id_ed25519-cert type -1
debug1: identity file /home/kramer65/.ssh/id_ed25519_sk type -1
debug1: identity file /home/kramer65/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/kramer65/.ssh/id_xmss type -1
debug1: identity file /home/kramer65/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
kex_exchange_identification: Connection closed by remote host

In the terminal on the pi I see nothing happening. But the output above also suggests that it never actually reaches the pi.

Could it be that ngrok doesn’t pass on the connection properly? Is the problem in the pi? or on my laptop locally? All tips are welcome!

[EDIT]

After some more debugging I found the problem has to be with the mobile connection. When I remove the 3G-dongle and connect the pi to the internet over wifi I can perfectly ssh into it using the ngrok address. But when I connect over 3G I cannot. I checked whether the internet over 3G works by ssh’ing into the pi over the wifi network and using curl ip.me to check whether the public ip changes when I connect over 3G (plus, a ping to 8.8.8.8 increases from 10ms to about 40ms).

I also checked the syslog and that doesn’t say the anything about the incoming message (I would expect a Started Session c7 of user pi). Furthermore, when I start the ngrok tunnel over 3G it adds the line below to the output. The rest (including "client session established") is the same though

lvl=warn msg="failed to check for update" obj=updater err="Post https://update.equinox.io/check: context deadline exceeded"

So why would the tunnel fail over 3G? Could it be that my telco closes all kinds of ports or blocks traffic? Any way of debugging this further?


Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.