I have a server configured with multiple interfaces and multiple VLANs. It works perfectly fine for all the local networks, it drops packets forwarded through my router for some reason though. And it’s not even consistent. Sometimes I can get it working for a couple days before it starts dropping packets again. I would love to keep digging but the only things but the only results I can get out of Google are people who need help setting up iptables.
$ cat /etc/issue Ubuntu 18.04.4 LTS n l
$ cat /etc/netplan/50-cloud-init.yaml network: version: 2 ethernets: enp10s0: dhcp4: true dhcp6: true enp6s0: dhcp4: false dhcp6: false vlans: vlan18: id: 18 link: enp6s0 dhcp4: true optional: true vlan150: id: 150 link: enp6s0 dhcp4: true optional: true vlan155: id: 155 link: enp6s0 dhcp4: true optional: true
The interface in question is
enp10s0. I had it on
enp6s0 in the VLANs for a while but moved it to a seperate NIC to isolate variables. That didn’t change anything.
$ netstat -s enp10s0 Ip: Forwarding: 2 4207683 total packets received 11 with invalid addresses 0 forwarded 0 incoming packets discarded 4197424 incoming packets delivered 2183348 requests sent out 21 outgoing packets dropped Tcp: 1634 active connection openings 1615 passive connection openings 150 failed connection attempts 1100 connection resets received 43 connections established 4207863 segments received 2190261 segments sent out 596 segments retransmitted 0 bad segments received 222 resets sent
I add the following first line to my iptables INPUT chain:
-p tcp -m tcp --dport 22 -j LOG --log-prefix "IPTABLES SEEN: "
I watch traffic using tcpdump:
tcpdump -n -e -vv -i enp10s0 port 22
Step 1: Prove it works locally
From my router
10.8.10.1 telnet to the server in question
10.8.10.11 port 22.
Jul 15 23:58:04 meji kernel: IPTABLES SEEN: IN=enp10s0 OUT= MAC=60:a4:4c:60:ce:ce:e0:63:da:21:c1:a5:08:00 SRC=10.8.10.1 DST=10.8.10.11 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=44677 DF PROTO=TCP SPT=48770 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0
23:58:04.335447 e0:63:da:21:c1:a5 > 60:a4:4c:60:ce:ce, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 44677, offset 0, flags [DF], proto TCP (6), length 60) 10.8.10.1.48770 > 10.8.10.11.22: Flags [S], cksum 0xbb2d (correct), seq 978415077, win 14600, options [mss 1460,sackOK,TS val 25150304 ecr 0,nop,wscale 7], length 0
SYN ACK follows like you would expect and everything is fine.
Step 2: compare to the router forwarded connection
nc -vz from my remote server (
22.214.171.124) to connect to the same ip/port.
00:54:44.427670 e0:63:da:21:c1:a5 > 60:a4:4c:60:ce:ce, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 56, id 18829, offset 0, flags [DF], proto TCP (6), length 60) 126.96.36.199.50176 > 10.8.10.11.22: Flags [S], cksum 0x8c20 (correct), seq 1566819019, win 65320, options [mss 1420,sackOK,TS val 1249821436 ecr 0,nop,wscale 6], length 0
nothing. nothing is logged.
No SYN ACK and a retransmit attempt comes through a moment later. The NIC is not reporting errors, iptables sees nothing, and I am left scratching my head. Where can I even look from here? Start digging in the kernel? The network drivers?