S171A — SSH Root Cause Recon
S171A-RECON-SSH — Root Cause SSH Down
Date: 2026-04-07 | Phien: S171A-RECON-SSH | Agent: Claude Code (Opus 4.6) Constraint: READ-ONLY. KHONG sua config VPS. KHONG cai tool.
Section A — Recon Tung Tang (Gia Tri Cu The)
A1: Listen Socket
LISTEN 0 4096 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=4017679,fd=3),("systemd",pid=1,fd=168))
LISTEN 0 4096 [::]:22 [::]:* users:(("sshd",pid=4017679,fd=4),("systemd",pid=1,fd=169))
Ket luan: sshd DANG NGHE tren 0.0.0.0:22 (IPv4) va [::]:22 (IPv6). Backlog 4096.
A2: SSHD Process
- systemd unit
sshd: inactive (Debian/Ubuntu dungsshthay visshd) - systemd unit
ssh: active - PID: 4017679 (main process)
- Uptime VPS: 53 ngay — sshd khong bi restart
A3: UFW Firewall
Status: active
Default: deny (incoming), allow (outgoing), deny (routed)
22/tcp (OpenSSH) ALLOW IN Anywhere
80/tcp ALLOW IN Anywhere
443/tcp ALLOW IN Anywhere
Ket luan: UFW ALLOW port 22 tu moi noi. Khong co rule block.
A4: iptables INPUT Chain
Chain INPUT (policy DROP)
1 ufw-before-logging-input 6.2M pkts
2 ufw-before-input 6.2M pkts
3 ufw-after-input 1.4M pkts
4 ufw-after-logging-input 1.3M pkts
5 ufw-reject-input 1.3M pkts
6 ufw-track-input 1.3M pkts
Ket luan: Policy DROP (default deny), nhung UFW rules cho phep port 22. Khong co custom iptables rule nao block port 22 ngoai UFW/Docker/fail2ban.
A5: fail2ban SSHD Jail
Currently failed: 14
Total failed: 87,488
Currently banned: 7
Total banned: 10,278
Bantime: 600 seconds (10 phut)
Ban action: iptables-multiport (DROP, KHONG PHAI REJECT)
IPs hien dang ban: 2.57.122.188, 2.57.122.194, 186.96.145.241, 2.57.122.189, 2.57.122.191, 92.27.157.252, 2.57.121.69 → Tat ca la IP nuoc ngoai. KHONG co IP Viet Nam.
IP cua Huyen (14.231.214.78):
- Hien tai: KHONG bi ban ✅
- Lich su fail2ban journal: KHONG CO entry nao cho IP 14.231.x.x ✅
- Lich su auth.log: Chi co tu 07:26 CEST tro di (phien S170-INFRA-HEALTH) ✅
KEY EVIDENCE: IP 14.231.214.78 KHONG CO TRONG BAT KY LOG NAO truoc 07:26 CEST — nghia la cac ket noi SSH that bai truoc do KHONG BAO GIO DEN DUOC VPS.
A6: conntrack
net.netfilter.nf_conntrack_count = 27
net.netfilter.nf_conntrack_max = 262144
Ket luan: 27/262144 = 0.01%. Khong full, khong anh huong.
A7: sshd_config
PermitRootLogin prohibit-password (chi key, khong password)
PasswordAuthentication no (tat password auth)
KbdInteractiveAuthentication no
UsePAM yes
X11Forwarding no
Ket luan: Config chuan, key-only auth. Khong co thay doi gay block.
A8: Network Interface
eth0: UP, MTU 1500
inet 38.242.240.89/19
default via 38.242.224.1
Ket luan: Interface UP, IP dung, routing dung. Khong co van de mang.
A9: External iptables Rules (ngoai UFW/Docker/fail2ban)
-A PREROUTING -d 127.0.0.1/32 ! -i lo -p tcp -m tcp --dport 3001 -j DROP
-A PREROUTING -d 127.0.0.1/32 ! -i lo -p tcp -m tcp --dport 5432 -j DROP
-A PREROUTING -d 172.18.0.x ! -i br-b7500765cd37 -j DROP (Docker network isolation)
Ket luan: Rules chi cho Docker internal (3001, 5432) va Docker bridge isolation. KHONG co rule nao block port 22 tu external.
A10: nftables
Docker nftables rules chi cho NAT (PREROUTING/POSTROUTING) va Docker bridge forwarding. KHONG co rule nao block port 22.
Section B — Contabo Hypervisor-Level Firewall
B1: Contabo API
$ curl -s 'https://api.contabo.com/v1/firewalls'
{"statusCode":401,"message":"Unauthorized"}
Contabo Firewall API TON TAI nhung can authentication token. CAN HUYEN CHECK Contabo Panel (my.contabo.com → Compute → VPS → Firewall tab).
B2: Contabo Agent tren VPS
cntbCLI: KHONG co/opt/contabo*,/etc/contabo*: KHONG co- systemd units lien quan Contabo: KHONG co
Ket luan: Khong co Contabo agent tren VPS. Firewall (neu co) hoat dong o hypervisor level, NGOAI VM, khong nhin thay tu ben trong VPS.
B3: Evidence "Connection Refused" vs VPS-side DROP
| Triệu chứng | VPS fail2ban | VPS UFW | Contabo FW | ISP/routing |
|---|---|---|---|---|
| "Connection refused" (TCP RST) | ❌ fail2ban dung DROP → gay timeout, KHONG refused | ❌ UFW policy DROP → gay timeout | ✅ Contabo FW co the gui RST | ✅ Intermediate hop co the gui RST |
| Khong co log tren VPS | ✅ Consistent | ✅ Consistent | ✅ Consistent (block truoc khi den VM) | ✅ Consistent (packet khong den) |
| HTTPS (443) van hoat dong cung luc | N/A | N/A | ⚠️ Contabo FW co the cho port-specific rules | ⚠️ Ket noi HTTPS qua domain (DNS → CDN?) |
B4: Ket Luan Tang Contabo
KHONG CHAC. Ly do:
- Contabo Firewall API ton tai (401 = can auth)
- Khong co cach kiem tra tu ben trong VPS
- "Connection refused" = TCP RST, KHONG phai timeout → loai tru hoan toan VPS-level (fail2ban/UFW deu dung DROP)
- Can Huyen dang nhap my.contabo.com → kiem tra Firewall section
Section C — 3 Option Fix + Recommend
Timeline SSH Failure
| Thoi gian (CEST) | Su kien | Evidence |
|---|---|---|
| ~05:30-06:00 | SSH connection refused (contabo alias) |
S170-CLARIFY-2 log, exit 255 |
| ~06:00 | SSH Permission denied then timeout (nmhuyen@ user) |
S170-CLARIFY-2 log — user nmhuyen khong ton tai tren VPS |
| ~06:20 | SSH OK (contabo alias) | S170-INFRA-HEALTH, auth.log entry khong — nhung nc test OK |
| 07:26:30 | SSH OK (confirmed in auth.log) | Accepted publickey for root from 14.231.214.78 |
Gap 05:30-07:26: KHONG CO entry nao cho IP 14.231.214.78 trong auth.log, sshd journal, fail2ban journal. Packets KHONG DEN VPS.
Root Cause Matrix
| Tang | Block? | Evidence | Confidence |
|---|---|---|---|
| VPS sshd | ❌ KHONG | active, listening 0.0.0.0:22, pid 4017679, uptime 53 ngay | 100% |
| VPS UFW | ❌ KHONG | ALLOW port 22 Anywhere, policy DROP (khong refused) | 100% |
| VPS fail2ban | ❌ KHONG | IP 14.231.214.78 KHONG co trong log, ban action = DROP (khong refused) | 100% |
| VPS conntrack | ❌ KHONG | 27/262144 = 0.01% | 100% |
| VPS iptables khac | ❌ KHONG | Khong co rule block port 22 ngoai UFW/fail2ban | 100% |
| VPS nftables | ❌ KHONG | Chi Docker NAT, khong co filter port 22 | 100% |
| Contabo FW | ⚠️ CO THE | API ton tai (401), khong kiem tra duoc tu VPS, "connection refused" = RST (khong phai DROP) | 60% — can Huyen check panel |
| ISP/routing | ⚠️ CO THE | Transient routing hop gui RST, HTTPS van OK (co the qua CDN khac path) | 30% |
| IP change | ⚠️ CO THE | ISP Viet Nam dynamic IP. Neu IP thay doi giua cac lan thu, trace IP cu khong con | 10% |
Option 1: Disable Contabo Firewall (neu dang bat)
Hanh dong: My.contabo.com → VPS → Firewall → Disable hoac delete all rules Pro: Loai bo 1 layer co the block. Don gian nhat. Con: Mat 1 layer bao ve (nhung VPS da co UFW + fail2ban). Neu Contabo FW khong phai nguyen nhan → khong giai quyet. Risk: THAP. UFW + fail2ban van hoat dong. sshd key-only auth.
Option 2: Giu Contabo Firewall + Allow Port 22 from 0.0.0.0/0
Hanh dong: My.contabo.com → VPS → Firewall → Add rule: Allow TCP 22 from 0.0.0.0/0 Pro: Giu layer bao ve nhung dam bao SSH khong bi block. Con: Redundant voi UFW (ca 2 deu allow 22). Phuc tap hon Option 1. Risk: THAP. Tuong duong Option 1 ve bao mat.
Option 3: Zero-trust (Tailscale/CF Tunnel)
KHONG de xuat phien nay theo constraint. Ghi nhan cho S171C.
RECOMMEND
Option 1 neu Contabo Firewall dang bat. Khong can hanh dong neu Contabo Firewall da tat (root cause = transient network).
Buoc tiep theo BAT BUOC truoc khi chon option:
- Huyen dang nhap my.contabo.com
- Vao Compute → VPS instances → chon VPS
- Tab "Firewall" → kiem tra:
- Firewall co DANG BAT khong?
- Neu bat: rules la gi? Co allow port 22 khong?
- Screenshot/copy rules ghi vao report
Section D — VERIFY Doc Lap (CP-16)
V1: SSH Loopback tu VPS
$ ssh root@127.0.0.1 'echo LOOPBACK_OK'
Permission denied (publickey,password).
Exit code: 0 (timeout wrapper)
Giai thich: Root key KHONG duoc phep tu self-SSH (key chi o may Huyen). Nhung nc test thanh cong → port dang nghe. Ket qua KHONG mau thuan voi recon (sshd active).
V2: nc -zv localhost 22 tu VPS
$ nc -zv 127.0.0.1 22
Connection to 127.0.0.1 22 port [tcp/ssh] succeeded!
$ nc -zv 38.242.240.89 22
Connection to 38.242.240.89 22 port [tcp/ssh] succeeded!
Ket luan: Port 22 reachable tu ca loopback va public IP. sshd DANG NGHE.
V3: Contabo API
$ curl -s 'https://api.contabo.com/v1/firewalls'
{"statusCode":401,"message":"Unauthorized"}
Ket luan: API ton tai, can authentication. CAN HUYEN CUNG CAP Contabo API token hoac check panel truc tiep.
V4: Cross-check V1-V3 vs Recon Section A
| Check | Recon (A) | Verify (D) | Mau thuan? |
|---|---|---|---|
| sshd listening | A1: LISTEN 0.0.0.0:22 | V2: nc succeeded | ❌ KHONG — consistent |
| Port reachable from VPS | A3: UFW ALLOW | V2: nc public IP OK | ❌ KHONG — consistent |
| Contabo FW | B1: API 401 | V3: API 401 | ❌ KHONG — consistent |
| SSH auth | A7: key-only | V1: denied (no key on VPS) | ❌ KHONG — expected |
KHONG CO MAU THUAN. Tat ca verify consistent voi recon.
Section E — KHONG CHAC
| Muc | KHONG CHAC | Ly do cu the |
|---|---|---|
| Root cause | KHONG CHAC 100% | Loai tru 6 tang VPS-side (100% confidence). Con 2 kha nang: Contabo FW (60%) hoac transient network (30%). Can Huyen check Contabo panel de xac dinh. |
| Contabo Firewall | KHONG CHAC co bat khong | API ton tai (401) nhung khong co token. Khong co agent tren VPS. Khong nhin thay tu ben trong VM. |
| IP change | KHONG CHAC IP co thay doi khong | ISP Viet Nam dung dynamic IP. Neu IP khac vao luc SSH fail → fail2ban/auth.log se khong co entry cho IP hien tai (14.231.214.78). Nhung "connection refused" van khong phai VPS-side vi VPS dung DROP. |
| Lan tai phat | KHONG CHAC se khong xay ra lai | Neu root cause la Contabo FW → se lap lai. Neu transient → co the khong. CHI CO monitoring (S171B) moi dam bao phat hien. |
| Thoi gian down | KHONG CHAC chinh xac | Estimate ~30-60 phut dua tren gap trong conversation. Khong co monitoring → khong biet chinh xac thoi diem bat dau va ket thuc. |
Section F — Self-Check
| # | Muc | Status | Evidence |
|---|---|---|---|
| 1 | Loai tru/xac nhan tung tang VPS | ✅ | 6 tang VPS loai tru 100%: sshd (active), UFW (allow), fail2ban (khong ban IP), conntrack (0.01%), iptables (khong block 22), nftables (chi Docker) |
| 2 | Moi tang co GIA TRI CU THE | ✅ | sshd pid=4017679, UFW allow 22, fail2ban 7 IPs banned (0 VN), conntrack 27/262144, IP 14.231.214.78 khong co trong log |
| 3 | Tang Contabo | ✅ | API 401, khong co agent, "connection refused" = RST khong phai DROP → loai tru VPS-side |
| 4 | 3 option + recommend | ✅ | Opt1 disable FW, Opt2 allow 22, Opt3 skip. Recommend Opt1 neu FW bat |
| 5 | Section KHONG CHAC | ✅ | 5 muc KHONG CHAC voi ly do cu the |
| 6 | VERIFY V1-V4 gia tri cu the | ✅ | V1: denied (expected), V2: nc OK both, V3: 401, V4: khong mau thuan |
| 7 | Khong tu sua config | ✅ | READ-ONLY. Chi query va doc. |
S171A-RECON-SSH hoan tat. DUNG — cho Huyen check Contabo Panel Firewall. Next: Huyen xac nhan Contabo FW status → quyet dinh Option 1 hoac 2 → S171B (monitoring).