Điều tra GPT Connector — Focus: Tại sao "đang chạy tốt → tự nhiên hỏng"?

Date: 2026-05-13 Priority: P0 Mode: Investigation NGAY LÚC GPT ĐANG LỖI Timing: CHẠY NGAY — không chờ — vì cần capture trạng thái VPS lúc lỗi

---

Câu hỏi trung tâm

GPT connector đang chạy bình thường, dùng vài tiếng thì tự nhiên lỗi. Đổi phiên GPT mới vẫn lỗi. Fix xong lại hết, dùng tiếp lại lỗi.

KHÔNG PHẢI schema mismatch (nếu schema lệch thì lỗi từ đầu, không "tự nhiên hỏng"). Phải tìm CÁI GÌ THAY ĐỔI giữa trạng thái "chạy tốt" và "hỏng".

---

Giả thuyết mạnh nhất: Tài nguyên VPS cạn kiệt dần

VPS $8/tháng, 6 containers, RAM hạn chế. Sau vài giờ sử dụng:

• Workers tích lũy memory → OOM hoặc swap chậm
• Worker bị kẹt request chậm (Qdrant, OpenAI) → hết worker → mọi request queue
• Connection pool exhausted → nginx upstream timeout

---

Phase 1 — NGAY BÂY GIỜ: Snapshot tài nguyên

# RAM tổng thể
free -h

# Memory + CPU per container
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}\t{{.NetIO}}\t{{.PIDs}}"

# Swap usage
swapon --show
cat /proc/swaps

# Uptime và load average
uptime

# Disk
df -h /

# OOM killer history
dmesg | grep -i "oom\|killed\|out of memory" | tail -20

# Agent Data process count (expect: 1 master + 2 workers = 3 uvicorn)
docker exec incomex-agent-data ps aux | grep uvicorn

# Agent Data worker memory per process
docker exec incomex-agent-data ps aux --sort=-%mem | head -10

---

Phase 2 — NGAY BÂY GIỜ: Nginx upstream status

# Nginx error log — upstream errors = workers overloaded
docker exec incomex-nginx cat /var/log/nginx/error.log | grep -i "upstream\|timeout\|connect.*refused\|no live" | tail -30

# Nginx access log — recent GPT requests
docker exec incomex-nginx cat /var/log/nginx/access.log | tail -50

# Count 502/504/499 in last 200 lines
docker exec incomex-nginx cat /var/log/nginx/access.log | tail -200 | awk '{print $9}' | sort | uniq -c | sort -rn

---

Phase 3 — NGAY BÂY GIỜ: Agent Data health internals

# Agent Data own logs — errors, exceptions, timeouts
docker logs incomex-agent-data --since 2h 2>&1 | grep -i "error\|exception\|timeout\|traceback\|memory\|killed\|worker" | tail -30

# Agent Data request timing — slow requests?
docker logs incomex-agent-data --since 2h 2>&1 | tail -50

# Qdrant status
docker logs incomex-qdrant --since 2h 2>&1 | grep -i "error\|compaction\|optimization\|oom\|slow" | tail -20

# PostgreSQL status
docker logs postgres --since 2h 2>&1 | grep -i "error\|fatal\|timeout\|connection\|lock" | tail -20

---

Phase 4 — NGAY BÂY GIỜ: Live probe so sánh

# Từ VPS gọi health endpoint — bao lâu?
time curl -s https://vps.incomexsaigoncorp.vn/api/health

# Gọi search — bao lâu?
API_KEY=$(grep AGENT_DATA_API_KEY /opt/incomex/.env | cut -d= -f2)
time curl -s -X POST https://vps.incomexsaigoncorp.vn/api/chat \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $API_KEY" \
  -d '{"message":"test"}'

# Gọi liên tiếp 5 lần — có request nào chậm bất thường?
for i in 1 2 3 4 5; do
  echo "--- Request $i ---"
  time curl -s -o /dev/null -w "HTTP %{http_code} Time: %{time_total}s\n" \
    https://vps.incomexsaigoncorp.vn/api/health
done

# Gọi MCP (đường Claude đi) — so sánh
time curl -s -X POST https://vps.incomexsaigoncorp.vn/api/mcp \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $API_KEY" \
  -d '{"jsonrpc":"2.0","method":"tools/list","id":1}'

---

Phase 5 — Kiểm tra connection pool / concurrent requests

# Số connections hiện tại đến nginx
docker exec incomex-nginx sh -c "netstat -an 2>/dev/null | grep :443 | grep ESTABLISHED | wc -l" 2>/dev/null || \
docker exec incomex-nginx sh -c "ss -s" 2>/dev/null

# Số connections đến agent-data (port 8000)
docker exec incomex-agent-data sh -c "netstat -an 2>/dev/null | grep :8000 | wc -l" 2>/dev/null || \
docker exec incomex-agent-data sh -c "ss -tn state established | grep :8000 | wc -l" 2>/dev/null

# Nginx worker connections limit
docker exec incomex-nginx cat /etc/nginx/nginx.conf | grep -i "worker_connections\|worker_processes"

---

Phase 6 — Timeline correlation

# Tìm event bất thường trong 6 tiếng gần nhất trên agent-data
docker logs incomex-agent-data --since 6h 2>&1 | grep -E "^\[|ERROR|WARN|CRITICAL|Traceback" | head -50

# Container restart events
docker events --since 6h --filter 'type=container' --filter 'event=die' --filter 'event=oom' --filter 'event=kill' --filter 'event=restart' 2>/dev/null &
EVENTS_PID=$!
sleep 3
kill $EVENTS_PID 2>/dev/null

---

Report

Upload to:

knowledge/current-state/reports/gpt-connector-live-state-investigation-2026-05-13.md

Include:

1. Exact memory/CPU numbers at time of investigation
2. Uvicorn worker count and memory per worker
3. Nginx upstream error count
4. Qdrant/PG status
5. Response times for live probes
6. Any OOM/restart/timeout events found
7. Conclusion: root cause of "works then breaks" pattern

Final fields

investigation_during_failure_window=true|false
total_ram_mb=<value>
agent_data_mem_mb=<value>
agent_data_mem_percent=<value>
uvicorn_workers_alive=<count>
nginx_upstream_errors_2h=<count>
oom_events_found=<count>
worker_restart_events=<count>
slow_requests_found=<count>
response_time_health_ms=<value>
response_time_search_ms=<value>
root_cause_pattern=<MEMORY_LEAK|WORKER_STUCK|OOM|CONNECTION_POOL|QDRANT_COMPACTION|UNKNOWN>

GPT Connector Live State Investigation | P0 | RUN IMMEDIATELY | 2026-05-13