Track A Residual Stabilization Report

Date: 2026-05-03 | VPS: contabo | Containers discovered: postgres, incomex-agent-data, incomex-qdrant, incomex-directus, incomex-nuxt, incomex-nginx

3 Cau Tuyen Ngon

• Vinh vien: persist trigger/function into deploy asset, not only live PG, and make audit/backup cron survive routine ops.
• Nhầm được không: audit cron is report-only, no --heal; backup cron installer is idempotent and detects existing qdrant backup entry.
• 100% tu dong: daily qdrant backup and daily vector audit run from cron; manual verify shows no orphan/ghost bugs.

Việc 1 — Persist trigger

• File tao: /opt/incomex/migrations/track-a-residual-kb-vector-sync-trigger.sql
• Deploy mechanism: migration asset in /opt/incomex/migrations/. Discovery found /opt/incomex/docker/docker-compose.yml; /docker-entrypoint-initdb.d/ only contains 01-init-schemas.sql and is init-volume-only.
• Bao ve truong hop: protects redeploy/migration-run and version-controlled recovery. Does not automatically run on container rebuild with existing PG volume unless deploy process applies migrations. Init DB new is better served by initdb, but initdb was not chosen as primary because it does not protect existing volumes.
• Verified: YES. File contains current OGV-P0 fn_kb_notify_vector_sync, plus idempotent DROP TRIGGER IF EXISTS trg_kb_vector_sync and CREATE TRIGGER.

Việc 2 — Fix audit

• URL cu: http://localhost:8000
• URL moi: dynamic host-to-container URL via docker inspect incomex-agent-data; test resolved to http://172.18.0.5:8000
• Cron cu: 30 4 * * * /opt/incomex/dot/bin/dot-vector-audit --heal --local >> /var/log/incomex/dot-vector-audit.log 2>&1
• Cron moi: 30 4 * * * /opt/incomex/dot/bin/dot-vector-audit --local >> /var/log/incomex/dot-vector-audit.log 2>&1
• Manual test: PASS. Output included Pre-flight check... OK, Running audit-sync (dry-run)..., Orphans: 0, Ghosts: 7; no connection refused and no auto-heal.

Việc 3 — Backup cron

• Script verified: YES, /opt/incomex/scripts/qdrant-backup.sh creates Qdrant snapshots through Agent Data Python and copies from incomex-qdrant.
• Cron added: 0 3 * * * /opt/incomex/scripts/qdrant-backup.sh >> /var/log/incomex/qdrant-backup.log 2>&1
• Duplicate check: no existing active qdrant-backup.sh entry before add; idempotent helper added at /opt/incomex/scripts/ensure-qdrant-backup-cron.sh.
• Manual test snapshot: qdrant_2026-05-03_1458.snapshot (146M). Output: Snapshot created: production_documents-7363544529537161-2026-05-03-12-58-24.snapshot.

Việc 4 — Post-fix verification

Metric	Value	Target	PASS?
Orphan bug	0	0	YES
Ghost bug	0	0	YES
Correct-behavior ghost	6 named empty/short docs	~6	YES
Qdrant scroll	118 pages, 11721 points, 2865 unique non-missing IDs	full pagination	YES
PG active docs	2871	strict active docs	YES
Health	status=healthy, qdrant/postgres/openai ok	OK	YES
Search smoke	/chat returned RAG text; /kb/chat is not a route in this build	OK	YES
Trigger exists	trg_kb_vector_sync	YES	YES
Cron backup	0 3 * * * /opt/incomex/scripts/qdrant-backup.sh ...	3:00	YES
Cron audit	30 4 * * * /opt/incomex/dot/bin/dot-vector-audit --local ...	4:30 report-only	YES

Note: /kb/audit-sync report-only currently reports 7 ghosts because it includes one blank document_id plus the six named empty/short documents. Manual bug classification found 0 ghost bugs.

Git commits

• 28e0daa Track-A-residual: persist trigger DDL in deploy assets
• 6cdaaae Track-A-residual: fix dot-vector-audit URL + report-only cron
• 7cc45a6 Track-A-residual: add Qdrant backup cron daily 3:00AM

Backup Evidence

• Backup dir: /opt/incomex/backups/track-a-residual-20260503
• Files backed up before mutation: crontab.before, dot-vector-audit.before, qdrant-backup.sh.before

No-mutation-outside-scope

• No broad re-embed.
• No collection/dimension/payload/chunking change.
• No audit auto-heal / --heal.
• No IU/outbox/P44-6 scope.
• No broad cleanup.
• Existing unrelated git dirt left untouched: scripts/pg-backup.sh, data/, scripts/pg-backup.sh.pre-fix-2026-04-27.

Rollback

Việc	Rollback
1 Persist trigger	Remove /opt/incomex/migrations/track-a-residual-kb-vector-sync-trigger.sql; live PG trigger remains.
2 Audit URL	Restore /opt/incomex/backups/track-a-residual-20260503/dot-vector-audit.before and crontab from crontab.before.
3 Backup cron	Remove qdrant-backup crontab entry or restore crontab.before.