Checkpoint — RP Production API Operator Fix / UI Truth / Smoke — 2026-06-06
CHECKPOINT — RP_PRODUCTION_API_OPERATOR_FIX — 2026-06-06 — STATUS PARTIAL — EXECUTION (delegated) — LIVE MUTATION DB YES (10 additive birth-neutral views) / PROD NO.
RESULT: the only RED control-plane lane (RP production API/UI) is fully root-caused with live evidence, all fixes engineered + proven, real smoke harness exists, UI cannot false-green, router updated. Remaining = operator restart/rebuild of incomex-nuxt (all-users action, not self-authorized) -> PARTIAL.
LIVE BASELINE (vps.incomexsaigoncorp.vn ~04:47 UTC): page/=200; 12 HEALTHY / 4 PARAM(400) / 2 500 / 1 404 (+ refresh-counts GET404=POST-only). Matches prior P0 rollup.
3 BREAKS:
- pivot-query 500 — jsonb group_spec
_neq-> Directus 400. FIX drop jsonb filter + filter cross in JS. PROVEN LIVE via in-container node: 200, active=37 cross=12. Patch patches/pivot-query.get.mjs (node --check ok). Existing route. - matrix 500 — /items/entity_labels?limit=-1 = 897,347 rows over Directus REST in 512MB container, 15s $fetch timeout -> 500 @28s. FIX rpQuery over new view v_rp_entity_label_facet_counts (226,052 agg rows). Patch patches/matrix.get.mjs (node --check ok). Existing route.
- registries/index 404 — client/server drift: /knowledge/registries page calls it but route not mounted in s174. FIX add handler (patches/registries-index.get.ts) or repoint page. NEW route -> rebuild required.
PIVOT_MISSING=14 (v_registries_pivot_node_contract count_status='PIVOT_MISSING'): 9 BY_DESIGN (cross/l2_drill w/ results: PIV-101..106,201,202,203), 3 DATA_EMPTY_LEVEL (PIV-204/205/206), 2 RETIRE_PHANTOM (PIV-020,MTX-TEST). Guard PASS_ALL_PIVOT_MISSING_EXPLAINED. None a production break. (Leaf-grain 138 also reported.)
TOPOLOGY: image nuxt-ssr-local:s174, mem_limit 512m; bind-mount /opt/incomex/deploys/nuxt-output -> /app/.output (existing-route hotfix needs NO rebuild; new route DOES); nginx -> nuxt:3000 for /api/registry|registries|registries-pivot + /; compose /opt/incomex/docker/docker-compose.yml; RP_PG_* + rpQuery (export rpQuery as m).
VIEWS (10, one tx RC=0, birth 1,210,816 before==after): v_rp_production_api_truth_status_v2 / _failure_inventory / _ui_no_false_green_guard (BLOCKED real_breaks=3 healthy=12) / _route_mapping_status / v_rp_entity_label_facet_counts / v_rp_pivot_missing_inventory_v2 / _resolution_plan / _no_false_green_guard (PASS 14/14) / v_rp_production_api_smoke_expectation / v_rp_production_api_router_v2 (RED_RP_PRODUCTION_API_ACTION_READY_AWAITING_OPERATOR_REBUILD_RESTART; self-flips GREEN on deploy). Rollback 99_rollback.sql.
SMOKE rp-production-smoke.sh: ran vs current prod -> FAIL 3 breaks exit 1 (detection proven). Expect PASS post-deploy. Operator script rp-apply-hotfix.sh = express bind-mount swap of matrix+pivot-query + restart + smoke + AUTO-ROLLBACK (fixes 2/3; 404 needs rebuild).
OPERATOR PACKET: PATH1 express hotfix (no rebuild, fixes 2 500s); PATH2 durable rebuild (3 source patches, fixes all). DB substrate already live (do not drop while matrix patch deployed). Git not touched (no reachable repo).
SAFETY: no fake PASS, no UI-ready-while-broken, no source edit, no prod mutation, no REAL_RUN, no event activation, no auth mutation, no OOM, no deep composite, rollback staged. All PASS.
GOTCHAS: pg_schema MCP broken->information_schema; write_file blocks .sh (write local+ssh push); KB no fenced code (WAF, prose+tables); DDL via ssh contabo->docker exec -i postgres psql; rpQuery export letter m build-specific; nuxt container no curl (use node); .output host bind-mount editable.
STAGED VPS /opt/incomex/docs/mcp-writes/rp-production-api-fix-2026-06-06/ {01_apply.sql,99_rollback.sql,rp-production-smoke.sh,rp-apply-hotfix.sh,patches/*}. Reports knowledge/dev/reports/architecture/rp-production-api-operator-fix-ui-truth-smoke-2026-06-06/00..12.
BLOCKER: operator restart/rebuild only; zero eng. NEXT: RP_PRODUCTION_DEPLOY_AND_POSTFIX_SMOKE then RP UI clean-vs-raw count truth + species-matrix hardening.