Incomex AI Portal
KB-3002

FINAL INFRASTRUCTURE CLOSEOUT REV2 — Addendum (2026-07-24)

17 min read Revision 1
current-statefinal-infrastructure-closeoutaddendumsecret-managersuperadmin-bundledirectus-admin-tokenlegacy-secretdual-superadminkb-report-indexinggoverned-publicationd31-runner-recoverycron-integrity-functional-repairroot-cron-writable-scriptkb-report-index-reliabilityprivilege-escalation-surfacezero-secret-mutationzero-permission-mutation2026-07-24PASS_FINAL_INFRASTRUCTURE_CLOSEOUT_READY_FOR_OWNER_ACCEPTANCEOWNER_DIRECTUS_LOGIN_ACCEPTANCE

FINAL INFRASTRUCTURE CLOSEOUT REV2 — Addendum

Verdict: PASS_FINAL_INFRASTRUCTURE_CLOSEOUT_READY_FOR_OWNER_ACCEPTANCE Executed (UTC): 2026-07-25 · Closeout series date: 2026-07-24 Entry state accepted: PASS_DIRECTUS_DUAL_SUPERADMIN_OWNER_READY · PASS_INTEGRITY_RUNTIME_SM_READS_REMOVED · WITH_HOLD_CRON_INTEGRITY_FUNCTIONAL_GAP Nature: closeout / clarification only. No production feature repair. No Directus identity, permission, policy or schedule change.

Mutation surface of this mission: two KB documents published (§3). SECRET_MANAGER_MUTATION = 0. DIRECTUS_PERMISSION_MUTATION = 0. PRODUCTION_RESTARTED = 0. MAC_FILES_MUTATED = 0. Zero credential values in this report.

1. Secret Manager resource clarification

The two resources are distinct Secret Manager secrets and must not be conflated.

# Resource Created Enabled versions Content Live status
1 DIRECTUS_ADMIN_TOKEN (legacy) 2026-05-22 1 single 64-char value, rotated away by C2B1-XV DEAD_HTTP_401
2 DIRECTUS_SUPERADMIN_BUNDLE (canonical) 2026-07-24 1 (ENABLED) structured JSON, two credential sets live, all auth PASS 
RESOURCES_ARE_DISTINCT                   = YES
LEGACY_DIRECTUS_ADMIN_TOKEN_STATUS       = DEAD_HTTP_401
SUPERADMIN_BUNDLE_LATEST_ENABLED_VERSION = YES
LEGACY_TOKEN_EQUALS_BUNDLE_DEFAULT_TOKEN = NO   (distinct sha256 fingerprints)

The legacy secret was probed with a static-token request, not a login — it cannot increment the 25-attempt login-fail lockout counter.

Correction to the INTEGRITY-SM-FALLBACK-R0 report (by addendum)

That report states, at its §"break-glass custody" paragraph:

"The current default-admin token is not present in Secret Manager (only the old rotated-away token is)."

This is correct only of DIRECTUS_ADMIN_TOKEN, and is wrong as a blanket claim about Secret Manager. Corrected statement:

CURRENT_DEFAULT_ADMIN_TOKEN_NOT_IN_LEGACY_SECRET  = YES   (not in DIRECTUS_ADMIN_TOKEN)
CURRENT_DEFAULT_ADMIN_TOKEN_IN_SUPERADMIN_BUNDLE  = YES   (present in DIRECTUS_SUPERADMIN_BUNDLE)

The original report's operative conclusions are unaffected: the runtime credential source remains the local root-protected env, and SCHEDULED_SECRET_MANAGER_READS = 0 still holds. Only the scope of the "not in Secret Manager" phrase is corrected. No new secret version was created to effect this correction.

2. Bundle result

DIRECTUS_SUPERADMIN_BUNDLE v1 — structure verified by keys only; no payload printed at any point.

BUNDLE_VALID_JSON      = YES
BUNDLE_TOP_KEYS        = default, owner
BUNDLE_OWNER_KEYS      = email, password, static_token
BUNDLE_DEFAULT_KEYS    = email, password, static_token

SUPERADMIN_BUNDLE_OWNER_EMAIL            = nmhuyen@gmail.com
SUPERADMIN_BUNDLE_OWNER_PASSWORD_LOGIN   = PASS
SUPERADMIN_BUNDLE_OWNER_TOKEN_IS_NULL    = YES
SUPERADMIN_BUNDLE_DEFAULT_EMAIL          = admin@example.com
SUPERADMIN_BUNDLE_DEFAULT_PASSWORD_LOGIN = PASS
SUPERADMIN_BUNDLE_DEFAULT_TOKEN_AUTH     = PASS

Lockout-safe method. Before any login was attempted, each bundle credential was compared by SHA-256 against the live-proven local custody file (owner-admin.env, breakglass/current.env). All three matched exactly, so every subsequent login was a known-positive probe — a stale bundle would have been caught by hash mismatch and skipped, never probed. Each successful login was logged out immediately; ACTIVE_ADMIN_SESSIONS = 0 was re-asserted afterwards.

BUNDLE_OWNER_PW_EQUALS_LOCAL    = YES
BUNDLE_DEFAULT_PW_EQUALS_LOCAL  = YES
BUNDLE_DEFAULT_TOK_EQUALS_LOCAL = YES

§4 conditional repair — NOT triggered

The bundle lacks no approved credential field: both credential sets are present and separate, the owner static token is correctly null, and all three live auth checks pass.

NO_SECRET_MUTATION_REQUIRED = YES
SECRET_MANAGER_MUTATION     = 0
SECRET_VERSIONS_CREATED     = 0
SECRET_VERSIONS_DELETED     = 0
SECRET_VERSIONS_DISABLED    = 0

3. Report indexing result

Measured entry state

Report Filesystem KB document storage Semantic search
integrity-sm-fallback-r0-2026-07-24.md present present searchable
directus-dual-superadmin-owner-ready-2026-07-24.md present ABSENT (404) absent
c2b1-s3-r1-simple-repoint-zero-admin-dependency-2026-07-22.md present ABSENT (404) absent

Root cause measured. The KB (Agent Data document store + Qdrant) is the system of record for published reports; the filesystem copies under /opt/incomex/docker/agent-data-repo/knowledge/** are working copies, gitignored at .gitignore:261 (reports/), so no git-based sync can ever carry them. Two reports were written to disk but never published through the governed document API. This is not a search-index failure — it is a publication-step omission: the documents were never in storage at all. (Corroboration: the C2B1-R0 packet exists only in the KB and never on disk, confirming the KB — not the filesystem — is canonical.)

Note also that reconcile-knowledge.sh (cron, 6-hourly) syncs KB → Directus knowledge_documents. It is the wrong direction to publish a report into the KB, and could never have closed this gap.

Action taken — governed path only

Both missing reports were published through the registered Agent Data document API (POST /documents, the same endpoint the KB MCP surface proxies and the same path that published the C2B1-R0 packet and the integrity report). No PostgreSQL hand-insert, no unregistered ingestion path, no new indexing framework.

DIRECT_PG_INDEX_WRITE             = 0
NEW_INDEXING_FRAMEWORK_CREATED    = 0
DOCUMENTS_PUBLISHED               = 2
Report Storage Full read to EOF Chars (KB vs disk) Integrity Search
directus-dual-superadmin-owner-ready-2026-07-24.md present, rev 1 truncated=false 12935 vs 12935 EXACT top hit, score 0.648, 4 chunks
c2b1-s3-r1-simple-repoint-zero-admin-dependency-2026-07-22.md present, rev 1 truncated=false 11774 vs 11774 EXACT top hit, score 0.847, 4 chunks
integrity-sm-fallback-r0-2026-07-24.md present (pre-existing) — 15645 vs 15646 −1 char top hit, score 0.861, 5 chunks

Both newly published documents are byte-for-byte content-equivalent to the on-disk originals (character counts verified against UTF-8 decode, not byte length — the files contain multi-byte characters, so wc -c overstates length by 73–95).

Both were read back in full to EOF and confirmed secret-free before publication: no token, password, prefix/suffix or fingerprint appears in either (the S3-R1 report asserts SECRET_PROPAGATION = 0; the dual-superadmin report carries "Zero credential values in this report" — both verified by inspection).

CANONICAL_REPORT_STORAGE_PRESENT = YES
CANONICAL_REPORT_INDEX_SEARCHABLE = YES
DIRECT_PG_INDEX_WRITE             = 0

HOLD_KB_REPORT_INDEX_RELIABILITY is not raised: the governed path indexed both reports successfully on first attempt.

One residual observation (feeds Ticket 4): the pre-existing KB copy of the integrity report is 15645 characters against 15646 on disk — a 1-character delta, almost certainly a trailing newline dropped at publication time. Cosmetic, pre-existing, not introduced here, and it does not affect readability or search. Recorded rather than silently normalised.

4. Four named follow-up tickets (registered, NOT executed)

Ticket 1 — D31-RUNNER-RECOVERY-R0

Measured state (this session):

source_system='dieu31-runner'  rows=168   last_seen_at = 2026-07-10 22:01:29Z
watchdog (hourly, truthful)    "Runner Điều 31 không chạy từ 2026-07-10T22:01:29.308Z (340h ago)"
system_issues TOTAL            = 226,408
other producers  last_seen_at  = 2026-07-25 01:30:10Z   (14–28 rows/day, continuous through the outage)

The runner has been silent since 2026-07-10; the watchdog now reports this truthfully (it previously raised false alarms from a dead credential). system_issues as a whole remains actively written by other producers throughout the window.

OUTAGE_SCOPE = RUNNER_SPECIFIC   (NOT total monitoring loss)

Future scope: identify why the runner stopped; restore its normal scheduled execution; verify the watchdog_fault heartbeat; no silent PASS.

Ticket 2 — CRON-INTEGRITY-FUNCTIONAL-REPAIR-R0

Measured state:

runners/sync-check.js:9   if (!AGENT_DATA_URL) throw new Error('Missing required env: AGENT_DATA_URL');
AGENT_DATA_URL            = UNSET anywhere in the web-test environment
machine-transition system_issues grants = create, read      (NO update)
cron-integrity.sh last two runs (2026-07-24 18:00, 2026-07-25 00:00):
  FATAL: HOLD_INTEGRITY_MONITOR_MACHINE_PERMISSION_GAP — machine identity lacks system_issues UPDATE
         (access=none). Refusing to run so 403s are not silently recorded as 'reopened'.

The new fail-closed assertion is working as designed and must be preserved.

Future scope: (1) restore the required AGENT_DATA_URL; (2) prove the real system_issues UPDATE requirement; (3) if UPDATE is genuinely required, grant only collection=system_issues, action=update via a narrow canary + DOT permission repair; (4) verify dedupe reopen/increment and autoResolveStale; (5) preserve the fail-closed assertion; (6) never substitute an admin token.

PREFERRED_PATH = NARROW_MACHINE_PERMISSION_VIA_CANARY_DOT
PG-direct is NOT the default solution.

Ticket 3 — ROOT-CRON-WRITABLE-SCRIPT-R0

LOCAL_PRIVILEGE_ESCALATION_SURFACE = YES

Measured — root's crontab executes three scripts owned by the non-root incomex user:

Script (all mode 755, owner incomex:incomex) Root cron schedule
…/web-test/scripts/integrity/cron-integrity.sh 0 */6 * * *
…/web-test/scripts/integrity/watchdog-monitor.sh 0 * * * *
…/web-test/scripts/integrity/scanner-counts.sh 0 */3 * * *

Parent directory …/scripts/integrity is also incomex:incomex 755. Anything able to write as incomex can therefore alter code that root subsequently executes — including replacing the file outright, since directory write permission allows unlink+create regardless of file mode. By contrast all 18 scripts under /opt/incomex/scripts are root:root (dir root:root 755) and are not exposed.

Future scope: determine whether the active deploy pipeline needs to overwrite those paths; if not, transfer ownership to root and retain only the necessary read/execute access; if it does, redesign the deploy/promotion step before changing ownership. No chown/chmod was performed in this mission.

Ticket 4 — KB-REPORT-INDEX-RELIABILITY-R0

Future scope: determine why published reports can exist in document storage (or on disk) but remain absent from semantic search; make governed indexing a reliable closeout gate; do not permit a report to reach PASS while required search indexing is missing.

Evidence contributed by this mission: the real failure mode observed here was not index drift but a missing publication step — two canonical reports lived only as gitignored working copies for 1 and 3 days respectively, with no automated path capable of noticing. A closeout gate should assert storage AND search for every report a mission declares canonical. The 1-character delta in §3 is a second, milder instance of unverified publication fidelity.

5. Final runtime verification

CORE_CONTAINERS               = 10
CORE_CONTAINER_CIDS_UNCHANGED = YES    (all 10 CIDs identical to session-entry baseline)
PRODUCTION_RESTARTED          = 0      (no container StartedAt falls within this session)

DIRECTUS_HEALTH               = PASS   (internal 200 · public 200)
NUXT_HEALTH                   = PASS   (https://incomexsaigoncorp.vn/ 200)
AGENT_DATA_HEALTH             = PASS   (/health 200)
POSTGRES_HEALTH               = PASS   (pg_isready: accepting connections)
QDRANT_HEALTH                 = PASS   (/healthz 200)
NGINX_HEALTH                  = PASS   (200)
BALO_GUARD_37                 = PASS   (BALO_ONE_CLICK_GUARD|PASS|checks=37)

DUAL_SUPERADMIN_COUNT         = 2
OWNER_ADMIN                   = nmhuyen@gmail.com     (Administrator · static token NULL)
BREAK_GLASS_ADMIN             = admin@example.com     (Administrator · password + static token PASS)
ACTIVE_ADMIN_SESSIONS         = 0

SCHEDULED_SECRET_MANAGER_READS = 0
DIRECTUS_PERMISSION_MUTATION   = 0
MAC_FILES_MUTATED              = 0

Authz no-drift (identical at session entry and exit, and to the 2026-07-24 baseline): permissions 1243 · policies 9 · roles 10 · access 10 · malformed 632.

Paired verifier dot-directus-owner-admin-verify --phase post-promote: 22 PASS / 0 FAIL. The read-only post-promote phase was chosen deliberately over final — final runs a knowledge_documents create/delete canary, and this closeout mission had no mandate to mutate.

SCHEDULED_SECRET_MANAGER_READS = 0 re-proven independently: zero active crontab lines invoke gcloud; watchdog-monitor.sh and cron-integrity.sh now contain no Secret Manager call. The only cron-referenced script still containing one is git-push-gh-daily.sh (v1), which appears solely in a commented-out crontab line — the scheduled job is git-push-gh-daily-v2.sh, which has no SM call.

Method note. All work was performed over SSH on the VPS. Secret handling used curl --config <(…) and stdin-fed scripts so that no credential ever entered a command line (ps-visible argv); comparisons emitted 12-character SHA-256 prefixes only.

6. Owner acceptance — PENDING

The dual super-admin topology is verified live and ready for the owner to exercise:

URL       https://directus.incomexsaigoncorp.vn
OWNER     nmhuyen@gmail.com   — password login PASS; static token NULL by design
BREAKGLASS admin@example.com  — preserved unchanged; password + static token PASS

The owner password is held only in /etc/incomex/c2b1/breakglass/owner-admin.env (root:root 0600, parent dir 0700) and in DIRECTUS_SUPERADMIN_BUNDLE. It is absent from compose, docker/.env, crontab, source, Git, this report, and the MacBook.

Optional and not blocking: the owner may enable 2FA from the Directus UI as a separate action.

7. Final next gate

PASS_FINAL_INFRASTRUCTURE_CLOSEOUT_READY_FOR_OWNER_ACCEPTANCE

NEXT GATE: OWNER_DIRECTUS_LOGIN_ACCEPTANCE

Four tickets are registered and unexecuted: D31-RUNNER-RECOVERY-R0, CRON-INTEGRITY-FUNCTIONAL-REPAIR-R0, ROOT-CRON-WRITABLE-SCRIPT-R0, KB-REPORT-INDEX-RELIABILITY-R0.

Landmines recorded

  • exit inside $(command substitution) only kills the subshell. A die() helper called as BINT="$(resolve_bint)" printed FATAL, returned empty, and the script continued — every subsequent HTTP probe silently returned 000, which reads as "dead credential" rather than "no endpoint". Validate the result after the assignment, never exit inside the substitution.
  • wc -m counts bytes, not characters, under the C/POSIX locale. It returned values identical to wc -c on UTF-8 files, which would have falsely shown a content-integrity mismatch against the KB's character count. Decode explicitly (len(open(f,'rb').read().decode('utf-8'))) when comparing.
  • The raw Agent Data POST /documents endpoint requires an API key that host-side callers do not hold; it returns 403 FORBIDDEN {"code":"FORBIDDEN"}. The authenticated MCP surface is the usable governed path for publication.
  • grep -oE over crontab -l matches commented-out lines, which made a non-scheduled SM-calling script look scheduled. Filter ^\s*# before concluding anything is on a schedule.
  • A docker inspect network-name template returning empty needs its documented fallback. Dropping the second {{range .NetworkSettings.Networks}} form from a copied resolver was what produced the empty-endpoint failure above.