Incomex AI Portal
KB-4FE0

DIRECTUS-OWNER-SUPERADMIN-DUAL — Closeout Report (2026-07-24)

13 min read Revision 1
current-statedirectus-dual-superadminowner-promotiondirectusbreak-glasssecret-managersuperadmin-bundledual-adminadmin-access-policyzero-restartlockout-safe2026-07-24PASS_DIRECTUS_DUAL_SUPERADMIN_OWNER_READYRETURN_TO_PRODUCT_WORK

DIRECTUS-OWNER-SUPERADMIN-DUAL — Closeout Report

Verdict: PASS_DIRECTUS_DUAL_SUPERADMIN_OWNER_READY Date (UTC): 2026-07-24 Entry gate: PASS_VPS_ADMIN_SECURITY_R0_INFRASTRUCTURE_CLOSEOUT / VPS_STABLE_FOR_PRODUCT_WORK=YES Next gate: RETURN_TO_PRODUCT_WORK Mutation surface: one directus_users row (owner) + one new Secret Manager secret. Zero container restarts. Zero credential values in this report.

0. Outcome — final topology achieved

nmhuyen@gmail.com  = active Super Admin (Administrator)   [OWNER, primary]
admin@example.com  = active Super Admin (Administrator)   [DEFAULT, preserved unchanged]
ACTIVE_FULL_ADMIN_COUNT = 2
ACTIVE_ADMIN_SESSIONS   = 0

The owner was promoted by attaching the existing Administrator role — it inherits admin_access through the existing role→policy link. No new role, policy, access row, or third admin was created. The default admin was not touched in any way.

3. Gate 0 — measured live truth (read-only, matched accepted entry state)

Identity ID Pre-state
Owner nmhuyen@gmail.com ae44e653-df5d-4a3c-bf6e-714485b616d2 active · role NULL · admin_access false · old 32-char static token present · TFA off · provider default
Default admin@example.com 6abdec55-d911-44df-af96-3cf60b9654af active · role Administrator · admin_access true · password login PASS · static token PASS
Administrator role a40a1070-0b62-4a7e-b2c0-bd4bce9d41ac —
Administrator policy (admin_access=t, app_access=t) 8a613123-2538-4943-9d67-b6d261a4789c —
role→policy access row 37e64534-2c50-48cd-97e4-a4ddf5f3f0df role-based (no per-user row)
Machine-transition user 73aa333b-3bf9-4763-b25f-27102ebb06b1 non-admin, identity intact
  • ACTIVE_FULL_ADMIN_COUNT = 1, ACTIVE_ADMIN_SESSIONS = 0 at entry.
  • Owner old token was orphaned: ≠ admin/machine/agent-data/SYNC/NUXT secrets; authenticated to nothing (no role → no perms); its only file references were three inert historical DB dumps (no live consumer). Safe to clear.

4. DOTs authored (minimal, paired, no new framework)

Reused-first check: dot-directus-breakglass-rotate is hard-scoped to admin@example.com only ("never touches any other user") → cannot promote the owner. Created exactly the two DOTs §4 allows:

DOT Tier Scope
/opt/incomex/dot/bin/dot-directus-owner-admin-promote B ONE atomic PATCH of the owner user; rollback-on-failure; never touches default admin / machine identity / any authz object / any container
/opt/incomex/dot/bin/dot-directus-owner-admin-verify A read-only paired verifier; zero negative password probes (lockout-proof); one temporary owner session, revoked

Both modelled on the proven break-glass pair: secret boundary via curl --config <(...) (never argv/temp file), SIGPIPE-safe password generation, login-then-logout session hygiene, fail-closed traps. 0755 root:root.

5. Recovery checkpoint (encrypted, offsite) — before any mutation

Logical name: DIRECTUS_OWNER_PROMOTE_PRE_20260724T095012Z Pipeline: existing backup-to-gdrive.sh (pg_dump | gzip | gpg streamed; plaintext never on disk).

Artifact:  incomex-prod-directus-db-20260724T095012Z.sql.gz.gpg   (+ .meta.json)
Remote:    gdrive-backup:incomex-encrypted-v1/  (append-only prefix; nothing evicted)
UPLOAD             = PASS
REMOTE_READBACK    = PASS   (downloaded from Drive, byte-for-byte)
SHA256_MATCH       = YES    6fa87de7aa758f308da84573cb445713386e978f12f777fad5c59e67f471abc1
                            (status == meta.json == remote readback)
Ciphertext         = OpenPGP pubkey enc, recipient keyid E800896AF06E9B61
Recipient fpr      = 31799F7AC051EBA68FB3F13BDCC1369B371837BA  (VPS holds pubkey only — cannot decrypt)
PLAINTEXT_RESIDUE  = 0

No restore test performed (same pipeline restore-proven 2026-07-21 / 2026-07-24).

6–9. Owner promotion, custody, default preservation

Atomic promotion (dot-directus-owner-admin-promote --execute) — a single transactional PATCH, authenticated by the break-glass admin static token (used as bearer only; never modified):

PATCH /users/ae44e653…  { status: active, role: Administrator, password: <fresh 32-char>, token: null }

Post-conditions (PG ground truth + live), all PASS:

OWNER_ROLE = Administrator     OWNER_ADMIN_ACCESS = true     OWNER_STATUS = active
OWNER_STATIC_TOKEN = NULL      OWNER_PASSWORD_LOGIN = PASS(200)
OWNER_OLD_STATIC_TOKEN = DENIED(401)
OWNER_CAN_READ_USERS/ROLES/POLICIES = PASS(200)   OWNER_SERVER_INFO_ADMIN_VIEW = PASS
DEFAULT admin still active (untouched)

§6 fresh password: generated in-DOT (32-char alnum, no reuse of any prior owner/default/SM value); handled only inside the DOT boundary.

§9 custody:

/etc/incomex/c2b1/breakglass/owner-admin.env   0600 root:root  (parent dir 0700)
OWNER_ENV_EMAIL_EQUALS_LIVE_USER = YES     OWNER_ENV_PASSWORD_LOGIN = PASS
Owner password value present ONLY in owner-admin.env — absent from compose, docker/.env,
.env.local, crontab, source, Git, KB, MacBook.
current.env (default break-glass) UNCHANGED (mtime 2026-07-23 11:25).

The spent rollback snapshot (owner-promote-previous.env) was shredded after the old token was proven dead.

§8 default admin preserved: active · Administrator · password login PASS · static token PASS. ACTIVE_FULL_ADMIN_COUNT = 2. Owner and default hold distinct tokens (owner NULL).

10. One Secret Manager bundle for both admins

Inventory: no existing structured dual-admin bundle (existing DIRECTUS_ADMIN_TOKEN is a single-value, now-rotated-away token; unsuitable). Created exactly one:

Secret:  DIRECTUS_SUPERADMIN_BUNDLE   (project github-chatgpt-ggcloud / 812872501910)
Labels:  purpose=directus-superadmin-dual, created-by=owner-promote-20260724
Versions: 1   (no prior version deleted — none existed)
Payload (JSON, one resource, two credential sets):
  owner:   {email: nmhuyen@gmail.com, password: <fresh>, static_token: null}
  default: {email: admin@example.com, password: <current verified>, static_token: <current verified>}

Verified by readback + live auth (no value printed):

SM_OWNER_EMAIL_EQUALS_LIVE = YES     SM_OWNER_TOKEN_IS_NULL = YES     SM_OWNER_PASSWORD_AUTHENTICATES = PASS
SM_DEFAULT_EMAIL_EQUALS_LIVE = YES   SM_DEFAULT_PASSWORD_AUTHENTICATES = PASS   SM_DEFAULT_TOKEN_AUTHENTICATES = PASS

Payload built in tmpfs, added via --data-file (never argv), tmpfs shredded. Neither live account was changed to match any old SM value.

11. Secret Manager call-frequency audit (read-only)

Runtime architecture (Directus service + six machine consumers):

NORMAL_RUNTIME_SECRET_MANAGER_READS        = 0     (cron-env.sh sources /opt/incomex/docker/.env;
                                                     machine consumers read machine-transition.env — local only)
LOCAL_ROOT_ONLY_ENV_IS_RUNTIME_SOURCE      = YES
SECRET_MANAGER_IS_OFF_VPS_RECOVERY_CUSTODY = YES

Classified gcloud secrets versions access call-sites:

Call-site Secret Cadence Class Action
deploys/web-test/scripts/integrity/watchdog-monitor.sh DIRECTUS_ADMIN_TOKEN hourly (fallback) DEAD_OR_VESTIGIAL HOLD (see below)
deploys/web-test/scripts/integrity/cron-integrity.sh DIRECTUS_ADMIN_TOKEN 6-hourly (fallback) DEAD_OR_VESTIGIAL HOLD (see below)
scripts/git-push-gh-daily.sh (v1) gh_pat_sync_secrets not scheduled (v2 in cron has no SM call) DEAD_OR_VESTIGIAL (superseded) none
dot/bin/dot-{e2e-test,pg-restore-verify,env-restore,schema-apply,knowledge-sync-agentdata,verify-ai-connections,ai-manifest} various on-demand only (none in cron/timers) RECOVERY_ONLY / manual none

Finding (HOLD sub-item — HOLD_SM_CADENCE_INTEGRITY_FALLBACK): The two web-test integrity monitors read DIRECTUS_ADMIN_TOKEN from docker/.env first and fall back to Secret Manager only if empty. The C2B1-XV rotation (2026-07-23) neutralized that key in docker/.env, so the SM fallback now fires every run (hourly + 6-hourly). Evidence measured this session: docker/.env value empty; SM DIRECTUS_ADMIN_TOKEN ≠ live admin token and returns HTTP 401 (rotated-away / dead). These scripts therefore fetch a dead token on a fixed cadence.

Why HOLD, not fix, in this mission:

  • It is not the Directus service or the six machine-consumer runtime path (those are 0 SM reads).
  • The fix is a code change to the web-test integrity subsystem (repoint to the correct current credential from a local canonical source, or gate the fallback), not a mere schedule change.
  • That subsystem is entangled with the open dieu31 investigation (the watchdog alerts are about "Runner Điều 31"). Safe reduction cannot be proven within this mission's scope.
  • SCHEDULE_MUTATION = 0. Owner promotion + SM bundle still PASS (§11 explicitly permits this).

Recommended follow-up (separate window): in watchdog-monitor.sh and cron-integrity.sh, replace the DIRECTUS_ADMIN_TOKEN acquisition with the current canonical local credential (e.g. the shared machine-transition token for read-only liveness checks), removing the SM fallback entirely → FREQUENT_SM_CALLS → 0. Do not update the SM DIRECTUS_ADMIN_TOKEN to the live token: that would institutionalise hourly SM reads of a live admin credential (worse).

12. Final verification

OWNER_SUPERADMIN_EMAIL         = nmhuyen@gmail.com        DEFAULT_SUPERADMIN_EMAIL = admin@example.com
OWNER_PASSWORD_LOGIN           = PASS                     DEFAULT_PASSWORD_LOGIN   = PASS
OWNER_STATIC_TOKEN             = NULL                     DEFAULT_STATIC_TOKEN     = PASS
OWNER_STATUS                   = active                   DEFAULT_STATUS           = active

ACTIVE_FULL_ADMIN_COUNT        = 2                        ACTIVE_ADMIN_SESSIONS    = 0
ONE_CANONICAL_SM_BUNDLE        = PASS (DIRECTUS_SUPERADMIN_BUNDLE, 1 version)
NORMAL_RUNTIME_SM_READS        = 0 (service+consumers); integrity-monitor fallback HELD (§11)
WORLD_READABLE_SECRET_FILES    = 0 (credential custody/config)

SHARED_MACHINE_IDENTITY        = PASS (granted 200 / admin surface 403 / token bound)
MACHINE_CONSUMERS              = machine identity intact; nuxt homepage 200, agent-data healthy,
                                 reconcile-tasks + reconcile-knowledge "All in sync" (4/6 live-checked;
                                 dot-collection-health / dot-apr-health share the same intact identity and
                                 run on their own cron — not invoked here, respecting the APR hard-lock)

CORE_CONTAINERS                = 10        CORE_CONTAINER_CIDS_UNCHANGED = YES     PRODUCTION_RESTARTED = 0
MAC_FILES_MUTATED              = 0 (all work via SSH on the VPS; DOTs staged in ephemeral session
                                 scratchpad then installed on the VPS; no MacBook / Mac MCP / user files touched)
ALL_DIRECTUS_MUTATIONS_VIA_DOT = YES       DIRECT_AGENT_WRITE_COUNT = 0

Paired verifier --phase final: 37 PASS / 0 FAIL. No-drift snapshot: permissions 1243 · malformed 632 · policies 9 · roles 10 · access 10 (identity promotion structurally cannot alter these — only one user row changed).

13. Deferred items (remain deferred — untouched)

dot-apr-execute / APR-0234 · workflow_steps/delete permission · 632 malformed permission rows · GCP legacy cleanup · GitHub legacy cleanup · off-GCP private-key escrow · 2FA setup (owner may enable later as a separate UI action — does not block this mission).

Observations (non-blocking, out of scope)

  • World-readable historical DB dumps contain tokens: /opt/incomex/backups/mysql-pre-pg-migration.sql and /opt/incomex/docker/directus-dump.sql (both 0644). Pre-existing; recommend a future hygiene pass (restrict perms or purge). Not modified here.
  • The HOLD_SM_CADENCE_INTEGRITY_FALLBACK finding above is worth folding into the dieu31 work.

Landmines recorded

  • docker exec -i inside an ssh 'bash -s' heredoc drains the script stdin → use docker exec (no -i) in bash-fed scripts, or run as a file with </dev/null.
  • Directus 11.5 admin access = policy-level (directus_policies.admin_access) reached via directus_access from a role; setting a user's role to Administrator is sufficient — no per-user access row needed.
  • Old owner static token authenticated to nothing pre-promotion (user had no perms) yet was still present; a bearer-token denial (401) — not a password login — is the lockout-safe way to prove it dead.
  • SM DIRECTUS_ADMIN_TOKEN (v1, 2026-05-22) holds a rotated-away token (401); two integrity monitors fetch it hourly/6-hourly because docker/.env was neutralised.

PASS_DIRECTUS_DUAL_SUPERADMIN_OWNER_READY → next gate RETURN_TO_PRODUCT_WORK.