Incomex AI Portal
KB-3151

INTEGRITY-SM-FALLBACK-R0 — Remove dead Secret Manager fallback from two integrity monitors (2026-07-24)

16 min read Revision 1
current-stateintegrity-sm-fallback-r0watchdog-monitorcron-integritysecret-managermachine-transitionlocal-machine-identityfail-closedfalse-alert-fixdieu31agent-data-urlsystem-issues-updatepermission-gapbreak-glassdual-superadminargv-exposurezero-restart2026-07-24PASS_INTEGRITY_RUNTIME_SM_READS_REMOVEDWITH_HOLD_CRON_INTEGRITY_FUNCTIONAL_GAPHOLD_INTEGRITY_MONITOR_MACHINE_PERMISSION_GAP

INTEGRITY-SM-FALLBACK-R0 — Remove dead Secret Manager fallback from two integrity monitors

  • Date (UTC): 2026-07-24
  • Scope: watchdog-monitor.sh, cron-integrity.sh (credential source only)
  • Entry state accepted: PASS_DIRECTUS_DUAL_SUPERADMIN_OWNER_READY, PASS_VPS_ADMIN_SECURITY_R0_INFRASTRUCTURE_CLOSEOUT

1. Verdict

PASS_INTEGRITY_RUNTIME_SM_READS_REMOVED
WITH_HOLD_CRON_INTEGRITY_FUNCTIONAL_GAP

NORMAL_RUNTIME_SECRET_MANAGER_READS = 0
WATCHDOG_MONITOR                    = PASS
CRON_INTEGRITY_MONITOR              = HOLD_PREEXISTING
PRODUCTION_REGRESSION               = 0

Per-monitor:

WATCHDOG_MACHINE_AUTH                    = PASS
WATCHDOG_FALSE_ALERT_FIXED               = YES
WATCHDOG_SECRET_MANAGER_READS            = 0
WATCHDOG_CURRENT_BEHAVIOR                = FUNCTIONAL

CRON_INTEGRITY_SECRET_MANAGER_READS      = 0
CRON_INTEGRITY_CURRENT_PRIMARY_FAILURE   = MISSING_AGENT_DATA_URL
CRON_INTEGRITY_MACHINE_UPDATE_PERMISSION = MISSING
CRON_INTEGRITY_FUNCTIONAL_STATUS         = HOLD
HOLD_INTEGRITY_MONITOR_MACHINE_PERMISSION_GAP = OPEN

Architecture achieved:

INTEGRITY_MONITORS_USE_LOCAL_MACHINE_IDENTITY = YES
ADMIN_CREDENTIAL_USED_BY_MONITORS             = NO

2. The two scripts and their schedules

Three on-disk copies of each name exist; only the deploys/web-test copy is scheduled and was touched.

Script (active path) Schedule (root crontab, unchanged)
/opt/incomex/deploys/web-test/scripts/integrity/watchdog-monitor.sh 0 * * * * — hourly
/opt/incomex/deploys/web-test/scripts/integrity/cron-integrity.sh 0 */6 * * * — every 6h

Untouched copies (not scheduled, not referenced by cron/systemd): /opt/incomex/docker/nuxt-repo/scripts/integrity/*, /opt/incomex/docker/nuxt-repo/web-rp-current/scripts/integrity/*.

No systemd unit or timer references either monitor. CRONTAB_IDENTICAL_TO_PRE = YES.

File integrity:

watchdog-monitor.sh  PRE  0547886957f8d5c8...  POST d55f7c112b5528df...
cron-integrity.sh    PRE  c68e419fe8618b70...  POST 35fd49fd52127027...

Checkpoint of originals: /root/integrity-sm-fallback-r0-checkpoint-20260724T144500Z/ (0700, originals + crontab + PRE sha256).

3. Old fallback behavior (measured, not assumed)

Both scripts resolved the Directus credential in three steps: env var → /opt/incomex/docker/.env DIRECTUS_ADMIN_TOKEN= → Secret Manager gcloud secrets versions access latest --secret=DIRECTUS_ADMIN_TOKEN --project=github-chatgpt-ggcloud.

Measured facts:

  • docker/.env:27 DIRECTUS_ADMIN_TOKEN= is present but empty (neutralised in an earlier mission). Extracted length = 0.
  • Cron sets no DIRECTUS_TOKEN. Therefore the Secret Manager branch was taken on every single run.
  • The Secret Manager value is the old rotated-away admin token. Live probe against the exact watchdog query returned HTTP 401.

Consequences measured:

  • watchdog-monitor.sh — curl -sfg ... || echo '{"data":[]}': -f turns the 401 into a curl failure, the || substitutes an empty result set, and the script printed WATCHDOG ALERT: Runner Điều 31 chưa bao giờ chạy hoặc không tìm thấy WATCHDOG issue. This was a false alert: 2 watchdog_fault rows exist in system_issues, and the machine identity retrieves them (200, 1 row with last_seen_at). An authentication failure was being reported as "the runner never ran" — indistinguishable from a genuine runner outage.
  • cron-integrity.sh — read Secret Manager, passed its fail-fast, then main.js died at import time: runners/sync-check.js:9 Error: Missing required env: AGENT_DATA_URL. First occurrence cron-20260402-120002; 162 consecutive crashed runs; === Done: lines in cron.log = 0. It therefore performed zero Directus HTTP calls — its 401s were unobservable, not absent — while still consuming one Secret Manager read per run.
  • The old FATAL path logged ${DIRECTUS_TOKEN:0:50} — a 50-character secret prefix into cron.log.
  • Both scripts passed the bearer token via curl -H, exposing it in argv (visible to any local user via ps) on every run. Verified with a positive control: the -H form is visible in ps -eww.

Scheduled Secret Manager reads caused by these two scripts: 24/day (watchdog) + 4/day (cron) = 28/day, all resolving to a dead credential.

4. Minimal patch

Only the two scripts were modified. No token value appears in either file (verified by byte comparison), in argv, in logs, or in this report.

Both scripts now resolve the credential from the canonical local file only:

/etc/incomex/c2b1/machine-transition.env   root:root 0600
key: MACHINE_TRANSITION_DIRECTUS_TOKEN

The file is sourced in place — no copy of the token was written anywhere. Writes were atomic (temp file in the same directory → bash -n → mv -f), with owner/mode preserved via --reference (incomex:incomex 755, unchanged).

watchdog-monitor.sh

  1. Docker-.env → Secret Manager chain replaced by sourcing the canonical machine credential.
  2. Absent/unreadable/empty credential now fails visibly (WATCHDOG FATAL: ...) instead of degrading.
  3. Bearer token passed via curl -K <(...) so it never enters argv.
  4. HTTP status is captured and classified. 401/403 now produce a distinct FATAL instead of being masked as "runner never ran" — this is the false-alert fix.
  5. All liveness logic (26h threshold, parsing, messages) left byte-identical.

cron-integrity.sh — secret hygiene plus one fail-closed assertion, nothing else:

  1. Secret Manager fallback removed; canonical machine credential sourced.
  2. Fail-fast replaced with an emptiness check. The ${DIRECTUS_TOKEN:0:50} log leak is gone, as is the old grep -qi "error\|not found\|permission" test, which would have false-positived on any token containing those substrings.
  3. Dead gcloud-SDK PATH scaffolding removed (verified it shadowed neither node nor python3).
  4. One fail-closed capability assertion added (see §5). It refuses to run when system_issues UPDATE is not genuinely granted, and logs HOLD_INTEGRITY_MONITOR_MACHINE_PERMISSION_GAP.
  5. AGENT_DATA_URL not fixed. main.js, dedupe.js, runners, schedules, log rotation: untouched.

5. Machine-identity capability proof

Identity: machine-transition@incomexsaigoncorp.vn — single access row, single policy machine-transition-policy, admin_access=false, app_access=false.

Required Directus operations, derived by tracing the actual code paths:

Operation Required by Class Machine identity
GET /items/system_issues watchdog query; dedupe.js lookup READ 200 — granted
GET /items/knowledge_documents runners/sync-check.js READ 200 — granted
GET /server/health health-gate.js READ (public) 200
POST /items/system_issues dedupe.js create SAFE_OPERATIONAL_WRITE granted
PATCH /items/system_issues/:id dedupe.js reopen/increment + autoResolveStale SAFE_OPERATIONAL_WRITE 403 — NOT granted

Confirmed by two independent sources that agree:

  • PostgreSQL directus_permissions for the policy on system_issues: create + read only.
  • Directus /permissions/me: read: full, create: full, update: {"access":"none"}.
  • Live non-mutating probe: PATCH /items/system_issues/999999999 → 403 (permission is evaluated before existence, so nothing could be modified). Residue check: 0 rows created in the preceding 45 minutes; newest row predates the probe.
MACHINE_IDENTITY_COVERS_REQUIRED_MONITOR_OPERATIONS:
  watchdog-monitor.sh = YES   (needs system_issues READ only)
  cron-integrity.sh   = NO    (needs system_issues UPDATE)

Why this is fail-closed rather than tolerated. dedupe.js:129 discards updateIssue()'s return value and returns action: 'reopened' unconditionally; main.js:147 then counts it as a successful reopen. A 403 would therefore be recorded as success while occurrence_count and last_seen_at never advance — and because the watchdog reads last_seen_at, a stale value would eventually make the watchdog report a dead runner. That is silent, self-reinforcing corruption of the integrity ledger. autoResolveStale (dedupe.js:201) does check the result but logs nothing.

Per the mission gate, no permission was granted and no admin credential was substituted. Instead cron-integrity.sh refuses to run and names the gap. The assertion distinguishes a dead credential (HTTP != 200) from a genuine permission gap, so a future credential failure is not misreported as this HOLD.

Verified branch behaviour (all exit non-zero):

normal run           -> FATAL: HOLD_INTEGRITY_MONITOR_MACHINE_PERMISSION_GAP (access=none)
invalid credential   -> FATAL: permission introspection failed (HTTP 401)      [distinct]
missing credential   -> FATAL: machine credential file not readable            [distinct]

6. Secret Manager calls before / after

                          BEFORE            AFTER
watchdog-monitor.sh       24 reads/day      0
cron-integrity.sh          4 reads/day      0
TOTAL SCHEDULED                28/day       0

SECRET_MANAGER_RUNTIME_CALLS_IN_TWO_SCRIPTS = 0
ADMIN_SECRET_REFERENCES_IN_TWO_SCRIPTS      = 0
LOCAL_MACHINE_IDENTITY_REFERENCE            = PASS (2/2 files)
SECRET_MANAGER_CALLS_DURING_RUN             = 0   (ps-monitored across all runs)
HTTP_401_FROM_DEAD_ADMIN_TOKEN              = 0

No cron entry on the host invokes gcloud secrets versions access. scanner-counts.sh (every 3h, same directory) contains no Secret Manager or admin-token reference — so the host-wide claim holds:

VPS_RUNTIME_SOURCE             = LOCAL_ROOT_PROTECTED_ENV
SECRET_MANAGER_PURPOSE         = OFF_VPS_RECOVERY_CUSTODY
SCHEDULED_SECRET_MANAGER_READS = 0

No weekly Secret Manager job was created. Secret Manager is now touched only on credential change, manual parity audit, or disaster recovery. The one-off Secret Manager reads in this mission were audit reads under that policy.

7. Runtime smoke

CORE_CONTAINERS               = 10
CORE_CONTAINER_CIDS_UNCHANGED = YES   (CID-set md5 26f215dc... identical pre/post)
PRODUCTION_RESTARTED          = 0     (every StartedAt predates this session)

DIRECTUS_HEALTH               = PASS  (200)
NUXT_HEALTH                   = PASS  (200)
NGINX_HEALTH                  = PASS  (200)
AGENT_DATA_HEALTH             = PASS  (docker healthcheck: healthy)
POSTGRES_HEALTH               = PASS  (accepting connections)
QDRANT_HEALTH                 = PASS  (docker healthcheck: healthy)
BALO_GUARD_37                 = PASS  (BALO_ONE_CLICK_GUARD|PASS|checks=37)

DUAL_SUPERADMIN_COUNT         = 2     (admin@example.com, nmhuyen@gmail.com — both active, Administrator, admin_access=true)
ACTIVE_ADMIN_SESSIONS         = 0
DIRECTUS_PERMISSION_MUTATION  = 0     (1243 total rows; machine policy 36 — the documented post-R0 state)
CREDENTIAL_ROTATION           = 0
MAC_FILES_MUTATED             = 0

Stated limitation. OWNER_ADMIN_LOGIN and DEFAULT_ADMIN_LOGIN were not exercised as live password logins. Doing so would create admin sessions (contradicting ACTIVE_ADMIN_SESSIONS = 0) and use a Super Admin credential, in a mission that touched no identity, password or token. Both accounts were verified structurally instead (active, Administrator role, admin_access=true), and the default-admin static token was proven live (HTTP 200). This is recorded as structural verification, not a login test.

Runs performed through the normal execution path only; no new testing framework was introduced.

8. ADDENDUM — dual admin ratification and default-token classification

DUAL_SUPERADMIN_TOPOLOGY_RATIFIED = YES
OWNER_ADMIN                       = nmhuyen@gmail.com
BREAK_GLASS_ADMIN                 = admin@example.com
ACTIVE_FULL_ADMIN_COUNT           = 2

Neither account was suspended, de-roled or deleted.

Three distinct values — measured, not inferred. Compared internally by byte/hash equality; no value or reusable fingerprint is recorded here.

Token Distinct? Live status
Old rotated-away Secret Manager token distinct from both others HTTP 401 — dead
Current admin@example.com static token distinct from both others HTTP 200 — live
Current machine-transition token distinct from both others HTTP 200 — live 
A == B = NO      A == C = NO      B == C = NO

The owner admin (nmhuyen@gmail.com) static token is NULL, as established in the preceding mission.

Runtime consumer survey of the current default-admin static token (searched by value, not variable name, across active container environments for all 10 containers, Compose and .env files, all user crontabs, /etc/cron.d, /etc/crontab, systemd units, /etc/incomex, /root, /usr/local/{bin,sbin}, /etc, /opt/incomex scripts and configs, /var/lib/incomex, /srv, /home):

CURRENT_DEFAULT_ADMIN_TOKEN_RUNTIME_CONSUMERS  = 0
CURRENT_DEFAULT_ADMIN_TOKEN_BREAKGLASS_CUSTODY = YES
DEFAULT_ADMIN_TOKEN_DISPOSITION                = RETAINED_BREAK_GLASS_ONLY

Sole occurrence: /etc/incomex/c2b1/breakglass/current.env (root:root 0600, directory 0700) — break-glass custody, not a runtime consumer. The current default-admin token is not present in Secret Manager (only the old rotated-away token is). Corroboration: incomex-nuxt carries the machine-transition token in NUXT_DIRECTUS_SERVICE_TOKEN, not the default-admin token.

The current default-admin token was not changed or deleted in this mission. Whether to remove the break-glass static token is a future security decision and does not block RETURN_TO_PRODUCT_WORK.

9. Deferred items — unchanged, verified untouched

Item State
workflow_steps/delete (perm 1550) Not restored. Machine policy = 36 rows vs governed 37 — the documented post-R0 state.
APR-0234 / dot-apr-execute Untouched.
632 malformed permission rows Untouched.
AGENT_DATA_URL crash in the dieu31 runner Not fixed — root cause of CRON_INTEGRITY_CURRENT_PRIMARY_FAILURE. Still present in code; it will resurface once the permission gate is resolved.
dieu31 Vòng B red flag (7.7% pass rate) Untouched, still open.
main.js, dedupe.js, runners, contract-reader.js Untouched.
post-deploy-check.sh Still contains a Secret Manager fallback, but is not scheduled (0 cron references; deploy-time only). Out of scope; left unchanged.
Directus users / roles / policies / permissions Zero mutations.

Observation for a future mission (not acted on). Both monitors are owned by incomex:incomex mode 755 yet executed by root cron, and now read a root-only credential. This is a pre-existing exposure — the incomex user could already cause root cron to execute arbitrary code. This mission reduces its blast radius: the credential these scripts resolve is now a non-admin machine token instead of a Super Admin token. Tightening ownership was not attempted because it risks the web-test deploy pipeline.

10. Next gate

RETURN_TO_PRODUCT_WORK

cron-integrity.sh reopens only in a dedicated product mission covering: fix AGENT_DATA_URL; establish the genuine need for system_issues UPDATE; if required, grant via canary + DOT — never by switching to an admin token or PG-direct.

No further credential mission is opened by this report.