KB-3CA5

VPS-RECOVERY-PREP — Rescue Non-Rebuildable VPS Assets Before Cleanup (2026-07-24)

22 min read Revision 1
current-statevps-recovery-preprecoverydisaster-recoverylocal-only-imagesimage-rescuedocker-saveencrypted-backupdrivegpgoff-vps-key-custodysecret-managerfingerprint-matchconfig-secret-bundlegit-bundleshallow-reposource-recoverynon-pg-stateuptime-kumanuxt-outputqdrantcleanup-candidateszero-mutationread-onlyPASS_VPS_RECOVERY_PREP_READY_FOR_MINIMUM_CLEANOFF_VPS_DECRYPTION_RECOVERYVPS-CLEAN-MINIMUM2026-07-24

VPS-RECOVERY-PREP — Rescue Non-Rebuildable VPS Assets Before Cleanup (2026-07-24)

VERDICT: PASS_VPS_RECOVERY_PREP_READY_FOR_MINIMUM_CLEAN Addendum: OFF_VPS_DECRYPTION_RECOVERY = PASS Mission VPS-RECOVERY-PREP SIMPLE REV1 + ADDENDUM — OFF-VPS DECRYPTION KEY CUSTODY Host Contabo vmi3080463 (38.242.240.89) · UTC window 2026-07-24 03:29Z → 06:45Z Authorization: accepts PASS_C2B1_XV_DOT_ONLY_ROTATION_RESTART_RESTORE_VERIFIED (2026-07-23) Mode: production READ-ONLY (absolute) except additive encrypted backup artifacts. Zero production mutation. Author: Claude Code (Opus 4.8, 1M ctx)


1. Verdict & PASS gate

Gate Result
POST_ROTATION_DB_RESTORE_PROVEN YESC2B1XV_POST_20260723T093848Z (restore-proven 2026-07-23; not re-run — no pipeline/artifact drift, no source-DB mutation since)
NUXT_IMAGE_RESCUE PASSnuxt-image-rescue/20260721… still present on Drive; not re-done (§5)
ALL_ACTIVE_LOCAL_ONLY_IMAGES_RESCUED YES — 4/4 this session (agent-data, claude-mcp, claude-kb, agent-api-executor) + Nuxt (2026-07-21)
ALL_NON_REBUILDABLE_ACTIVE_STATE BACKED_UP — uptime-kuma DB, served nuxt-output, directus files rescued; qdrant+PG covered by existing offsite; 0 HOLD_UNKNOWN
CANONICAL_CONFIG_SECRET_BUNDLE PASS
UNPUSHED_PRODUCTION_SOURCE_RESCUED YES
REMOTE_READ_BACK PASS (7/7 new artifacts, server-side sha256 match)
DOWNLOAD_DECRYPT_VERIFY PASS (7/7: 4 image archives structurally verified; 3 tar bundles decrypted/cloned/restored)
PLAINTEXT_RESIDUE 0
PRODUCTION_CONTAINERS_RESTARTED 0
PRODUCTION_CONFIG_MUTATED 0
PRODUCTION_DATA_MUTATED 0
CLEANUP_ACTIONS_EXECUTED 0
MAC_FILES_MUTATED 0

Addendum (off-VPS key custody): GPG_PRIVATE_KEY_IN_RECOVERY_BUNDLE=0 · OFF_VPS_PRIVATE_KEY_CUSTODY=PASS · PUBLIC_PRIVATE_FINGERPRINT_MATCH=PASS · PERSISTENT_GPG_PRIVATE_KEY_ON_VPS=0 · TEMP_PRIVATE_KEY_RESIDUE=0.

Recovery question answered — "if the VPS or a local-only image is lost right after cleanup, do we have enough to rebuild production?"YES. Every active non-rebuildable asset is offsite, encrypted, and download-decrypt-verified; the decryption key lives off-VPS in GCP Secret Manager with a proven fingerprint match.


2. Production snapshot (read-only)

Host: 6 vCPU / ~11 GiB · disk /dev/sda1 84G used / 96G (87%) · uptime 161d. Docker 29.2.1 (containerd/overlayfs image store) · 11 running containers · canonical Compose /opt/incomex/docker/docker-compose.yml (7 services) + claude-mcp/ + claude-kb/ project files.

Container CID (12) Image ID (12) Image ref Restarts Health Compose proj/svc
incomex-directus d612dfefb030 23f2f526599b directus/directus:11.5 0 healthy docker/directus
incomex-nuxt a97751e0d316 72715a92885c web-test/nuxt-ssr@sha256:72715a… (pinned) 2 healthy docker/nuxt
incomex-agent-data 73c3187a43e2 9acd60503290 agent-data-vps-prod:9acd60503290-20260717 0 healthy docker/agent-data
incomex-agent-api-executor 1d841832a6fe 54df5be4d9fc agent-api-executor-local:v1 0 healthy docker/agent-api-executor
incomex-nginx e50172b59a25 1d13701a5f9f nginx:alpine 0 docker/nginx
incomex-claude-mcp c7039e296421 7c0096602a1c claude-mcp-local:phase1a 2 healthy claude-mcp/claude-mcp
incomex-claude-kb b1b40494fe7d 3cf3065bf66e claude-kb-local:v1 2 healthy claude-kb/claude-kb
uptime-kuma 81d9f483b123 7337368a7787 louislam/uptime-kuma:2 0 healthy (docker run, no compose)
postgres 22540b78de8f 004f63c1e580 postgres:16 0 healthy docker/postgres
incomex-qdrant 2b5c328f0b4d 0425e3e03e7f qdrant/qdrant:latest 0 healthy docker/qdrant
pg-restore-test-20260520… 0cbc668c76f9 004f63c1e580 postgres:16 0 none (cleanup candidate)

Published ports (host): nginx 80/443 (0.0.0.0); agent-api-executor 127.0.0.1:8090; uptime-kuma 127.0.0.1:3001; postgres 127.0.0.1:5432. All others container-internal only. Networks: docker_incomex (8), claude_mcp_net (4), bridge (2: uptime-kuma, pg-restore-test). Docker usage: Images 36 unique / 38.63GB (31.64GB reclaimable) · Build cache 79 / 6.697GB (0 active) · Local Volumes 9 / 10.37GB (9.582GB reclaimable). Active mounts (source → size): directus uploads 8K / extensions 16K · nuxt-output 43M · qdrant/data 229M · uptime-kuma 38M · postgres data 3.4G · nginx 7.1M · letsencrypt 152K · docs/mcp-writes 7.1M · lark-client 79M · dot/specs 84K · qdrant local snapshots 1.8G.


3. Image classification (§5) — verified by repo digest + registry, not tag

Image (active) ID Classification Evidence
directus/directus:11.5 23f2f526599b PULLABLE_VERIFIED docker manifest inspect by digest = AVAILABLE
nginx:alpine 1d13701a5f9f PULLABLE_VERIFIED AVAILABLE
postgres:16 004f63c1e580 PULLABLE_VERIFIED AVAILABLE
louislam/uptime-kuma:2 7337368a7787 PULLABLE_VERIFIED AVAILABLE
qdrant/qdrant:latest 0425e3e03e7f PULLABLE_VERIFIED AVAILABLE
web-test/nuxt-ssr@72715a… 72715a92885c LOCAL_ONLY (already rescued 2026-07-21) registry DENIED (AR API disabled proj 812872501910); SOLE_PROVEN_RUNNABLE
agent-data-vps-prod:9acd… 9acd60503290 LOCAL_ONLY_RESCUE_REQUIREDrescued docker manifest inspect = errors (local-only)
claude-mcp-local:phase1a 7c0096602a1c LOCAL_ONLY_RESCUE_REQUIREDrescued local-only
claude-kb-local:v1 3cf3065bf66e LOCAL_ONLY_RESCUE_REQUIREDrescued local-only
agent-api-executor-local:v1 54df5be4d9fc LOCAL_ONLY_RESCUE_REQUIREDrescued local-only

No additional active local-only image found. (5 public images are pullable-by-digest; 5 local-only are all now offsite-proven.)


4. Local-only image rescue (§6)

Pipeline per image: docker save <exact image ID>gzip -6gpg --encrypt (public recipient, plaintext never on disk, tmpfs /dev/shm staging) → upload to dedicated Drive prefix → remote server-side sha256 read-back → download-back → decrypt in tmpfs with off-VPS private key → verify docker archive structure → shred. NO docker load, no new daemon, no container created.

Structural verification per archive (all PASS): archive opens · manifest.json well-formed · Config blob present · every Layers[] blob present (LAYERS_MISSING=0) · rootfs.diff_ids count == layer count · negative control (1-byte-flipped ciphertext) = FAILED_CLOSED · PERSISTENT_GPG_PRIVATE_KEY_ON_VPS=0 · TEMP_PRIVATE_KEY_RESIDUE=0.

Image ciphertext bytes layers verify
agent-data 9acd60503290 330,787,558 10 PASS
claude-mcp 7c0096602a1c 71,646,238 10 PASS
claude-kb 3cf3065bf66e 65,457,874 10 PASS
agent-api-executor 54df5be4d9fc 53,801,944 9 PASS

Nuxt image (72715a…) not re-done — the 2026-07-21 offsite encrypted rescue artifact remains present on Drive and its hash/read-back were PASS at creation.


5. Non-PG state classification & rescue (§7)

Mount / state Classification Disposition
uptime-kuma kuma.db (monitor defs + history) BACKUP_REQUIRED rescued — sqlite .backup online snapshot (integrity_check=ok), no container stop
deploys/nuxt-output (served frontend build, gitignored, NOT in nightly bundle) BACKUP_REQUIRED rescued (tar)
directus uploads + extensions/l2-checkpoint-guard (physical files) BACKUP_REQUIRED rescued (tar; extension source also in /opt/incomex git → §7 source bundle)
Qdrant vectors (production_documents 20,058 pts + iu_core_iu_chunks 149 pts) BACKUP_REQUIRED already covered — nightly incomex-prod-vps-config-*.tar.gz.gpg (encrypted, offsite, fresh 2026-07-23) + 8 daily local snapshots (/opt/incomex/backups/qdrant, newest 2026-07-24). No new artifact needed (§3).
postgres data (/opt/workflow/postgres/data, 3.4G) BACKUP_REQUIRED already covered — restore-proven C2B1XV_POST + nightly PG dumps + local pg-backup.sh rotation
nginx conf.d/secrets, letsencrypt BACKUP_REQUIRED already covered by nightly config bundle and re-captured in config/secret bundle (§6)
/var/log/mcp-audit, /run/incomex, node_modules/.output/.nuxt caches EPHEMERAL not backed up (logs / runtime sockets / rebuildable cache)

The three BACKUP_REQUIRED-and-not-yet-covered items are consolidated into incomex-nonpg-state-20260724.tar.gz.gpg (21,426,369 B). Download-decrypt-verify: kuma.db PRAGMA integrity_check=ok, nuxt-output/nitro.json present, negative-control FAILED_CLOSED, no private key. 0 HOLD_UNKNOWN.


6. Config/secret bundle (§8) + off-VPS key custody (Addendum)

incomex-config-secret-bundle-20260724.tar.gz.gpg (5,568,617 B; 751 files). Contents: 3 compose files · all env/secret files (docker/.env, /opt/incomex/.env*, secrets/.env.production, scripts/cron-env.sh, claude-mcp/kb env+deploy-secrets) · /etc/incomex whole tree incl. c2b1/breakglass/current.env, machine-transition.env, reconcile/, backup/recipient.pub · active nginx tree · /opt/incomex/scripts · dot/bin + dot/specs (DOT wrappers incl. 318–322 + SQL) · rclone.conf · 5 production systemd units · root crontab · recovery-locator.txt (SM resource name + fingerprints, NO payload) · production-inventory.json (image IDs/refs/mounts/ports).

Hard guarantees: plaintext never left tmpfs; staged → encrypted → uploaded → shredded (residue 0). Pre-encrypt and post-download scans for BEGIN … PRIVATE KEY armor = 0 hitsGPG_PRIVATE_KEY_IN_RECOVERY_BUNDLE=0. Download-decrypt-verify: recipient.pub / docker/.env / docker-compose.yml / recovery-locator.txt / breakglass current.env all present; negative-control FAILED_CLOSED.

Off-VPS decryption key custody (Addendum verdicts)

  • Recipient (public, on VPS): FPR 31799F7AC051EBA68FB3F13BDCC1369B371837BA, enc subkey E800896AF06E9B61.
  • Private key custody: GCP Secret Manager projects/github-chatgpt-ggcloud/secrets/DIRECTUS_BACKUP_GPG_PRIVATEKEY_RECOVERY (proj# 812872501910, userManaged europe-west3, version 1 enabled). testIamPermissions confirms secretmanager.versions.access available independent of the VPS.
  • PUBLIC_PRIVATE_FINGERPRINT_MATCH=PASS — imported SM private key to a tmpfs keyring during verification; its fingerprint 31799F7A…371837BA == recipient.pub fingerprint. Keyring shredded on every exit.
  • PERSISTENT_GPG_PRIVATE_KEY_ON_VPS=0 — root secret keyring empty; ~/.gnupg/private-keys-v1.d empty; recursive armor scan of /root/.gnupg,/etc/incomex,/opt/incomex = only 1 hit, a documentation reference in docs/recovery/…runbook.md (0 base64 armor lines, 0 END markers — prose, not key material).
  • No second recipient created, no new key generated, Owner not asked to perform technical steps (per addendum).

7. Git / source recovery (§9)

incomex-source-recovery-20260724.tar.gz.gpg (78,845,741 B). Two mechanisms:

(a) Complete repos → git bundle --all (clone-proven on download-back):

Repo HEAD refs cloned remote
/opt/incomex 714d4d54 (feat/s177-sprint1-round-a) 5 NONE (local-only)
/opt/incomex/dot c6a55793 (main) 1 NONE
/opt/incomex/git/ui-preview.git (bare) f001e57b 12 NONE
/var/lib/incomex/ui-preview-notebook.git (bare, LIVE) 1dc4497f 26 NONE

/opt/incomex bundle carries the tracked source for claude-mcp (17 files) and claude-kb (8 files) image builds.

(b) SHALLOW repos → faithful full-dir tar (bundles cannot encode shallow state) — LANDMINE: nuxt-repo and agent-data-repo are shallow clones (.git/shallow present). A --all bundle of them passes git bundle verify but CANNOT be cloned — it drags in historical vps-daily-*/vps-backup-* branches whose boundary parents lie beyond the shallow horizon (error: Could not read <parent> on clone/fetch). Even a main-only bundle fails to clone. Fix: tar the whole repo directory (worktree + .git incl. shallow markers, excluding rebuildable node_modules/.output/__pycache__). Restores to an identical working shallow repo (proven on download-back):

Repo shallow HEAD main commits unpushed
docker/nuxt-repo (+ web-rp-current linked worktree d04d8e5) true 2ea41583 356 ahead of origin (GitHub)
docker/agent-data-repo true 7f13207f 20 ahead of origin (GitHub)

Also included: deploy/agent-api-executor image-build source (Dockerfile + 7 files; gitignored, in no repo) + source-manifest.txt. Download-decrypt-verify: all 4 bundles cloned, both shallow tars restored (git log works), agent-api-executor Dockerfile present, no private key.

Dependency noted: deep history of nuxt/agent-data lives on GitHub (web-test.git / agent-data-test.git) but GitHub access needs credentials not present in the VPS git context at prep time — so it could not be confirmed live. The shallow-repo tars are therefore the authoritative VPS-unique rescue (they carry HEAD + all local branches + unpushed tips + working tree). git fetch --unshallow from GitHub is an optional post-restore deepening, not a recovery prerequisite. No commit/push/fetch was performed to satisfy recovery.


8. Recovery index (§11)

Drive base: gdrive-backup:incomex-encrypted-v1/. All artifacts GPG-encrypted to recipient 31799F7A…371837BA; restore = fetch → decrypt with SM private key (import to tmpfs GNUPGHOME, shred after) → gunzip/untar or docker load.

asset live_location classification recovery_artifact drive_path sha256 remote_readback verification restore_method cleanup_disposition
agent-data image img 9acd60503290 LOCAL_ONLY image tar.gz.gpg vps-recovery-prep/20260724/incomex-image-agent-data-9acd60503290-20260724.docker.tar.gz.gpg d5d787ad…5e3b4394 PASS struct PASS decrypt·gunzip·docker load keep image (active)
claude-mcp image img 7c0096602a1c LOCAL_ONLY image tar.gz.gpg vps-recovery-prep/20260724/incomex-image-claude-mcp-7c0096602a1c-20260724.docker.tar.gz.gpg 03c00b36…abf34f2 PASS struct PASS decrypt·gunzip·docker load keep image (active)
claude-kb image img 3cf3065bf66e LOCAL_ONLY image tar.gz.gpg vps-recovery-prep/20260724/incomex-image-claude-kb-3cf3065bf66e-20260724.docker.tar.gz.gpg 754ef109…5dcab970 PASS struct PASS decrypt·gunzip·docker load keep image (active)
agent-api-executor image img 54df5be4d9fc LOCAL_ONLY image tar.gz.gpg vps-recovery-prep/20260724/incomex-image-agent-api-executor-54df5be4d9fc-20260724.docker.tar.gz.gpg bb1d81bc…9f2d41e7 PASS struct PASS decrypt·gunzip·docker load keep image (active)
nuxt image (72715a) img 72715a92885c LOCAL_ONLY image tar.gz.gpg (2026-07-21) nuxt-image-rescue/20260721T085609Z-nuxtc0-405619/incomex-nuxt-image-…docker.tar.gz.gpg cdb59869…7089afa1 PASS (07-21) load-verified (07-21) decrypt·gunzip·docker load keep image (active)
uptime-kuma + nuxt-output + directus files binds BACKUP_REQUIRED state tar.gz.gpg vps-recovery-prep/20260724/incomex-nonpg-state-20260724.tar.gz.gpg 03d87be9…7088cef1 PASS kuma integrity=ok decrypt·untar → restore paths keep
config + secrets + /etc/incomex + compose + cron + systemd scattered BACKUP_REQUIRED config tar.gz.gpg vps-recovery-prep/20260724/incomex-config-secret-bundle-20260724.tar.gz.gpg 943c60ed…d44c6e07 PASS sentinels PASS, 0 privkey decrypt·untar → replace files keep
git bundles + shallow-repo tars + build src repos BACKUP_REQUIRED source tar.gz.gpg vps-recovery-prep/20260724/incomex-source-recovery-20260724.tar.gz.gpg 6cd82fb0…d1fb5ad8 PASS 4 clone + 2 restore PASS decrypt·untar → git clone <bundle> / extract shallow tar keep
Directus PostgreSQL DB postgres:16 vol BACKUP_REQUIRED DB dump gz.gpg (2026-07-23) c2b1xv-post/C2B1XV_POST_20260723T093848Z/incomex-prod-directus-db-20260723T093848Z.sql.gz.gpg 1e77da4c…50f5d2a PASS (07-23) restore-proven (07-23) decrypt·gunzip·psql restore keep
Qdrant vectors + host configs qdrant vol + FS BACKUP_REQUIRED nightly config bundle (2026-07-23) incomex-prod-vps-config-20260723T183701Z.tar.gz.gpg (nightly meta.json) nightly PASS + 8 daily local snapshots decrypt·untar → restore snapshot keep

GPG private key — custody = GCP SM DIRECTUS_BACKUP_GPG_PRIVATEKEY_RECOVERY (off-VPS); never placed in any recovery artifact; fetch via gcloud secrets versions access at restore time only.


9. Cleanup-candidate classification (§10) — DELETE NOTHING; classify only

Bar for DELETE_SAFE: not referenced by any container/process/cron/systemd and not a restore-proven artifact to keep and not an unrescued local-only active image and holds no unique production state and clear read-only evidence.

Candidate Classification Evidence
pg-restore-test-20260520… container + its volume 46dbcc89… QUARANTINE test restore artifact; 0 references in scripts/cron/systemd; currently running (removal needs a stop → a cleanup action for next mission)
/tmp/snap_before.sql, /tmp/c1_snapshot.sql, /tmp/c1_introspect.sql DELETE_SAFE tiny introspection snapshots (2026-06-23), unreferenced
/tmp/batch0-*-2026-07-02/* DELETE_SAFE old batch staging tarballs in /tmp, unreferenced
Unused images: old nuxt-ssr-local:*, nuxt-p10d-*, docker-nuxt, docker-agent-data, mysql:8.0, node:20-alpine, python:3.1x-slim, hello-world, busybox, *curl* DELETE_SAFE not used by any container, not compose-referenced; public ones re-pullable
Old agent-data image variants (pre-c1-schema, pre-b1, s178-cutllm*, pre-s178, agent-data-test:latest) QUARANTINE distinct IDs, unreferenced, superseded by active rescued 9acd60503290; conservative (lineage of a local-only image)
Build cache (79 entries / 6.697GB, 0 active) DELETE_SAFE rebuildable
Dangling volumes 32b405…,5838dc…,886272…,d2e14f…; pg-dry-run-*-data (×3) DELETE_SAFE old test/dry-run restore volumes, not mounted by any container
uptime-kuma-data volume QUARANTINE orphan (active uptime-kuma uses the bind mount, already rescued) — inspect before delete
Old local PG dumps in /opt/incomex/backups/{100000x,dieu44_*,dot-iu-cutter-*} + /root/p0-6-* + /root/directus-pre-iucore-6000x-…dump (≈8–9 GB unencrypted) DELETE_SAFE superseded by restore-proven offsite artifacts; unreferenced (also removes plaintext-dump exposure)
/opt/incomex/backups/pg/directus_2026-07-*.sql.gz KEEP active pg-backup.sh local rotation (cron 27 2 * * *)
/opt/incomex/backups/qdrant/*.snapshot (8 daily) KEEP active qdrant-backup.sh rotation + secondary recovery source
/root/*-checkpoint-* (c0/c1/c2a/c2b1*/nuxt-img-h1) rollback checkpoints HOLD_UNKNOWN rollback safety nets; retention is Owner's call, not derivable read-only

Headline reclaimable for VPS-CLEAN-MINIMUM: ~31.6 GB unused images + 6.7 GB build cache + 9.6 GB reclaimable volumes + ~8–9 GB old plaintext dumps. Nothing was deleted, quarantined, or moved.


10. Zero-mutation proof

  • Containers: 11 running, identical CIDs, restart counts and StartedAt unchanged vs the start-of-session snapshot (nuxt restarts=2 is the pre-existing self-restart pattern, unrelated).
  • Images: docker system df Images = 36 unique IDs at end == 36 at start. Only docker save / inspect / manifest inspect were used — all read-only; no load/build/pull/rmi/prune.
  • Production config/data: no compose/env/nginx/DOT edit; no Directus/PG/schema write; /etc/incomex/c2b1/breakglass/current.env mtime unchanged (2026-07-23), recipient.pub unchanged (2026-07-19).
  • Prohibited list (§2): every item (rm/rmi/prune, DROP/DELETE/TRUNCATE, stop/restart/recreate, rotate, chmod, edit compose/env, touch Mac) = NOT executed.
  • Key hygiene: root GPG secret keyring = 0 keys; every private-key import was to a tmpfs GNUPGHOME shredded on exit; /dev/shm residue = 0 files at close.
  • Self-introduced transients reverted: an empty /dev/shm/vpsrec dir removed; a safe.directory=* entry I added to root ~/.gitconfig during bundle verification was un-set (5 pre-existing operational entries retained) → net-zero VPS mutation.
  • MacBook: untouched — all work over SSH on the VPS; MAC_FILES_MUTATED=0.
  • Additive-only footprint: 7 new encrypted artifacts on Drive + reusable recovery scripts left at /root/vpsrec-*.sh (contain no secrets).

11. Exact next gate

VPS-CLEAN-MINIMUM — and only that. Recovery capability is proven; the classified DELETE_SAFE/QUARANTINE set above is the input for minimal cleanup. Do not begin cleanup from this mission.

Note for the cleanup mission: pg-restore-test-20260520… and uptime-kuma-data are QUARANTINE (their removal requires stopping a running container / verifying an orphan volume); the /root/*-checkpoint-* set is HOLD_UNKNOWN pending an Owner retention decision. Do not promote HOLD_UNKNOWN to DELETE_SAFE by inference.