Incomex AI Portal
KB-213F

QDRANT Vector Liveness and Orphan R1 — 2026-07-25

45 min read Revision 1
qdrantvector-syncorphanmonitoringreport2026-07-25

QDRANT VECTOR LIVENESS AND ORPHAN — R1

Verdict: HOLD_QDRANT_ORPHAN_AND_MONITOR_BLIND (Stage 2 triggered and executed) Executed (UTC): 2026-07-25 · Mode: READ-ONLY Entry state accepted: PASS_FINAL_INFRASTRUCTURE_CLOSEOUT_READY_FOR_OWNER_ACCEPTANCE · VPS_STABLE = YES Mutation surface of this mission: one KB document published (this report). Zero production mutations (§14).

1. Executive verdict

Five questions, answered from live evidence.

# Question Answer
1 Orphan vectors present now? YES — exactly 2, both in production_documents. Neither is the historical root cause.
2 Eligible source records missing vectors? NO — 0 in both collections.
3 Vector-sync pipeline alive, idle, or stalled? production_documents = ALIVE / AUTOMATIC (last index today 06:00:25Z). iu_core_iu_chunks = IDLE BY DESIGN (gate deliberately shut, no new eligible source, no deployed writer). Neither is stalled.
4 OpenAI embedding usable, incl. billing/quota? YES. 30 successful embeddings today (13 at 03:27–03:33Z, 17 at 06:00Z). Zero quota/billing/auth errors anywhere. OPENAI_API_BILLING_BLOCKER_CURRENT = NO.
5 Historical soft-delete resurrection bug still live? The 2026-05-03 path is FIXED (trigger + listener both patched and deployed; all 47 historical orphans gone). A different, un-fixed resurrection path is live and produced both current orphans.

The mission's stop-conditions are not met. Stage 2 ran automatically. Two independent defects were proven, plus a third that explains why neither was ever reported.

CURRENT_TOTAL_ORPHANS               = 2
MISSING_ELIGIBLE_SOURCE_IDS         = 0
ELIGIBLE_BUT_UNINDEXED_COUNT        = 0
DUPLICATE_POINT_IDS                 = 0
MALFORMED_PAYLOAD_COUNT             = 9      (production_documents)
STALE_VERSION_IDS                   = 10     (production_documents)
QDRANT_ENGINE_HEALTH                = PASS
OPENAI_EMBEDDING_CANARY             = NOT_RUN — recent success proven (§8)
HISTORICAL_ROOT_CAUSE               = FIXED
NEW_ROOT_CAUSE                      = REINDEX_RACE_RESURRECTION (live, unfixed)
NO_ALERT_ROOT_CAUSE                 = MONITOR_BROKEN

The single most important finding is not the 2 orphans. It is that the daily vector audit has been dead since 2026-07-11, and the system has had no working vector-integrity monitor for 14 days — the orphan count is small only by luck.

2. Qdrant health

Single node, no cluster, no aliases.

Field production_documents iu_core_iu_chunks
status green green
optimizer_status ok ok
points_count 20 092 149
indexed_vectors_count 20 172 0
vector dimension 1536 1536
distance metric Cosine Cosine
vector name "" (unnamed) unnamed
segments 3 3
shard / replication 1 / 1 1 / 1
on_disk_payload true true
indexing_threshold 10 000 10 000
payload indexes NONE (payload_schema {}) NONE (payload_schema {})
aliases none none

Two values need explicit interpretation rather than alarm:

  • iu_core_iu_chunks.indexed_vectors_count = 0 is not a fault. 149 points is far below the collection's indexing_threshold = 10 000, so Qdrant deliberately has not built an HNSW graph and serves that collection by exact scan. Correct behaviour at this size.
  • production_documents.indexed_vectors_count (20 172) > points_count (20 092) is the normal Qdrant accounting of vectors still resident in segments pending vacuum after deletion. It is not a duplicate-vector signal; the independent duplicate check below returns 0.

Engine log analysis. 423 005 log lines retained, spanning 2026-06-12T00:00:59Z → 2026-07-25T08:58Z:

ERROR lines            = 0
WARN lines             = 0
panic / corrupt / fail = 0
HTTP 200               = 422 787
HTTP 401               = 4
HTTP 404               = 3

QDRANT_ENGINE_HEALTH   = PASS
CURRENT_QDRANT_ERRORS  = 0 (all classes)
LAST_QDRANT_ERROR_AT   = NONE
RestartCount           = 0   (running since 2026-03-18T07:03:47Z, health=healthy, FailingStreak=0)
storage                = 232 MB   ·  host disk 61 % used

The 7 non-2xx responses are fully accounted for and none is an engine fault:

  • 4 × HTTP 401 on 2026-07-24T07:24–07:28Z from 172.18.0.7 (nginx) with User-Agent: Wget — an unauthenticated probe from a prior session. Pre-existing, not from this mission.
  • 3 × HTTP 404 on 2026-07-25T05:55Z from 172.18.0.5 — this mission's own read-only probes (/collections/aliases, /collections/*/index, wrong paths on this Qdrant version; the correct /aliases returned 200 with an empty list). Disclosed here rather than omitted.

Honest bound on the historical aggregation. §4 asks for aggregation since 2026-05-24. Docker log retention for this container begins 2026-06-12; the 2026-05-24 → 2026-06-11 window is not retained and cannot be measured. The Qdrant container has RestartCount = 0 and has run continuously since 2026-03-18, so no restart-crash occurred in the unretained window, but per-request error counts for those 19 days are UNKNOWN, not 0. Recorded rather than assumed.

Read traffic is not indexing evidence. Per §4 this report never infers pipeline health from HTTP 200s. Every liveness claim below rests on PUT …/points write records, PG state and embedding counters.

3. Collection / source map

Collection authority was proven, not assumed.

Collection Canonical source authority Proof
production_documents incomex_metadata.kb_documents (Agent Data KB) QDRANT_COLLECTION=production_documents in the agent-data runtime env; vector_store.upsert_document writes payload.document_id from kb_documents.data->>'document_id'; _run_audit compares exactly these two sets.
iu_core_iu_chunks directus IU Core — unit_version (lifecycle_status='enacted') ⋈ v_ui_iu_three_axis_envelope, registry iu_vector_sync_point iu_qdrant_collection_registry row: iu_core_iu_chunks / 1536 / Cosine / openai:text-embedding-3-small / purpose=iu_core_per_iu_chunks / status=active.

The per-IU boundary rule holds: production_documents is absent from the IU registry, and no IU point carries a document_id payload. The two corpora have never mixed.

4. Complete orphan comparison

Both directions, full pagination, no sampling.

4.1 production_documents

QDRANT_SCROLL_COMPLETE      = YES   (41 pages × 500, 20 092 points == API points_count)
SOURCE_POPULATION_COMPLETE  = YES   (12 114 kb_documents rows, unfiltered read)

TOTAL_POINTS                = 20 092
UNIQUE_SOURCE_IDS           = 9 930      (+ 9 points carrying an empty document_id)
PG_ACTIVE                   = 9 933
PG_SOFT_DELETED             = 2 181
EXPECTED_SOURCE_IDS         = 9 385      (eligible after the live exclusion contract)

ORPHAN_SOURCE_IDS           = 2
MISSING_ELIGIBLE_SOURCE_IDS = 0
DUPLICATE_POINT_IDS         = 0
DUPLICATE (document_id, chunk_index) pairs = 0
MALFORMED_PAYLOAD_COUNT     = 9
STALE_VERSION_IDS           = 10

Exclusions applied from the live contract, read out of the deployed fn_kb_notify_vector_sync and _classify_non_vectorizable, not invented for this report:

Exclusion Count Authority
registries/** 225 trigger filter 2
directory_like_path (no . in basename) 217 _classify_non_vectorizable
operations/tasks/comments/** 91 trigger filter 1
.tmpl template files 10 _classify_non_vectorizable
empty/short body (< 10 chars) 4 trigger filter 4 + listener guard
empty key 1 trigger filter 3

The 2 orphans — the complete list, nothing withheld:

# document_id PG deleted_at PG vector_status Qdrant chunks
1 operations/tasks/comments/comment-1842 2026-05-27T10:19:56.991315Z ready 1
2 registries/workflow_step_relations/wsr-87 2026-07-22T11:17:22.021502Z ready 1

A decisive predicate: across all 2 181 soft-deleted rows, exactly 2 179 carry vector_status='deleted' (the correct API delete path) and exactly 2 carry vector_status='ready' — and those 2 are precisely the orphans. deleted_at IS NOT NULL AND vector_status='ready' is a perfect, zero-false-positive detector for this leak class. That single predicate is the cheapest possible monitor and is carried into §12.

MALFORMED_PAYLOAD_COUNT = 9 — 9 points whose document_id is the empty string. They are unattributable to any source row and are invisible to the existing audit, which maps a missing document_id to "" on both sides and therefore matches them against the one empty-key PG row. Not caused by this mission; recorded as a standing hygiene item.

Note the payload shape, because it is a trap for any future auditor: chunk metadata lives at payload.metadata.chunk_index / payload.metadata.total_chunks, not at the payload root. A root-level check reports all 20 092 points malformed — a false alarm this mission hit and corrected before reporting.

STALE_VERSION_IDS = 10 — measured authoritatively by importing the live _split_text (CHUNK_SIZE=4000, CHUNK_OVERLAP=400) from the running container, re-chunking every PG body and comparing chunk-for-chunk against the stored Qdrant content:

compared              = 9 928
EXACT_MATCH           = 9 918   (99.90 %)
CHUNK_COUNT_MISMATCH  = 1       knowledge/current-state/plans/vps-migration-plan.md
                                (3 points present, 2 expected → 1 residual chunk from a longer
                                 earlier revision that was never deleted)
CONTENT_MISMATCH      = 9       PG body has moved on; the stored vector text is an older revision

All 10 carry vector_status='ready', so PG asserts they are in sync when they are not — a second, quieter false-green surface. No PG write occurred between the scroll and the body read (0 rows written since 06:00:25Z), so these are genuine, not a measurement race.

indexed_but_not_eligible = 543 — a real asymmetry, not a defect. The PG trigger excludes registries/** and operations/tasks/comments/** from NOTIFY, but the HTTP API upsert path indexes them unconditionally. Both classes therefore have vectors that no trigger event can ever maintain or retract. This asymmetry is exactly what makes the two orphans permanent (§10).

4.2 iu_core_iu_chunks

QDRANT_SCROLL_COMPLETE      = YES   (1 page, 149 points == API points_count)
SOURCE_POPULATION_COMPLETE  = YES

TOTAL_POINTS                = 149
UNIQUE_SOURCE_IDS (unit_id) = 141
EXPECTED_SOURCE_IDS         = 141
registry: indexed rows      = 149   ·  indexed unique units = 141   ·  dryrun = 3

ORPHAN_SOURCE_IDS           = 0
MISSING_ELIGIBLE_SOURCE_IDS = 0
DUPLICATE_POINT_IDS         = 0
DUPLICATE point_key         = 0
MALFORMED_PAYLOAD_COUNT     = 0     (149/149 carry all six required boundary fields)
STALE_VERSION_IDS           = 0     (indexed_digest ≠ content_digest on 0 of 149 rows)

The envelope holds 146 enacted IUs; 141 are indexed. The 5 unindexed are:

a67cb4bc-f60f-40ab-8c94-3edd508cb215    body_len = 0
1601c9b3-07e5-4b69-abc9-61087f6834ab    body_len = 0
d898dc3c-3bc8-4f3a-b94f-032f29984e68    body_len = 0
e877ce90-d20d-4fb2-b76e-ded2f36a51a8    body_len = 0
bdd69aba-8202-4197-970c-95b3ba644b47    body_len = 0

All five have a zero-length body. There is nothing to embed; the enacted-with-authoritative-body contract excludes them. MISSING_ELIGIBLE_SOURCE_IDS = 0 is therefore correct, not a rounding-down. Every registry row still maps to a currently-enacted unit (0 stranded rows).

The 3 dryrun rows are correctly NOT lost production vectors. They are the two collection manifests (iu_core.composer.pilot-doc-001, iu_core.autocut.file-001) and _corpus/corpus-manifest — source_kind = collection/corpus, unit_id NULL, indexed_digest NULL. The live contract (purpose='iu_core_per_iu_chunks', plus the DB CHECK iu_vector_sync_point_boundary_chk) forbids non-iu source kinds in this collection. They were never supposed to become indexed and are classified as intentional dryrun bookkeeping.

5. Pipeline mode and liveness

5.1 production_documents — ALIVE

VECTOR_SYNC_PIPELINE_MODE        = AUTOMATIC
VECTOR_SYNC_GATE_ENABLED         = N/A (no gate governs this collection)

LAST_SUCCESSFUL_INDEX_AT         = 2026-07-25T06:00:25Z
LAST_SUCCESSFUL_EMBEDDING_AT     = 2026-07-25T06:00:25Z
LAST_SUCCESSFUL_QDRANT_UPSERT_AT = 2026-07-25T06:00:25Z
LAST_SUCCESSFUL_LEDGER_ACK       = 2026-07-25T06:00:25Z  (vector_status='ready' written back to PG)

NEW_ELIGIBLE_SOURCE_SINCE_LAST_INDEX = 0
ELIGIBLE_BUT_UNINDEXED_COUNT         = 0
ELIGIBLE_CHANGED_AFTER_INDEX_COUNT   = 10   (the STALE_VERSION_IDS of §4.1)

Two writers exist, and only one of them is actually carrying the traffic:

  1. Synchronous HTTP API path (server.py → _sync_vector_entry → vector_store.upsert_document → OpenAI → Qdrant PUT /points). This is the live workhorse: 30 671 point upserts and 20 850 point deletes in the retained Qdrant log.
  2. PG NOTIFY listener (trg_kb_vector_sync → fn_kb_notify_vector_sync → pg_vector_listener.py). Structurally alive — pg_stat_activity shows 2 idle backends holding LISTEN kb_vector_sync; from 172.18.0.5, one per uvicorn worker, since 2026-07-22T14:29:14Z.

The listener has logged zero [pg-vector-sync] events since container start. This is explained, not hand-waved: every kb_documents write in that window is either (a) under a trigger-excluded prefix (registries/** — the 6-hourly Directus sync, 17 documents per cycle) so no NOTIFY is ever emitted, or (b) arrives through the HTTP API which has already upserted the vector synchronously. The listener is idle because it has had nothing to do, not because it is dead. It is nonetheless unproven under load — no measurement in this mission exercised it, and none could without a PG write, which the hard locks forbid. Recorded as LISTENER_FUNCTIONALLY_UNPROVEN rather than claimed as PASS.

5.2 iu_core_iu_chunks — IDLE BY DESIGN

VECTOR_SYNC_PIPELINE_MODE        = MANUAL_ONLY
VECTOR_SYNC_GATE_ENABLED         = NO
  iu_core.vector_sync_enabled = false   (last changed 2026-05-24T11:06:53.928989Z)

LAST_SUCCESSFUL_INDEX_AT         = 2026-05-24T11:06:53Z
LAST_SUCCESSFUL_EMBEDDING_AT     = 2026-05-24T11:06:53Z
LAST_SUCCESSFUL_QDRANT_UPSERT_AT = 2026-05-24T11:06:53Z
LAST_SUCCESSFUL_LEDGER_ACK       = 2026-05-24T11:06:53Z

NEW_ELIGIBLE_SOURCE_SINCE_LAST_INDEX = 0
ELIGIBLE_BUT_UNINDEXED_COUNT         = 0
ELIGIBLE_CHANGED_AFTER_INDEX_COUNT   = 0

Classification per §6:

NO_NEW_ELIGIBLE_SOURCE       = YES
PIPELINE_NOT_EXPECTED_TO_RUN = YES
→ IDLE BY DESIGN   (not MANUAL_ONLY BACKLOG, not STALLED, not FALSE GREEN)

Three independent proofs that no writer is scheduled or deployed:

  • No scheduler entry. All 103 root crontab lines, the incomex crontab, /etc/cron.d, and all 23 systemd timers were enumerated. Not one references vector_sync, iu-cutter, apply_iu_set or iu_core_iu_chunks.
  • No deployed writer binary. vector_sync.py exists nowhere on the host or in any container. cutter_agent trees exist only under /opt/incomex/dot/iu-cutter* (staging snapshots) and /tmp/iu_onboard_9000x (the 9000x driver's scratch copy). The 2400x and 9000x runs were one-shot drivers, exactly as their macros describe.
  • No writes at the engine. In 43 days of retained Qdrant log, iu_core_iu_chunks received zero point writes. Its entire traffic is 43 snapshot create/delete cycles (the daily qdrant-backup.sh) plus this mission's 2 read-only scrolls.

The three-layer boundary enforcement remains intact and the registry shows 0 drift across all 149 indexed rows. Nothing is stalled here; the gate is shut on purpose and there is no work waiting.

6. Why indexing stopped after 2026-05-24

It did not stop. Two different collections were conflated in the premise.

  • iu_core_iu_chunks last indexed 2026-05-24T11:06:53Z because the iu-core-9000x/qdrant_onboarding driver finished its 88-point batch, closed the gate in its finally block, and was never intended to run again. The dot_config gate row and the last registry row share the same timestamp to the millisecond — the closing of the gate is the last event. No failure occurred.
  • production_documents never stopped at all. It indexed today at 06:00:25Z and has been indexing continuously; it grew from 19 684 points on 2026-07-11 to 20 092 now.

The date 2026-05-24 marks the completion of a one-shot corpus onboarding, not the onset of an outage. Reconciling the historical figures in the mission brief: indexed = 149 and dryrun = 3 are exactly what iu_vector_sync_point holds today, unchanged — 61 rows from iu_core_2400x_full_reindex (2026-05-23) plus 88 from iu-core-9000x/qdrant_onboarding (2026-05-24).

7. Eligible backlog

production_documents  ELIGIBLE_BUT_UNINDEXED_COUNT = 0
iu_core_iu_chunks     ELIGIBLE_BUT_UNINDEXED_COUNT = 0
                      TOTAL BACKLOG                = 0

There is no embedding backlog anywhere. Active-document vector status across all 9 933 live kb_documents rows:

vector_status count assessment
ready 9 924 correct
error 4 all four DO have complete, exact vectors in Qdrant (verified individually: 1, 2, 1 and 4 chunks, all byte-exact against the live chunker) — the status field is stale, dating from failed attempts on 2026-05-31 → 2026-06-07 that later succeeded. A reporting defect, not a missing vector.
none 3 empty/folder documents — correctly non-vectorizable
(null) 1 inline-777b3e0d-…, 19-char body, indexed anyway via the API path
pending 1 non-eligible by the live contract

8. OpenAI key / quota / billing diagnosis

OPENAI_API_KEY_PRESENT = YES
OPENAI_API_KEY_SOURCE  = /etc/incomex/agent-data.env  (root:root 0600)
                         → compose env_file → incomex-agent-data process env
EMBEDDING_MODEL        = text-embedding-3-small
                         (QDRANT_EMBED_MODEL unset → live code default, vector_store.py:306)
DIMENSION              = 1536   — matches BOTH collection configs
IU registry            = openai:text-embedding-3-small / 1536 / Cosine   — consistent

No key value, prefix, suffix or fingerprint was read, printed or logged at any point. Only the variable's presence and length were observed.

Log evidence — every error class from §8, full container lifetime:

Pattern Count First / last
insufficient_quota 0 —
invalid_api_key 0 —
rate_limit_exceeded / RateLimit 0 —
billing_hard_limit 0 —
credit balance 0 —
model_not_found 0 —
dimension mismatch 0 —
AuthenticationError / APIError 0 —
Vector sync failed / Vector synchronization failed 0 —

Recent success is proven, so the canary was NOT run. §8 authorises exactly one canary conditionally — "if no successful embedding request is proven in the last seven days". That condition is not met, so the authorisation does not activate and no external API call was made. Production evidence is stronger than a canary would have been:

  • 13 embeddings at 2026-07-25T03:27–03:33Z. Three reports were published via tool=upload_document (status 200) and now hold 5 + 4 + 4 chunks in Qdrant, each requiring a live OpenAI call — upsert_document embeds every chunk unconditionally, with no digest short-circuit.
  • 17 embeddings at 2026-07-25T06:00:25Z. The 6-hourly Directus registries sync, matching PUT /collections/production_documents/points in the Qdrant log to the same second.
  • /health reports embed_calls: 17, embed_tokens: 1298 on the live store instance, and services.openai.status = "ok".
  • Vectors verified non-degenerate: sampled points from both collections return dim = 1536, L2 norm = 1.000000, 1536/1536 non-zero components. Real embeddings, not placeholders.
OPENAI_EMBEDDING_CANARY            = NOT_RUN — recent success proven (30 successful
                                     embeddings on 2026-07-25, the day of this mission)
OPENAI_API_BILLING_BLOCKER_CURRENT = NO

Per §8, ChatGPT subscription status was treated as unrelated to API billing and played no part in this conclusion.

9. Historical 47-orphan reconciliation

All 47 document IDs from the OGV-0 report of 2026-05-03 were re-checked individually against live Qdrant and live PG — not sampled.

HISTORICAL_47_STILL_PRESENT = 0
HISTORICAL_47_REMOVED       = 47
NEW_ORPHANS_SINCE_2026_05_03 = 2
CURRENT_TOTAL_ORPHANS        = 2

All 47 remain in PG as soft-deleted rows carrying vector_status='deleted', and all 47 hold zero Qdrant chunks. The historical leak was fully cleaned and has not recurred through its original mechanism.

The 6 historical "ghosts" are likewise unchanged and remain correct behaviour: empty-body folder documents and one .tmpl fixture, all still excluded by the live contract.

10. Soft-delete prevention status

LISTENER_CHECKS_DELETED_AT          = YES
TRIGGER_EMITS_DELETE_ON_SOFT_DELETE = YES (but see the ordering defect below)
DELETE_API_ORDER                    = Qdrant delete FIRST, then PG soft-delete
DELETE_CAN_BE_FOLLOWED_BY_REUPSERT  = YES   ← still true, by a different path

HISTORICAL_ROOT_CAUSE_FIXED         = YES

Both 2026-05-03 P0 fixes are deployed and correct.

pg_vector_listener.py:76 (file mtime 2026-05-03):

if doc_data.get("deleted_at") is not None or doc_data.get("vector_status") == "deleted":
    store.delete_document(doc_id or key)   # never resurrect a soft-deleted doc
    return

fn_kb_notify_vector_sync (live PG definition) emits a semantic DELETE when deleted_at transitions NULL → NOT NULL, plus a defence-in-depth guard that refuses to emit an upsert for any already-deleted row.

But two gaps remain, and together they produced both current orphans.

Gap A — filter-before-delete ordering in the trigger

The three prefix filters execute before the OGV-P0 soft-delete block:

1. operations/tasks/comments/**  → RETURN early      ← both orphan classes exit here
2. registries/**                 → RETURN early
3. empty key                     → RETURN early
--- only now: OGV-P0 soft-delete → emit semantic DELETE
4. body length < 10              → RETURN early
--- NOTIFY

For any document under those two prefixes, a soft-delete returns at step 1 or 2 and never emits a DELETE. Both orphans live under exactly those prefixes. Because the API upsert path indexes those prefixes anyway (§4.1, the 543 indexed-but-not-eligible documents), such vectors can be created but can never be retracted by an event. This is why the orphans are permanent and will not self-heal.

Gap B — no deleted_at re-check in _run_reindex (the live resurrection path)

server.py _run_reindex() re-reads each document and upserts it, guarding only on data is None:

data = pg_store.get_doc(KB_COLLECTION, _fs_key(doc_id))
if data is None:
    details.append({"document_id": doc_id, "status": "not_found"}); continue
#  ← no deleted_at check here
result = store.upsert_document(...)
pg_store.update_doc(KB_COLLECTION, _fs_key(doc_id), {"vector_status": "ready"})

_run_audit builds ghost_ids from documents active at time T₀. _run_reindex then re-reads and upserts at T₁ > T₀. Any document soft-deleted inside (T₀, T₁] is resurrected in Qdrant and stamped vector_status='ready' in PG — bypassing both P0 fixes, because neither the trigger nor the listener is on this code path. The sibling loop inside /kb/audit-sync at line ~2273 does check deleted_at; _run_reindex does not. The asymmetry is the bug.

11. Why no alert fired

NO_ALERT_ROOT_CAUSE = MONITOR_BROKEN
                      (+ METRIC_ONLY_COVERS_REGISTERED_ROWS for the IU surface)

11.1 Monitoring inventory — measured, not assumed

Surface Schedule Metric measured Threshold Last run Current result
dot-vector-audit daily 04:30 orphans + ghosts report-only, no alert routing 2026-07-25 04:30 FAIL (connection refused) — 15 consecutive
dot-sync-orphan-scan daily 05:00 AgentData ↔ Directus report-only 2026-07-25 05:00 19 findings — does not look at Qdrant at all
dot-kb-verify.sh daily 21:00 Check 3 "PG vs Qdrant sync gap" none 2026-07-24 21:00 prints only PG active docs: 9930 — never queries Qdrant
dot-trigger-guard daily 03:15 Directus event triggers ALL CLEAR 2026-07-25 03:15 does not cover trg_kb_vector_sync on incomex_metadata
uptime-kuma "Qdrant Health" 60 s keyword on /api/health up/down live up — liveness of the process only, no drift/orphan/backlog metric
watchdog-monitor.sh hourly integrity/runner — 2026-07-25 zero references to vector/qdrant/embed/orphan
v_iu_vector_sync_status on demand drift over registered rows — — drifted = 0 — structurally blind (§11.3)

No monitor anywhere measures orphan count, eligible backlog, last-successful-index age, embedding health, or listener heartbeat against Qdrant.

11.2 The primary cause — a deploy silently reverted the fix

dot-vector-audit runs daily and has failed every run for 15 days:

outcome sequence (whole log, oldest → newest):  28 FAIL · 15 OK · 1 FAIL · 52 OK · 15 FAIL ←now
last successful run targeted   http://172.18.0.5:8000   (the container IP)
current runs target            http://localhost:8000    → connection refused → exit 1
agent-data HostConfig.PortBindings = {}   — the container publishes NO host port

The proof of the revert is exact:

md5 /opt/incomex/dot/bin/dot-vector-audit                              = 1de50afe16022904caac9bc37b72c174
md5 /opt/incomex/backups/track-a-residual-20260503/…audit.before       = 1de50afe16022904caac9bc37b72c174
md5 /opt/incomex/docker/nuxt-repo/dot/bin/dot-vector-audit             = 1de50afe16022904caac9bc37b72c174
md5 /opt/incomex/docker/nuxt-repo/web-rp-current/dot/bin/…             = 1de50afe16022904caac9bc37b72c174

file mtime                = 2026-07-11 04:47
git status                =  M dot/bin/dot-vector-audit
git HEAD (commit 6cdaaae "fix dot-vector-audit URL + report-only cron") STILL CONTAINS THE FIX

The live file is byte-identical to its own pre-fix backup. A deploy on 2026-07-11 04:47 overwrote the working tree with the stale nuxt-repo copy, discarding the Track-A-residual fix while git HEAD continues to record it as applied. This is the same 2026-07-11 revert event already recorded for the nginx/OAuth regression.

The revert removed two things at once:

-_AGENT_DATA_IP="$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' incomex-agent-data …)"
-LOCAL_URL="${AGENT_DATA_LOCAL_URL:-http://${_AGENT_DATA_IP:-localhost}:8000}"
+LOCAL_URL="http://localhost:8000"
-API_KEY="${AGENT_DATA_API_KEY:-${ENV_API_KEY:-test-key-local}}"     # ENV_API_KEY read from docker/.env
+API_KEY="${AGENT_DATA_API_KEY:-test-key-local}"

So even if the endpoint were reachable, the audit would now authenticate with the literal string test-key-local and be rejected. Two independent breakages, one revert.

set -euo pipefail + exit 1 means cron discards the failure silently. There is no alert routing, no heartbeat, and no Cron Heartbeat push tied to this job — so 15 consecutive failures produced exactly zero signal.

11.3 What the blindness cost, precisely

The last successful run already reported the first orphan:

Documents: 9837 · Vectors: 19684 · Orphans: 1 · Ghosts: 0 · Status: needs_cleanup
  Orphan vector doc IDs (1):
    - operations/tasks/comments/comment-1842

That finding was true, visible, and never acted on. The monitor then went blind on 2026-07-11, and the second orphan (registries/workflow_step_relations/wsr-87, 2026-07-22) was created inside the blind window and has never been reported by any system until this mission.

v_iu_vector_sync_status cannot detect unregistered sources — confirmed from the live definition:

SELECT point_key, …, indexed_digest IS NOT NULL AND indexed_digest IS DISTINCT FROM content_digest AS drifted, …
  FROM iu_vector_sync_point p;

It selects from iu_vector_sync_point alone, with no join to unit_version or v_ui_iu_three_axis_envelope. A source that was never registered is not a row, and a view cannot report a row that does not exist. Its drifted = 0 therefore means "nothing registered has drifted", never "nothing eligible is missing". Per §6, drift = 0 was not equated with pipeline health anywhere in this report.

12. Stage 2 findings

Stage 2 triggered on CURRENT_TOTAL_ORPHANS > 0 and ran automatically without Owner confirmation, scoped to production_documents and the delete/reindex path.

Finding 1 — REINDEX_RACE_RESURRECTION (ongoing, unfixed)

Field Evidence
symptom Vector survives a soft-delete; PG stamped vector_status='ready' after deleted_at was set
affected count 2 (100 % of current orphans)
first timestamp 2026-05-27T10:19:57.000865Z
latest timestamp 2026-07-22T11:17:22.065463Z
source identity operations/tasks/comments/comment-1842; registries/workflow_step_relations/wsr-87
expected lifecycle soft-delete → vector removed → vector_status='deleted'
writer / delete path DELETE /documents/{id} deletes the vector first, then soft-deletes in PG (server.py:1748)
trigger/listener behavior Both correct in isolation; neither is on the offending path
embedding result success — a real 1536-d vector was written
Qdrant result 1 chunk present for a document that PG says is deleted
ledger result vector_status='ready' — PG asserts healthy
code evidence _run_reindex (server.py ≈2487) guards only data is None; the parallel /kb/audit-sync heal loop at ≈2273 does check deleted_at
database evidence see the forensic reconstruction below
log evidence UNAVAILABLE — the agent-data container was recreated 2026-07-22T14:27Z; both events predate the retained log
ongoing or historical ONGOING — no code change since; the race is reproducible today
root cause _run_reindex re-upserts a document soft-deleted between the audit snapshot (T₀) and the re-read (T₁)
minimal prevention point one if data.get("deleted_at") is not None: continue inside _run_reindex, mirroring the guard the sibling loop already has

Forensic reconstruction from kb_documents_history. fn_kb_snapshot stores the OLD image (OLD.data) on BEFORE UPDATE, so each row records the state before the write that created it. For registries__workflow_step_relations__wsr-87:

history id at OLD image
125571 11:17:21.258557 rev 1, deleted_at null, vector_status pending
125572 11:17:21.337623 rev 2, deleted_at null, pending
125573 11:17:22.022447 rev 2, deleted_at null, ready
125574 11:17:22.065463 rev 3, deleted_at set, vector_status deleted

Live row today: rev 3, deleted_at = 11:17:22.021502, vector_status = ready, updated_at = 11:17:22.021502.

The write at 11:17:22.065463 therefore moved the row out of the fully-deleted state and back to ready — 40 ms after the delete committed. Because pg_store.update_doc uses an atomic JSONB merge (data = data || %s), the applied patch is recoverable exactly: it changed vector_status only, leaving updated_at, revision and deleted_at untouched.

Exactly two call sites in the entire codebase emit that patch shape — {"vector_status": "ready"} with nothing else — and both are inside _run_reindex/the heal loop (server.py ≈2308 and ≈2530). Every other writer includes updated_at (_sync_vector_entry), revision (the delete API), or a different value. No ordering of any other pair of writers can produce the observed row. The identical signature appears on comment-1842 (history ids 62052–62053, 2026-05-27), so the mechanism is stable across two months and two unrelated documents.

dot-vector-audit-schedule — which would install a --heal job — is not scheduled on this VPS (0 crontab matches; it writes a macOS launchd plist). The cron audit runs report-only. The two resurrections were therefore caused by ad-hoc heal/reindex invocations, consistent with dot-start-all / dot-ai-start, both of which call these endpoints, and with the C2B1-HYB-S2 canary window on 2026-07-22 (the wsr-87 payload literally carries the label REL C2B1HYBS2-20260722T111657Z). The exact invoking process is UNPROVEN — the logs are gone.

Finding 2 — LEDGER_COVERAGE_BLIND_SPOT / filter-before-delete (why it is permanent)

Both affected documents live under prefixes that fn_kb_notify_vector_sync filters out before reaching its OGV-P0 soft-delete branch (§10, Gap A). No subsequent event can ever emit a corrective DELETE for them. The resurrection is not merely uncorrected — it is uncorrectable by the event system, and will persist until a repair mission removes the points explicitly.

Finding 3 — MONITOR_BROKEN (§11.2)

Fully evidenced above: a 2026-07-11 deploy reverted dot-vector-audit to its pre-fix state, breaking both the endpoint resolution and the API key, and 15 consecutive silent failures followed.

Causes explicitly ruled out

QDRANT_ENGINE_FAILURE          RULED OUT — 0 errors / 0 warns in 423 005 lines, 0 restarts
OPENAI_API_FAILURE             RULED OUT — 30 successful embeddings today, 0 errors of any class
PIPELINE_NOT_SCHEDULED         TRUE for iu_core_iu_chunks, but BY DESIGN with 0 backlog
PIPELINE_STALLED               RULED OUT — production_documents indexed today; IU has no work waiting
GATE_DISABLED_BY_DESIGN        CONFIRMED for iu_core_iu_chunks (correct state)
ELIGIBILITY_RULE_EXCLUDED_SRC  CONFIRMED benign — the 5 unindexed IUs all have body_len = 0
SOFT_DELETE_RESURRECTION       Original 2026-05-03 mechanism FIXED; a NEW mechanism is live

No repair or cleanup was performed.

13. Minimal remediation order

Ordered by dependency. Nothing below was executed.

# Action Why this order
1 Restore dot-vector-audit from git HEAD (git checkout -- dot/bin/dot-vector-audit, commit 6cdaaae). Restores both the container-IP resolver and the real API key. Repairing data while blind risks silent recurrence. Restore sight first. This is a pure revert of an accidental clobber — no new code.
2 Investigate why the 2026-07-11 deploy overwrote /opt/incomex/dot/bin/ from docker/nuxt-repo/. Until this is understood, fix #1 will be reverted again by the next deploy. Feeds ROOT-CRON-WRITABLE-SCRIPT-R0 (the same directory is incomex-writable while root cron executes it).
3 Patch _run_reindex — add the deleted_at guard the sibling heal loop already has. Closes the live resurrection path. Without it, any cleanup can be undone by the next reindex.
4 Move the three prefix filters in fn_kb_notify_vector_sync to AFTER the OGV-P0 soft-delete branch. Makes the two affected classes self-healing. Note this makes soft-deletes of registries/** emit DELETE — verify against the Directus sync's expectations before applying.
5 Only then remove the 2 orphan points, the 9 empty-document_id points, and re-index the 10 stale documents. Cleanup is safe only once #3 and #4 hold; otherwise it hides evidence and the leak recurs — the same conclusion OGV-0 reached in 2026-05.
6 Reconcile the 4 vector_status='error' rows that actually hold correct vectors. Cosmetic ledger accuracy; no retrieval impact.

14. Minimal monitoring recommendation (design only — NOT implemented)

Per §12: no new framework, one existing cron plus one compact read-only verifier.

Reuse the existing 04:30 dot-vector-audit slot. Once restored (remediation #1) it already yields orphan and ghost counts. Add a single read-only SQL verifier alongside it — the whole design is five queries, no new service, no new schedule:

Required metric Cheapest existing surface
ORPHAN_COUNT SELECT count(*) FROM kb_documents WHERE data->>'deleted_at' IS NOT NULL AND data->>'vector_status'='ready' — exact, zero false positives on today's data (2/2), and needs no Qdrant scroll at all
LAST_SUCCESSFUL_INDEX_AGE now() - max(updated_at) over kb_documents where vector_status='ready'; for IU, max(updated_at) of iu_vector_sync_point
ELIGIBLE_BUT_UNINDEXED_COUNT count of active rows passing the live eligibility contract whose vector_status <> 'ready'; for IU, enacted-with-body units LEFT JOIN iu_vector_sync_point — the join v_iu_vector_sync_status lacks (§11.3)
EMBEDDING_CANARY_STATUS /health → data_integrity.embed_calls delta between runs; > 0 proves live embedding without spending an API call
VECTOR_LISTENER_HEARTBEAT SELECT count(*) FROM pg_stat_activity WHERE query LIKE 'LISTEN kb_vector_sync%' — expect 1 per uvicorn worker (currently 2)

Two properties the current setup lacks and that this design must have:

  • Fail loudly. dot-vector-audit exits 1 into a cron log nobody reads. Route the result to the existing uptime-kuma push monitor family (Cron Heartbeat, PG Backup * already use it), so a missing run is itself an alert. A monitor that can die silently is not a monitor.
  • Assert its own target. The failure mode here was a monitor pointed at an endpoint that does not exist. It must fail as MONITOR_MISCONFIGURED, distinctly from no issues found.

15. No-mutation proof

Every value below was re-measured at session exit and compared against session entry.

QDRANT_POINT_MUTATION = 0
    production_documents  entry 20 092 → exit 20 092   (status green, both)
    iu_core_iu_chunks     entry    149 → exit    149
    zero PUT/POST /points, zero points/delete, zero collection create/delete,
    zero snapshot created by this mission

POSTGRESQL_MUTATION   = 0
    kb_documents            12 114 rows  (unchanged)
    iu_vector_sync_point       152 rows  (unchanged)
    iu_core.vector_sync_enabled = false  (unchanged, still 2026-05-24T11:06:53Z)
    every statement issued was SELECT / pg_get_viewdef / information_schema;
    the psycopg2 sessions ran with set_session(readonly=True)

DIRECTUS_MUTATION     = 0    no item, permission, policy, role or flow touched
CODE_MUTATION         = 0    md5 unchanged: server.py 380c9bdb…, vector_store.py 7eb74143…,
                             pg_vector_listener.py 988cd374…;
                             /opt/incomex/dot/bin/dot-vector-audit still 1de50afe… (mtime 2026-07-11)
CRON_MUTATION         = 0    103 root crontab lines, unchanged; no systemd unit touched
CONTAINER_RESTART     = 0    all 10 core containers: CIDs identical, RestartCount unchanged,
                             no StartedAt falls within this session
MAC_FILES_MUTATED     = 0    work confined to the session scratchpad

Gates NOT opened, endpoints NOT called: iu_core.vector_sync_enabled left false; /kb/cleanup-orphans, /kb/reindex-missing and /kb/audit-sync were never invoked — the orphan comparison in §4 is this mission's own independent read-only scroll and PG read, deliberately not the service's audit endpoint.

Two disclosures, made rather than omitted:

  1. 3 × HTTP 404 against Qdrant at 05:55Z were mine — read-only probes of /collections/aliases and /collections/*/index, wrong paths for this Qdrant version. No state change. Counted in §2.
  2. One empty scratch file was accidentally created inside incomex-agent-data:/tmp/.c47.py by a malformed shell redirect, and removed immediately (verified absent; /tmp listing clean). Zero bytes, never executed, outside /app. CODE_MUTATION = 0 still holds, but the event is recorded rather than silently dropped.

The one authorised external call was NOT made. §8's single OpenAI canary is conditional on no proven recent success; success was proven, so no external API request was issued by this mission at all.

Method note. All work was performed over SSH on the VPS. OPENAI_API_KEY and QDRANT_API_KEY were never read, printed, passed on a command line, or written to any file — scripts were fed to docker exec -i … python3 - on stdin and read the keys from the process environment. No vector values, payload bodies, tokens or credentials appear in this report.

16. Exact next gate

HOLD_QDRANT_ORPHAN_AND_MONITOR_BLIND

NEXT GATE: VECTOR_AUDIT_MONITOR_RESTORE_R0

The gate is the monitor, not the orphans. Two orphan points are a small, bounded retrieval-quality defect. Fourteen days with no working vector-integrity monitor is the systemic risk, and it is the precondition for trusting any subsequent cleanup.

The prior gate OWNER_DIRECTUS_LOGIN_ACCEPTANCE is unaffected by this mission and remains open.

Registered, not executed:

VECTOR-AUDIT-MONITOR-RESTORE-R0     restore dot-vector-audit from git HEAD; add push-based
                                    heartbeat so a dead monitor is itself an alert
DEPLOY-CLOBBER-INVESTIGATION-R0     why the 2026-07-11 deploy reverted /opt/incomex/dot/bin/
                                    (converges with ROOT-CRON-WRITABLE-SCRIPT-R0)
REINDEX-DELETED-GUARD-R0            add the deleted_at guard to _run_reindex
KB-TRIGGER-FILTER-ORDER-R0          move prefix filters after the OGV-P0 soft-delete branch
QDRANT-ORPHAN-CLEANUP-R0            remove 2 orphans + 9 empty-document_id points, re-index
                                    10 stale documents — ONLY after the four above

Do not clean or repair until the next directive.

Landmines recorded

  • Qdrant chunk metadata is nested. production_documents payloads carry chunk_index / total_chunks under payload.metadata, not at the payload root. A root-level required-field check reports all 20 092 points malformed. The real count is 9.
  • indexed_vectors_count > points_count is normal, not duplication — it counts vectors still resident in segments awaiting vacuum. Conversely indexed_vectors_count = 0 on a small collection is normal too: below indexing_threshold (10 000) Qdrant builds no HNSW graph.
  • A naive "is the stored chunk a prefix of the body" staleness test yields ~40 % false positives. _split_text breaks on paragraph/sentence/word boundaries with a 400-char overlap. The only sound test is to import the live _split_text from the running container and compare chunk-for-chunk: 3 920 false positives collapsed to 10 real ones.
  • fn_kb_snapshot stores OLD.data, not NEW.data. Reading kb_documents_history as a post-image timeline inverts every conclusion about write ordering.
  • pg_store.update_doc is an atomic JSONB merge (data = data || patch), not read-modify-write. This is what makes the applied patch shape forensically recoverable — and it is why the guilty writer could be narrowed to two call sites out of seven.
  • git status is a monitor. The live dot-vector-audit was byte-identical to its own pre-fix backup while git HEAD still held the fix, showing as a plain M. A file can be silently reverted by a deploy with no error anywhere; only comparing the working tree to HEAD reveals it.
  • docker logs --since/--until parses in host local time (CEST), while PG timestamps are UTC. A two-hour offset silently returns the wrong window and looks like "no events".
  • A cron job that exit 1s is not a monitor. 15 consecutive daily failures produced zero signal. Absence of a run must itself alert, or the monitor's death is indistinguishable from good health.