Incomex AI Portal
KB-F64D rev 2

P3D — Vector Search Reliability Hardening — Implementation Prompt rev2

14 min read Revision 2
p3dvector-searchhardeningimplementationpromptrev2production

P3D — Vector Search Reliability Hardening — Implementation Prompt rev2

Date: 2026-05-11 Status: PROMPT rev2 — GPT patched, APPROVED_FOR_AGENT_DISPATCH_AFTER_USER_CONFIRMATION Design ref: knowledge/dev/laws/dieu44-trien-khai/design/p3d-vector-search-reliability-hardening-design.md Review ref: knowledge/dev/laws/dieu44-trien-khai/reviews/gpt-review-vector-search-hardening-design-and-prompt-rev1-2026-05-11.md Hard boundary: NO Qdrant mutation, NO reindex, NO IU vector, NO TAC/IU migration, NO DB writes

0. Objective

Harden the already-passing search boost patch into production-grade infrastructure that will support both current legacy KB search and future IU vector search.

This pack must solve, in one coherent production pass:

  1. Deployment durability: search boost survives agent-data recreate/rebuild.
  2. Canary/regression: repeatable search tests run after restart/rebuild/deploy.
  3. Conservative recency tie-break: tiny boost only when exact/path/title boost already exists.
  4. Audit warning cleanup: health/audit should not warn on intentionally non-vectorizable empty/folder/short docs.
  5. Unified search contract: same rerank contract will be reused by future IU vector; IU-specific chunking remains deferred.

1. Hard boundaries

  • No Qdrant point delete/upsert.
  • No Qdrant collection replacement.
  • No new Qdrant collection.
  • No bulk reindex.
  • No auto-heal.
  • No /kb/reindex.
  • No /kb/reindex-missing.
  • No /kb/cleanup-orphans write mode.
  • No PG schema mutation.
  • No DB INSERT/UPDATE/DELETE.
  • Do not update kb_documents rows, including test_empty.md.tmpl; classify in code instead.
  • No trigger/function/index/permission changes.
  • No TAC/IU migration.
  • No IU vector implementation.
  • No Pack 1 DDL.
  • Do not recreate the whole stack; only agent-data may be rebuilt/recreated.
  • Do not touch Qdrant container except read-only point-count checks.

Allowed:

  • Back up and patch Agent Data code.
  • Back up and minimally patch docker-compose only to add a safe build: directive for agent-data, if preflight confirms it is correct.
  • Create /opt/incomex/dot/bin/dot-search-canary.
  • Rebuild/recreate only agent-data to prove durability.
  • Upload KB report.

2. Preflight

set -uo pipefail
TS=$(date +%Y%m%d-%H%M%S)
LOG="/tmp/p3d-vector-hardening-${TS}.log"
exec > >(tee -a "$LOG") 2>&1

echo "=== P3D VECTOR SEARCH HARDENING PREFLIGHT $TS ==="

docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}'

AGENT_DATA=$(docker ps --filter "name=agent-data" --format '{{.Names}}' | head -1)
QDRANT=$(docker ps --filter "name=qdrant" --format '{{.Names}}' | head -1)
COMPOSE_DIR="/opt/incomex/docker"
REPO_DIR="/opt/incomex/docker/agent-data-repo"
COMPOSE_FILE="$COMPOSE_DIR/docker-compose.yml"

echo "AGENT_DATA=$AGENT_DATA"
echo "QDRANT=$QDRANT"
echo "COMPOSE_FILE=$COMPOSE_FILE"
echo "REPO_DIR=$REPO_DIR"

[ -n "$AGENT_DATA" ] || { echo "BLOCKED: Agent Data container not found"; exit 2; }
[ -n "$QDRANT" ] || { echo "BLOCKED: Qdrant container not found"; exit 2; }
[ -f "$COMPOSE_FILE" ] || { echo "BLOCKED: compose file not found"; exit 2; }
[ -d "$REPO_DIR" ] || { echo "BLOCKED: repo dir not found"; exit 2; }

cd "$REPO_DIR"
git status --short
git log --oneline -5
[ -f Dockerfile ] || { echo "BLOCKED: Dockerfile missing"; exit 2; }

echo "--- Dockerfile COPY evidence ---"
grep -nE 'COPY|ADD' Dockerfile || true

echo "--- Compose agent-data stanza preview ---"
grep -nA60 -B5 'agent-data:' "$COMPOSE_FILE" | head -90

echo "--- Health before ---"
curl -s http://localhost:8000/health | python3 -m json.tool || true

echo "--- Qdrant points before ---"
docker exec "$QDRANT" sh -lc "curl -s http://localhost:6333/collections/production_documents" | tee /tmp/p3d-qdrant-before-${TS}.json
python3 - <<'PY'
import json
p='/tmp/p3d-qdrant-before-' + __import__('os').environ.get('TS','') + '.json'
PY

If repo is dirty in files you need to edit, inspect and report. Do not overwrite unrelated changes.

Record current Qdrant points_count manually from the JSON if script parsing fails.

3. Backup

BACKUP_DIR="/tmp/p3d-vector-hardening-backup-${TS}"
mkdir -p "$BACKUP_DIR"
cp "$REPO_DIR/agent_data/vector_store.py" "$BACKUP_DIR/vector_store.py.before"
cp "$REPO_DIR/agent_data/server.py" "$BACKUP_DIR/server.py.before"
cp "$COMPOSE_FILE" "$BACKUP_DIR/docker-compose.yml.before"
[ -f /opt/incomex/dot/bin/dot-search-canary ] && cp /opt/incomex/dot/bin/dot-search-canary "$BACKUP_DIR/dot-search-canary.before" || true

echo "BACKUP_DIR=$BACKUP_DIR"

4. Implementation tasks

4.1 Deployment durability: compose build for agent-data only

Goal: make patch survive container recreate/rebuild. Current docker cp hotfix is not enough.

Steps:

  1. Inspect compose service definition.
  2. If agent-data lacks build:, add only this under the agent-data service while preserving existing image: agent-data-local:latest and environment/volumes/ports:
    build:
      context: ./agent-data-repo
      dockerfile: Dockerfile
  1. Do not edit other services.
  2. Do not change .env.
  3. Do not change Qdrant service.

Validation before rebuild:

cd "$COMPOSE_DIR"
docker compose config >/tmp/p3d-compose-config-${TS}.yml

If compose config fails, restore compose backup and report BLOCKED.

4.2 Canary script

Create or replace:

/opt/incomex/dot/bin/dot-search-canary

The canary must:

  • call the actual Agent Data search endpoint/tool used by searchKnowledge;
  • first capture and print the response shape for one query if parser is uncertain;
  • test eight queries T1–T8;
  • report rank and PASS/FAIL;
  • exit non-zero if any hard test fails;
  • not mutate anything.

Required T1–T8:

T1 query="GPT Review P3D Step 1 Re-authored Spec Pack 1 Directive" expected_slug="gpt-review-p3d-step1-reauthored-spec-and-pack1-directive" max_rank=2
T2 query="p3d-pack1-readonly-inventory-prompt revision 2" expected_slug="p3d-pack1-readonly-inventory-prompt" max_rank=1
T3 query="gpt-directive-agent-run-step1-checkpoint-and-pack1-inventory-readonly" expected_slug="gpt-directive-agent-run-step1-checkpoint-and-pack1-inventory-readonly" max_rank=1
T4 query="vector search freshness root cause" expected_slug="p3d-vector-search-freshness-audit" max_rank=3
T5 query="operating rules SSOT" expected_slug="operating-rules" max_rank=3
T6 query="vector search unified contract SSOT" expected_slug="vector-search" max_rank=3
T7 query="P3D information unit text-as-code requirements spec" expected_slug="p3d-information-unit-text-as-code-requirements-spec" max_rank=1
T8 query="hiến pháp constitution v4" expected_slug="constitution" max_rank=3

Important: parser must be adapted to actual response. Do not assume the template JSON shape is correct.

Run canary before rebuild to establish current PASS:

chmod +x /opt/incomex/dot/bin/dot-search-canary
/opt/incomex/dot/bin/dot-search-canary | tee /tmp/p3d-search-canary-before-rebuild-${TS}.log

4.3 Conservative recency tie-break

Patch agent_data/vector_store.py only if metadata contains usable timestamp fields or current result candidates include created/updated timestamps.

Rules:

  • Recency boost max +0.005.
  • Only apply when existing path/title/tag/directory boost is already > 0.
  • Decay to zero over 30 days.
  • If timestamps are unavailable in candidate metadata, do not mutate Qdrant payload and do not query PG per candidate. Record recency_tiebreak=SKIPPED_METADATA_UNAVAILABLE.
  • Feature flag in code: SEARCH_RECENCY_TIEBREAK, default true.
  • No compose/env edit.

Do not let recency alter broad semantic queries with no exact/path/title boost.

4.4 Audit warning cleanup — code classification only, no DB writes

Patch Agent Data health/audit logic only if you can clearly identify the code path.

Goal:

  • Empty document_id, directory-like paths, and body length <10 should be classified as non_vectorizable or excluded from ghost warning counts.
  • Empty/folder/short docs should not cause misleading ghost warnings.
  • Ratio threshold should be made meaningful for chunked docs. Prefer improving message/classification over simply raising threshold.

Hard boundary:

  • No DB writes.
  • Do not update vector_status rows.
  • Do not delete/reindex/upsert vectors.
  • /kb/audit-sync {"auto_heal": false} only.

If audit code path is not clear, do not patch. Report audit_cleanup=DEFERRED_CODE_PATH_UNCLEAR.

4.5 Unified KB/IU search contract shape

Minimal code/design-only compatibility:

  • If low-risk, allow rerank function to accept optional collection_name or source parameter with default production_documents.
  • Do not change behavior for legacy KB collection.
  • Do not create IU collection.
  • Do not implement IU vector.
  • Document in report that future IU vector must reuse rerank logic and add payload fields: unit_id, canonical_address, unit_version_id, content_hash.

5. Static validation before rebuild/restart

cd "$REPO_DIR"
python3 -m py_compile agent_data/vector_store.py agent_data/server.py
git diff --stat
git diff -- agent_data/vector_store.py agent_data/server.py "$COMPOSE_FILE" || true
cd "$COMPOSE_DIR"
docker compose config >/tmp/p3d-compose-config-after-${TS}.yml

If any validation fails, rollback files and report FAIL/BLOCKED. Do not restart.

6. Build/recreate durability proof

Only after validation passes:

cd "$COMPOSE_DIR"
docker compose build agent-data

docker compose up -d --force-recreate agent-data
sleep 30
curl -s http://localhost:8000/health | python3 -m json.tool
/opt/incomex/dot/bin/dot-search-canary | tee /tmp/p3d-search-canary-after-recreate-${TS}.log

Only agent-data may be force-recreated. Do not recreate all services.

Verify Qdrant point count unchanged:

docker exec "$QDRANT" sh -lc "curl -s http://localhost:6333/collections/production_documents" | tee /tmp/p3d-qdrant-after-${TS}.json

7. Audit verification

Read-only only:

curl -s -X POST http://localhost:8000/kb/audit-sync \
  -H "Content-Type: application/json" \
  -d '{"auto_heal": false}' | python3 -m json.tool | tee /tmp/p3d-audit-after-${TS}.json

curl -s http://localhost:8000/health | python3 -m json.tool | tee /tmp/p3d-health-after-${TS}.json

Report whether warning improved, remained, or is now meaningful. Do not auto-heal.

8. Rollback

Rollback if:

  • build fails;
  • health fails;
  • canary fails after recreate;
  • Qdrant point count changes;
  • broad semantic search regresses;
  • compose change causes unintended service changes.

Rollback commands:

cp "$BACKUP_DIR/vector_store.py.before" "$REPO_DIR/agent_data/vector_store.py"
cp "$BACKUP_DIR/server.py.before" "$REPO_DIR/agent_data/server.py"
cp "$BACKUP_DIR/docker-compose.yml.before" "$COMPOSE_FILE"
if [ -f "$BACKUP_DIR/dot-search-canary.before" ]; then
  cp "$BACKUP_DIR/dot-search-canary.before" /opt/incomex/dot/bin/dot-search-canary
else
  rm -f /opt/incomex/dot/bin/dot-search-canary
fi
cd "$COMPOSE_DIR"
docker compose build agent-data
docker compose up -d --force-recreate agent-data
sleep 30
curl -s http://localhost:8000/health

9. Git commit

If PASS:

cd "$REPO_DIR"
git status --short
git add agent_data/vector_store.py agent_data/server.py
# If compose file is outside repo, record its checksum and backup path in report.
git commit -m "P3D: harden vector search rerank durability and canary" || true

If compose file is not in the same repo, do not force it into repo. Record exact path, diff, checksum, and backup.

10. Required report

Upload to:

knowledge/dev/laws/dieu44-trien-khai/reports/p3d-vector-search-reliability-hardening-implementation-report.md

Report fields:

phase_status=PASS|FAIL|ROLLED_BACK|BLOCKED
mode=IMPLEMENTATION
search_boost_behavior=PASS|FAIL
production_durability=PASS|FAIL|BLOCKED
compose_build_added=true|false
agent_data_rebuilt=true|false
agent_data_force_recreated=true|false
canary_status=PASS|FAIL|BLOCKED
recency_tiebreak=IMPLEMENTED|SKIPPED_METADATA_UNAVAILABLE|DEFERRED
recency_safe_for_semantic=true|false|n/a
audit_warning_cleanup=IMPLEMENTED|DEFERRED_CODE_PATH_UNCLEAR|FAILED
audit_sync_status_after=<value>
qdrant_points_before=<n>
qdrant_points_after=<n>
qdrant_mutation_performed=false
db_write_performed=false
pg_schema_mutation_performed=false
no_reindex_performed=true
rollback_performed=true|false
health_after=healthy|unhealthy|unknown
unified_search_contract_ready=true|false

Include:

  • compose diff summary;
  • code diff summary;
  • canary before/after recreate results;
  • audit before/after summary;
  • Qdrant point count before/after;
  • git commit hash if created;
  • warnings/deferred items.

11. Final response expected from Agent

vector_search_hardening_status=PASS|FAIL|ROLLED_BACK|BLOCKED
report_path=knowledge/dev/laws/dieu44-trien-khai/reports/p3d-vector-search-reliability-hardening-implementation-report.md
production_durability=PASS|FAIL|BLOCKED
canary_status=PASS|FAIL|BLOCKED
recency_tiebreak=IMPLEMENTED|SKIPPED_METADATA_UNAVAILABLE|DEFERRED
audit_warning_cleanup=IMPLEMENTED|DEFERRED_CODE_PATH_UNCLEAR|FAILED
qdrant_points_before=<n>
qdrant_points_after=<n>
db_write_performed=false
qdrant_mutation_performed=false
rollback_performed=true|false
health_after=<status>

Return only the final response plus critical blocker notes.

Back to Knowledge Hub knowledge/dev/laws/dieu44-trien-khai/prompts/p3d-vector-search-reliability-hardening-implementation-prompt.md