Incomex AI Portal
KB-4E38

dot-iu-cutter v0.1 — Thread Retrieval and User Interaction Design

15 min read Revision 1
dot-iu-cutterdesignretrievaluser-interactionaudience-scopedrev5d

dot-iu-cutter v0.1 — Thread Retrieval and User Interaction Design (D11)

Date: 2026-05-15 Status: DESIGN DRAFT Baseline: rev5d §14 Scope: DESIGN ONLY. v0.1 prepares metadata + hooks; does NOT build external UI/search.

1. Purpose

Define thread-first retrieval as a primary AI/Agent capability — turning Semantic Threads (D9) into a retrieval primitive — and define audience-scoped search as an access-control requirement, not just a filter. Define the Thread Context Pack that AI/Agent receives, retrieval metrics, fallback strategy, search-gap feedback signals, and audience-specific visibility rules.

2. Scope

  • Thread-first retrieval flow
  • Thread resolution from query
  • Thread Context Pack structure
  • Fallback to vector search
  • Retrieval quality metrics + hooks
  • Search-gap feedback signals
  • Audience-scoped search (AI/Agent, employee, partner, customer)
  • Visibility / readiness / publication filters
  • Access-control guardrail (security requirement)
  • Positive recursion via search usage → semantic health

Out of scope: thread object family and lifecycle (D9); health-signal sink (D3); UI build (forbidden in v0.1, rev5d §11, §14.2 boundary).

3. Dependencies

  • rev5d §14 (full), §13.1.4, §13.2 (access-control guardrail)
  • D9 (thread objects)
  • D3 (health signals)
  • Đ24 (label/vocabulary), Đ32 (risk for visibility), Đ33/Đ43 (PG placement), Đ37 (escalation queue), Đ39 (universal_edges traversal)

4. Key Decisions

4.1 Thread-First Retrieval is Primary (Q47, Q50; criterion 42, 45)

AI/Agent default behavior: do not start with raw vector search when structured thread knowledge exists. Vector search is a fallback.

Query (user or agent)
  → Thread Resolution (4.2)
  → Candidate Thread Ranking (4.3)
  → Context Pack Build (4.4)
  → Authority/Recency/Source Ordering (4.5)
  → Missing/Suspect Signal Attach (4.6)
  → Answer with citations (each claim → source unit + revision)
  → Fallback if confidence < threshold or no thread matched → vector search (4.7)

The choice of thread-first vs vector is observable and logged as a retrieval_metric event.

Rationale: rev5d §14.1; acceptance criterion 42, 45.

4.2 Thread Resolution (Q48; criterion 43)

Resolution sources (combined):

Signal Source
Keyword / tsvector match semantic_thread.title, alias, description
Đ24 label match Controlled vocabulary mapping
Embedding similarity Query embedding vs thread centroid embedding
Explicit thread reference User/agent names a thread by ID or canonical address
Authority hint Query refers to a law/design/code → prefer threads anchored there

Resolution returns a ranked candidate set with (thread_id, score, signal_breakdown). Threshold-based decision:

  • Single high-confidence match → use it.
  • Multiple plausible matches → return top-N as candidate threads in the context pack (multi-thread mode) and let consumer decide.
  • No match above threshold → emit missing_thread signal → fallback (4.7).

4.3 Candidate Thread Ranking (Q48)

Re-rank by:

  1. Relevance score (4.2).
  2. Recency of last activity.
  3. Authority (enacted > draft > runtime).
  4. Health status (stale, overbroad, too_narrow, noisy_retrieval penalize).
  5. Coverage of expected lifecycle chain.

4.4 Thread Context Pack (Q49, Q51; criterion 44)

The Context Pack is a pre-assembled, ordered bundle returned to AI/Agent. Minimum fields:

thread_id
thread_title
purpose
canonical_units (ordered list of core units)
supporting_units (related but secondary)
latest_reports (most recent reports/reviews in this domain)
related_design_code_links (if available; via universal_edges)
known_gaps (expected_artifact_missing entries from D9)
suspect_links (low-confidence or flagged memberships)
last_health_status (latest health check result from D3)
authority_or_risk_level
recommended_reading_order
source_revisions (revision of each unit included)
audience_scope_applied (which audience filters ran)
fallback_signal (if any threshold was missed and is being reported)

AI/Agent should NOT read the raw thread. It receives this structured pack, already ordered, scoped, and annotated.

Q51 yes: each included unit carries source, revision, authority, risk_class, and the thread carries last_health_status.

4.5 Authority / Recency / Source Ordering

Within canonical_units, default order:

  1. Lifecycle stage (law → design → code → test/report → incident).
  2. Authority (enacted > draft > runtime).
  3. Recency within the same lifecycle stage.
  4. Canonical address ordering as tie-breaker.

recommended_reading_order is a separate field reflecting this; consumers can override only when explicit.

4.6 Missing / Suspect Signal Attach (Q52; criterion 47)

Each Context Pack carries inline:

  • known_gaps: from expected_artifact_missing (D9 §4.9).
  • suspect_links: memberships with confidence < threshold, status='needs_human', or recent wrong_link signal.

AI/Agent must surface these to the human consumer when answering — answers based on suspect material must be flagged.

4.7 Fallback to Vector Search (Q50; criterion 45)

Fallback triggers:

  1. No thread above resolution threshold (missing_thread).
  2. All candidate threads have last_health_status ∈ {overbroad, too_narrow, stale, noisy_retrieval} above policy threshold (weak_thread).
  3. Consumer explicitly requests raw search (debug / discovery mode).

Fallback flow:

Vector search (Qdrant) → top-K units → audience filter (4.10) → assemble ad-hoc pack
  → log fallback_signal = true → emit search-gap feedback (4.9)

The fallback result is annotated as such — consumer knows it is not thread-resolved.

4.8 Retrieval Quality Metrics (Q53 supporting; criterion 46)

All metrics are instrumentation hooks; v0.1 defines them, implementation follows when the retrieval layer is built.

Metric Meaning
thread_hit_rate % queries resolved by thread (vs fallback)
fallback_vector_rate % queries requiring vector fallback
context_sufficiency_rate % queries where context pack alone sufficed
wrong_thread_rate % queries matched to wrong thread
missing_thread_rate % queries where no thread existed but should have
noisy_context_rate % context packs containing irrelevant units
agent_extra_search_rate % times agent searched outside the pack after receiving it
user_reject_rate % thread suggestions rejected by user

Targets: initial maturity > 80% thread_hit_rate; strategic ambition > 90% for recurring AI/Agent domain queries (rev5d §14.4).

Metrics are PG-backed counters/events (target DB: directus; layer: Não). Classification: requires_instrumentation.

4.9 Search Gap Feedback (Q52, Q56; criterion 47, 50)

Signals emitted to feedback channels:

Signal Trigger
search_gap Query had no good result from any source
missing_thread Query implies a domain with no thread yet
weak_thread Thread exists but too few units or low confidence
noisy_thread Thread matched but units irrelevant
wrong_audience_result Internal content leaked to wrong audience scope — treated as a security/governance issue (rev5d §14.2 guardrail)
expected_artifact_missing Thread lifecycle chain has gaps (originating from D9 §4.9)

All signals route to Decision Backlog Registry (D5) + Segmentation Health Report (D3) + governance queue (Đ37). No new notification system is created (criterion 38).

Search usage thus closes the positive-recursion loop (P10, criterion 50): usage → gap detection → thread enrichment → better retrieval next cycle.

4.10 Audience-Scoped Search (Q53, Q54, Q55; criterion 48, 49)

Audience Sees Does not see
AI/Agent (internal, authorized) Material allowed by role, task scope, tool permission, data classification Secrets, unauthorized drafts, restricted incidents, blocked materials
Employee Approved SOP, runbook, enacted policy Draft designs, code internals, raw incidents
Partner Published guides, API docs, integration specs Internal governance, incidents, drafts
Customer FAQ, public policy, product docs Everything internal

Flow:

Query
  → Identify audience/scope (auth context or explicit parameter)
  → Apply visibility / readiness / publication filters (4.11)
  → Resolve thread/process/document view for that audience
  → Return ONLY approved, published, audience-appropriate units
  → Hide/redact internal-only material
  → Log search_gap if result is weak or empty after filtering

Access-Control Guardrail (rev5d §14.2): Audience-scoped search is an access-control requirement, not merely retrieval quality. wrong_audience_result is a security/governance event, not just a search-quality signal. Escalation goes to Đ37 governance and Đ32 risk review.

Rationale: rev5d §14.2; criterion 48.

4.11 Visibility / Readiness / Publication Filters (criterion 49)

Each unit carries (or inherits via thread):

Filter Values
visibility public / partner / employee / internal / restricted
readiness draft / under_review / approved / published / superseded
publication_state unpublished / published / withdrawn
risk_class low / standard / high (Đ32)
authority enacted / draft / runtime (Đ0-G distinction)

Default audience policy:

  • Customer-facing search: only visibility ∈ {public} AND publication_state='published' AND readiness='published'.
  • Partner-facing: {public, partner} AND published.
  • Employee-facing: {public, partner, employee} AND readiness ∈ {approved, published}.
  • AI/Agent-internal: per role/task scope/tool permission; respects all flags.

risk_class='high' material requires explicit consent / authorization regardless of audience.

4.12 v0.1 Boundary (rev5d §14.2)

v0.1 prepares: metadata fields, visibility/readiness/publication hooks, audience scope concept, retrieval metric hooks, search-gap signals.

v0.1 does NOT build: external-facing UI/search, customer portal, partner portal, customer FAQ surface, retrieval layer implementation.

When the retrieval layer is implemented, all hooks defined here must be respected.

4.13 Consumer Contract (AI/Agent Interaction)

AI/Agent consumers MUST:

  1. Use thread-first retrieval before vector search.
  2. Respect audience scope from their auth context.
  3. Surface suspect_links and known_gaps in their answers.
  4. Emit agent_extra_search_rate event when searching outside a delivered context pack.
  5. Cite units by canonical_address + source_revision for every claim.
  6. Treat wrong_audience_result as a hard error, not a quality concern.

Consumer contract is a policy in v0.1; enforcement mechanism is the retrieval layer (out of v0.1 scope).

4.14 Positive Recursion via Search Usage (Q56; criterion 50)

Search usage feeds:

  • Health signals (D3) when retrieval fails or returns noisy content.
  • Decision Backlog Registry (D5) when systemic gaps emerge.
  • Intake / enrichment (D9) when missing threads become recurrent.
  • Capability intake (D4) when retrieval improvements require new TAC/KG capability.

This is the closing loop of P10 positive recursion.

5. PG Storage per Object (Design Intent — No DDL)

Object Target DB Layer Notes
retrieval_metric_event directus Não Per-query event
retrieval_metric_aggregate directus Não Materialized view / rollup
search_gap_signal directus Não Routes to D3 + D5
audience_filter_policy directus Cổng Visibility / readiness / publication policy
thread_context_pack_cache directus (optional) Não Performance hook; deferred
consumer_contract_log directus Não For wrong_audience_result security events

Final placement per Đ33/Đ43 — D7/D10.

6. Schema Gaps

  1. retrieval_metric_event table — instrumentation gap (requires_instrumentation).
  2. retrieval_metric_aggregate — rollup/materialized view definition pending.
  3. search_gap_signal — schema and routing config.
  4. audience_filter_policy — per-role / per-task scope policy; may collide with existing access-control surface.
  5. visibility / readiness / publication_state / risk_class / authority fields on tac_logical_unit — current presence unclear; if missing, treat as schema gap.
  6. thread_centroid_embedding — per-thread aggregate for ranking (4.2); new artifact.
  7. wrong_audience_result event type — must be a distinct security/governance event, not merged with search quality signals.
  8. Consumer auth context propagation — how AI/Agent's role/scope reaches the retrieval layer; protocol gap.
  9. thread_context_pack_cache — optional performance artifact, deferred.
  10. Tie-in to existing Qdrant collections — fallback path implementation gap (NOT to be solved in v0.1).

7. Law References

Surface Law
Vocabulary (filter values) Đ24
Risk / authority gating Đ32
Governance escalation Đ37
Universal edges traversal Đ39
Schema compat Đ44
PG placement Đ33 / Đ43
Birth gate / authority distinction Đ0-G

8. Open Questions

  1. Exact threshold for thread_hit_rate initial target — defer to policy/D5 backlog.
  2. Embedding scheme for thread centroids — share with Qdrant or separate? Defer to D7/D10 + retrieval-layer design.
  3. Customer/partner audience formal definitions — defer to product/governance.
  4. Whether wrong_audience_result triggers automatic rollback of the response or only logs — recommendation: log + block + escalate to Đ37, no auto-rollback.
  5. Caching of Context Packs vs always-fresh — recommendation: always-fresh in v0.1 until metrics justify caching.

9. Coverage

Questions covered (primary): Q47, Q48, Q49, Q50, Q51, Q52, Q53, Q54, Q55, Q56. Questions covered (secondary): Q42, Q43 (supports D9).

Acceptance criteria covered:

  • 42 (thread-first retrieval)
  • 43 (thread resolution)
  • 44 (Thread Context Pack)
  • 45 (fallback to vector search)
  • 46 (retrieval quality metrics)
  • 47 (search-gap feedback signals)
  • 48 (audience-scoped search)
  • 49 (visibility/readiness/publication filters)
  • 50 (search usage → semantic health → positive recursion)

Schema gaps: 10 named (see §6).

Law dependencies: Đ24, Đ32, Đ33/Đ43, Đ37, Đ39, Đ44, Đ0-G.

Open questions: 5 (see §8).

Law conflicts encountered: none. Access-control guardrail honored — wrong_audience_result treated as security event.

Back to Knowledge Hub knowledge/dev/laws/dieu44-trien-khai/design/dot-iu-cutter-v0.1-thread-retrieval-user-interaction-design-2026-05-15.md