04 — Runtime Observation Substrate (Workstream C) — APPLY-PACKET (rehearsed, not committed)

4.1 Decision

A new two-table ledger, header + detail. Not an extension of dot_iu_command_run (wrong grain — IU-command governance layer), not events-only (events are notifications, not a durable run ledger). Mirrors the proven job_queue shape; reuses event_outbox.correlation_id as the wire key.

4.2 DDL (staged contabo:/tmp/02_observation_substrate.sql)

CREATE TABLE IF NOT EXISTS process_run_observation (
  process_run_id        uuid PRIMARY KEY,
  process_candidate_code text NOT NULL,           -- PROC-CAND:dot:kg
  process_definition_code text,                   -- set only after canon/birth
  correlation_id        text NOT NULL,            -- shared across components
  parent_run_id         uuid,                     -- nested runs
  status                text NOT NULL DEFAULT 'observed'
        CHECK (status IN ('observed','started','step_started','step_completed',
                          'completed','failed','stuck','cancelled','orphan')),
  source_system         text NOT NULL,            -- dot_runtime|job_queue|workflow_engine|dry_run
  idempotency_key       text,
  input_ref text, output_ref text, error_ref text, evidence_ref jsonb,
  started_at timestamptz, ended_at timestamptz,
  created_at timestamptz NOT NULL DEFAULT now()
);
-- + indexes on correlation_id, process_candidate_code, status, parent_run_id;
--   partial unique on idempotency_key.

CREATE TABLE IF NOT EXISTS process_component_observation (
  component_run_id uuid PRIMARY KEY,
  process_run_id   uuid NOT NULL REFERENCES process_run_observation(process_run_id),
  parent_run_id    uuid,
  correlation_id   text NOT NULL,
  dot_code text, component_role text, event_code text, queue_job_id uuid,
  status text NOT NULL DEFAULT 'observed' CHECK (status IN (...same set...)),
  input_ref text, output_ref text, error_ref text, evidence_ref jsonb,
  idempotency_key text, source_system text NOT NULL,
  started_at timestamptz, ended_at timestamptz,
  created_at timestamptz NOT NULL DEFAULT now()
);
-- + indexes on process_run_id, correlation_id, dot_code, queue_job_id;
--   partial unique on idempotency_key.

4.3 Status lifecycle

observed → started → step_started → step_completed → completed (happy path); side states failed, stuck, cancelled, orphan. Header tracks the run; detail tracks each component (producer/verifier/step).

4.4 Rehearsal proof

Rehearsed in BEGIN … ROLLBACK: birth_registry 1,158,148 before == after (CREATE TABLE + a test INSERT are birth-free — no birth trigger attaches to a fresh table), test insert into the header succeeded, rolled back clean. DROP-reversible (02_observation_rollback.sql).

4.5 Why not committed

The tables are empty with no consumer until the DOT execution wrapper / queue-event bridge exists (doc 06). Committing inert base tables is heavier than additive views and conflicts with the standing owner direction to avoid infra drift. They are apply-ready — the next macro applies them as step 1 of the dry-run. This satisfies Workstream C ("live, rehearsed, or apply-ready").

4.6 Retention / cleanup / birth implications

• Retention: keep completed runs 90 days, failed/stuck 180 days (audit), then archive; policy mirrors the existing auto-instantiate log-retention pattern (035_auto_instantiate_log_retention_policy.sql).
• Cleanup: a stuck sweeper marks runs whose started_at exceeds an SLA with no ended_at.
• Birth implications: the ledger is observation only — writing a run row is not a process birth. Birth remains the separate, owner-gated pivot_definitions/process-definition insert. Observation rows are evidence that enables a birth decision; they never are one.