Fail-Closed Probe Suite Policy
Checker Policy — Fail-Closed Probe Suite (L2)
Generalizes: v0.2 second-pass self-codex probe suite (28 probes, 0 fail-open) and the missing-file probes (7/7 fail-closed). Level: 2.
The contract
For every invalid / adversarial input, the checker must fail closed: nonzero exit, no PASS token, no certificate, no digest, no seal-like output. The aggregate assertion the suite must prove:
any_invalid_exit0 = false
any_PASS_emitted_for_invalid = false
any_SEAL_emitted_for_invalid = false
Per-probe shape
Each probe declares:
(id, category, mutation, must = FAIL|PASS, expected_exit, actual_exit, PASS_emitted, SEAL_emitted).
- Invalid input →
must=FAIL, exactexpected_exit(nonzero),PASS_emitted=false,SEAL_emitted=false. - Positive controls →
must=PASS, exit 0,PASS_emitted=true, butSEAL_emitted=falsealways (a base packet never seals). At least one positive control is required so the checker is not trivially always-fail.
Detector-correctness rule (the key gotcha)
A seal/PASS token only counts as emitted when the process exits 0.
The first v0.2 run false-flagged probes because the harness substring-matched
SEAL / PROMOTED inside rejection messages like ORACLE_CLAIMS_SEAL_REJECTED.
The verifier was correct; the probe detector was wrong. Rule: gate
"token emitted" on exit==0. And when the detector itself is fixed, record the
fix transparently — do not hide it.
Required probe classes (generalized)
The base harness (harness/tkt_base_harness.py) implements the ten mandatory
base classes:
| # | Class | Invalid input | Expected |
|---|---|---|---|
| P1 | missing load-bearing file | drop a manifest-listed file | FAIL (≠0) |
| P2 | stale hash | mutate a file, keep old manifest hash | FAIL |
| P3 | duplicate manifest path | same path twice in manifest | FAIL |
| P4 | unlisted file | add a governed file not in manifest | FAIL or reported per policy |
| P5 | report-PASS-but-evidence-absent | report claims PASS, evidence file missing | FAIL |
| P6 | invalid-emits-certificate | invalid input that tries to emit cert/PASS | FAIL, no cert |
| P7 | duplicate object ID | sample registry with a dup id | FAIL |
| P8 | raw-evidence-without-hash/pointer/regen | evidence record missing a required field | FAIL |
| P9 | NVSZ root invented | designation by a non owner/operator | FAIL |
| P10 | semantic-PASS-without-IU | assert SEMANTIC_TEXT_AS_CODE_PASS / IU_TRACEABILITY_PASS with no IU input | FAIL (overclaim guard) |
A packet that targets L2 SHOULD also carry domain-specific probes (tamper, stale
hash on its own corpus, authority-claim — see authority_firewall_policy.md).
Independence requirement
Probes run against the checker reconstructed fresh from the governed store into a
clean temp dir (e.g. mktemp -d), not against the scratch workbench, so exits are
real and independent of leftover state.
Fail-closed contract
The suite PASSES only when every invalid probe failed closed (nonzero, no PASS, no
seal) AND every positive control passed (exit 0, no seal). Any single fail-open
(exit0 on invalid, or a PASS/seal token on invalid) fails the whole L2 claim.