fix(intake): deterministic quarantine in the gold harness + Jev gold-set baseline #15

Merged

clawdie merged 2 commits from fix/jev-gold-shortcircuit-and-baseline into main

2026-10-06 16:25:09 +02:00

clawdie commented

2026-10-06 16:19:11 +02:00

Owner

What

Two related changes to the content-intake Jev gold benchmark.

Harness fix — scripts/jev-classify.py gold computed the deterministic safety verdict with run_pre_safety_filter() but called Jev first, so a secret-bearing candidate (gold-08) was refused by the shared evaluator's secret scan and raised instead of emitting the verdict the filter had already decided. It now short-circuits action_override == "quarantine" to reject + quarantine with no model call. Non-quarantine overrides (requires_redaction) still reach Jev unchanged.
Baseline evidence — first recorded gold baseline under content-intake/jev-gold-baseline/.

Results (three runs per phase)

Phase	Run 1	Run 2	Run 3
Pre-fix	76.9%	76.9%	76.9%
Post-fix	84.6%	84.6%	84.6%

Site and action predictions are 13/13 stable across the post-fix runs.
Stable misses: gold-07 (archive vs reference accept), gold-09 (accept_with_edits vs reference reject). Both are action judgment; site routing is correct.
gold-08 now scores correctly (deterministic reject + quarantine) instead of erroring.

Gates

python3 scripts/check-site-boundaries.py — PASS
python3 scripts/verify-links.py — PASS
sh scripts/check-secrets.sh — clean
sh scripts/run-tests.sh — 31 tests OK (1 skipped)
python3 -m py_compile scripts/jev-classify.py — OK

Caveats

Reference labels are agent-authored, not a human gold set (see umbrella-pilot/REVIEW.md).
Transport: direct TypeSafe jev-latest, 1 attempt per call, caller pi-gold-benchmark. Cost not returned by the gateway.
Jev is non-deterministic; three runs is evidence from those runs, not a calibration set.

Policy note for the operator

AGENTS.md states generated evidence belongs under the ignored reports/ directory and that generated reports must not be written into tracked content-intake/ paths. This PR tracks content-intake/jev-gold-baseline/ to match the existing tracked Jev evidence directories (recon-path-consistency/, classify-wiki-fix/). Flagging the tension so the operator can reconcile the rule with current practice.

## What Two related changes to the content-intake Jev gold benchmark. 1. **Harness fix** — `scripts/jev-classify.py gold` computed the deterministic safety verdict with `run_pre_safety_filter()` but called Jev first, so a secret-bearing candidate (`gold-08`) was refused by the shared evaluator's secret scan and raised instead of emitting the verdict the filter had already decided. It now short-circuits `action_override == "quarantine"` to `reject` + `quarantine` with no model call. Non-quarantine overrides (`requires_redaction`) still reach Jev unchanged. 2. **Baseline evidence** — first recorded gold baseline under `content-intake/jev-gold-baseline/`. ## Results (three runs per phase) | Phase | Run 1 | Run 2 | Run 3 | | -------- | ----- | ----- | ----- | | Pre-fix | 76.9% | 76.9% | 76.9% | | Post-fix | 84.6% | 84.6% | 84.6% | - Site and action predictions are 13/13 stable across the post-fix runs. - Stable misses: `gold-07` (`archive` vs reference `accept`), `gold-09` (`accept_with_edits` vs reference `reject`). Both are action judgment; site routing is correct. - `gold-08` now scores correctly (deterministic `reject` + `quarantine`) instead of erroring. ## Gates - `python3 scripts/check-site-boundaries.py` — PASS - `python3 scripts/verify-links.py` — PASS - `sh scripts/check-secrets.sh` — clean - `sh scripts/run-tests.sh` — 31 tests OK (1 skipped) - `python3 -m py_compile scripts/jev-classify.py` — OK ## Caveats - Reference labels are agent-authored, not a human gold set (see `umbrella-pilot/REVIEW.md`). - Transport: direct TypeSafe `jev-latest`, 1 attempt per call, caller `pi-gold-benchmark`. Cost not returned by the gateway. - Jev is non-deterministic; three runs is evidence from those runs, not a calibration set. ## Policy note for the operator `AGENTS.md` states generated evidence belongs under the ignored `reports/` directory and that generated reports must not be written into tracked `content-intake/` paths. This PR tracks `content-intake/jev-gold-baseline/` to match the existing tracked Jev evidence directories (`recon-path-consistency/`, `classify-wiki-fix/`). Flagging the tension so the operator can reconcile the rule with current practice.

clawdie added 2 commits

2026-10-06 16:19:11 +02:00

fix(intake): short-circuit quarantined gold items to the deterministic verdict e9a8161c5a

The gold harness computed the deterministic safety verdict with
run_pre_safety_filter() but called Jev first, so a secret-bearing candidate
(gold-08) was refused by the shared evaluator's secret scan and raised instead
of emitting the verdict the filter had already decided. Skip the model call for
action_override == "quarantine" and record reject + quarantine directly.

Gates: py_compile OK; gold benchmark 11/13 (was 10/13 plus 1 error).

docs(intake): record the Jev gold-set baseline and 3-run stability a4206578e8

First recorded content-intake gold baseline: 11/13 (84.6%) post-fix, stable
across three runs, with site and action predictions 13/13 stable. Pre-fix was
10/13 (76.9%) with gold-08 erroring. gold-07 and gold-09 are stable action
misses.

Gates: check:boundaries, check:links, check:secrets PASS; run-tests 31 OK.

clawdie merged commit 2ea0820085 into main

2026-10-06 16:25:09 +02:00

clawdie deleted branch fix/jev-gold-shortcircuit-and-baseline

2026-10-06 16:25:09 +02:00

clawdie referenced this pull request from a commit

2026-10-06 16:25:10 +02:00

Merge pull request 'fix(intake): deterministic quarantine in the gold harness + Jev gold-set baseline' (#15) from fix/jev-gold-shortcircuit-and-baseline into main