fix(evidence): report unmanifested folders + manifest the identity-decision probe set #115
Loading…
Reference in a new issue
No description provided.
Delete branch "docs/evidence-coverage-and-identity-manifest"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What
The Jev evidence gate (
scripts/check-evidence.py) skipped anydocs/evidence/*folder that lacked amanifest.jsonbefore running any check, and its summary ("1 manifested set(s) checked, 0 errors") read as a full pass even though 5 of 6 folders were never examined. This PR makes the gap visible and brings one more folder under the gate.M of T set(s) checked, K skipped (no manifest), errors, then lists the unmanifested folders. It stays non-fatal (exit 0 on skips), so CI remains green.contributor-identity-decision— adds ajev-evidence-manifest.v1manifest with subject/evidence commits, evaluator provenance, and sha256 digests, bringing the identity-decision probes under the schema, commit, hash, JSON, and secret checks.Net effect on the gate output:
Jev advisory (3 runs,
jev_probe,typesafe/jev-latest)Asked whether the fix should broaden. Stable answers:
scope_now=gate_with_immediate(confidence 0.97) — fix the gate's silent skip and manifest the immediate folder, defer full backfill.strict_default_ready≈ 0.08 — do not make a missing manifest fail by default yet.non_jev_folders=kind_field(0.81) — give non-Jev audits their own valid form rather than forcing the Jev schema.silent_skip_is_risk≈ 0.55,look_broader≈ 0.73.This PR follows that: coverage + immediate manifest now; strict mode, folder backfill, and the
kindfield are deferred.Gates
./scripts/check-evidence.py— 2 of 6 sets checked, 0 errors./scripts/ci-checks.sh— All checks passed (Prettier, wiki-lint 762/0, date gate, OKF gate, evidence gate, whitespace gate)Deferred (follow-ups)
--require-manifests/ strict mode (Jev: ~0.08 ready).jev-migration-triageandagent-policy-evidence-review-2026-09-29.kindfield socommand-inventoryandphase-0-track-b-valhalaare not forced into the Jev schema.Notes
ZAI_API_KEY=<your-key>incommand-inventory), so no real credential is being skipped today.check-evidence.py skipped any docs/evidence/* folder without a manifest.json before running any check, and the summary ("N manifested set(s) checked, 0 errors") read as a full pass even though most folders were never examined. Record the skipped folders and report coverage (M of T checked, K skipped, errors) so the gap is visible. Stays non-fatal: failing on a missing manifest would break CI until every legacy folder is backfilled, which is a separate, deferred change. Gates: ./scripts/ci-checks.sh -- All checks passed.Review — verified in a disposable worktree at
c2aa33dConfirmed good:
mainbyte-for-byte (all 6 files), and the gate now reports2 of 6 set(s) checked, 4 skipped (no manifest)— the silent-skip is at least disclosed.subject_commit f06d855(#113 merge) andevidence_commit 7cc7ea2both resolve in history.Nits (non-blocking):
evidence_commitconvention is inconsistent. Here it is the pre-merge branch commit (7cc7ea2); injev-layered-memory-reconcileit is the merge commit (92be481). Pick one and state it in the manifest contract, or the field means different things per set.evaluation_phase: "decision_support"is not in the reference manifest (post_merge), andcheck-evidence.pydoes not validate the value. Either document the allowed phases or add an enum check, otherwise typos pass silently.The 4 unmanifested sets are an acknowledged deferral. Approve from my side once #116 lands on top.