fix(evidence): neutral v2 manifest with kind-scoped fields; plan-review evidence #117

Merged

clawdie merged 2 commits from feat/evidence-manifest-v2-and-plan-evidence into main 2026-10-06 18:11:53 +02:00

Conversation 0 Commits 2 Files changed 13 +732 -79

clawdie commented 2026-10-06 17:52:52 +02:00

Owner

Copy link

Implements the reduced evidence-manifest change that Jev grounded — not the original plan. The two probes are preserved in docs/evidence/evidence-manifest-plan-review/.

What Jev changed about the plan

Question	Jev	Conf	Effect
Non-Jev audits under the Jev manifest	separate_check	0.85	dropped command-inventory / platform_capture from the manifest scope
Schema versioning	bump v2, require kind	0.86	neutral evidence-manifest.v2
Jev blocks scope	kind-scoped	0.95	no fabricated model/provider/threshold on non-Jev sets
Strict-mode wiring	advisory + manual	0.81	overrules the reviewer — not wired into CI
Hashing scope	curated_core	0.95	hash decision artifacts, not every file
Historical backfill	defer lean	0.47	deferred, not fabricated

A first grounding probe returned 0.13, but its question embedded the "wire strict mode into CI" premise Jev then rejected. Re-asked cleanly → 0.83. The confounded number is documented in the evidence README.

Changes

• scripts/check-evidence.py — evidence-manifest.v2; required kind (jev_triage/jev_probe/system_inventory/platform_capture); identities/evaluator/policy required only for jev_*; --require-manifests (advisory default, manual); --evidence-dir; robustness fix so rel_path no longer assumes the tree is inside the repo.
• docs/evidence/README.md — v2 schema, kind table, kind-scoped fields, coverage status, strict-mode semantics.
• 2 manifests — bumped to v2 and given a kind.
• scripts/tests/test_evidence_gate.py — kind required/invalid, kind-scoped acceptance, old-schema rejection, strict-mode pass/fail (9 tests, was 4).
• docs/evidence/evidence-manifest-plan-review/ — the two probes, results, raw responses, and a v2 manifest.

Deliberately not done

• No manifest for agent-policy-evidence-review-2026-09-29 / jev-migration-triage — the gate verifies a commit exists, not that it is the right one; backfilling would fabricate provenance.
• Strict mode is not wired into ci-checks.sh.
• kind for non-Jev folders: the kinds are defined (system_inventory, platform_capture) but the folders stay unmanifested pending a kind-specific check.

Scope note

overall_grounding was 0.72 (<0.75) — Jev does not confidently endorse adopting the full plan, which is why this PR is the grounded subset only.

Gates

check-evidence.py 3 of 7, 0 errors · --require-manifests exit 1 as intended · python-tests 9 OK · ci-checks.sh All checks passed.

Decisions left to you

kind_split is ambiguous (distinct 0.59 / single 0.41); this PR takes distinct. Changeable in one line if you disagree.

Implements the **reduced** evidence-manifest change that Jev grounded — not the original plan. The two probes are preserved in `docs/evidence/evidence-manifest-plan-review/`. ### What Jev changed about the plan | Question | Jev | Conf | Effect | | --- | --- | --- | --- | | Non-Jev audits under the Jev manifest | **separate_check** | 0.85 | dropped `command-inventory` / `platform_capture` from the manifest scope | | Schema versioning | **bump v2, require kind** | 0.86 | neutral `evidence-manifest.v2` | | Jev blocks scope | **kind-scoped** | 0.95 | no fabricated `model`/`provider`/threshold on non-Jev sets | | Strict-mode wiring | **advisory + manual** | 0.81 | **overrules the reviewer** — not wired into CI | | Hashing scope | **curated_core** | 0.95 | hash decision artifacts, not every file | | Historical backfill | defer lean | 0.47 | deferred, not fabricated | A first `grounding` probe returned **0.13**, but its question embedded the "wire strict mode into CI" premise Jev then rejected. Re-asked cleanly → **0.83**. The confounded number is documented in the evidence README. ### Changes - **`scripts/check-evidence.py`** — `evidence-manifest.v2`; required `kind` (`jev_triage`/`jev_probe`/`system_inventory`/`platform_capture`); `identities`/`evaluator`/`policy` required only for `jev_*`; `--require-manifests` (advisory default, manual); `--evidence-dir`; robustness fix so `rel_path` no longer assumes the tree is inside the repo. - **`docs/evidence/README.md`** — v2 schema, kind table, kind-scoped fields, coverage status, strict-mode semantics. - **2 manifests** — bumped to v2 and given a `kind`. - **`scripts/tests/test_evidence_gate.py`** — kind required/invalid, kind-scoped acceptance, old-schema rejection, strict-mode pass/fail (9 tests, was 4). - **`docs/evidence/evidence-manifest-plan-review/`** — the two probes, results, raw responses, and a v2 manifest. ### Deliberately not done - No manifest for `agent-policy-evidence-review-2026-09-29` / `jev-migration-triage` — the gate verifies a commit *exists*, not that it is the *right* one; backfilling would fabricate provenance. - Strict mode is **not** wired into `ci-checks.sh`. - `kind` for non-Jev folders: the kinds are defined (`system_inventory`, `platform_capture`) but the folders stay unmanifested pending a kind-specific check. ### Scope note `overall_grounding` was **0.72 (<0.75)** — Jev does not confidently endorse adopting the *full* plan, which is why this PR is the grounded subset only. ### Gates `check-evidence.py` 3 of 7, 0 errors · `--require-manifests` exit 1 as intended · python-tests 9 OK · `ci-checks.sh` All checks passed. ### Decisions left to you `kind_split` is ambiguous (distinct 0.59 / single 0.41); this PR takes **distinct**. Changeable in one line if you disagree.

clawdie added 1 commit 2026-10-06 17:52:52 +02:00

fix(evidence): neutral v2 manifest with kind-scoped fields; plan-review evidence ... 96522d00b2

Reduced scope to the subset Jev grounded (see
docs/evidence/evidence-manifest-plan-review/).

Manifest schema -> evidence-manifest.v2 (neutral name; naming principle):
- require schema_version and kind; kinds = jev_triage, jev_probe,
  system_inventory, platform_capture.
- identities/evaluator/policy are required only for jev_* kinds, so a non-Jev
  audit never fabricates a model/provider/threshold.

Non-Jev audits leave the Jev manifest scope (Jev: separate_check 0.85):
command-inventory and phase-0-track-b-valhala keep folder-agnostic checks and
are documented, not forced into the Jev schema.

Strict mode stays advisory and manual (Jev: advisory_manual 0.81, overrules the
reviewer's earlier "wire it into CI" advice). --require-manifests fails only
when invoked; not wired into ci-checks.sh. --evidence-dir added for tests.

Deferred, not fabricated (Jev: defer lean; historical commits 0.30):
agent-policy-evidence-review-2026-09-29 and jev-migration-triage stay
unmanifested -- the gate verifies a commit exists, not that it is correct.

Also: robustness fix -- rel_path no longer assumes --evidence-dir is inside the
repo (the new tests found it). Hashing stays a curated subset (0.95).

Evidence: docs/evidence/evidence-manifest-plan-review/ (2 probes, results, raw,
manifest v2).

Gates: check-evidence.py 3 of 7, 0 errors; --require-manifests exit 1 as
intended; python-tests OK (9 tests); ci-checks.sh All checks passed.

clawdie referenced this pull request from a commit 2026-10-06 18:07:25 +02:00

fix(evidence): enforce common manifest fields and confine file paths

clawdie added 1 commit 2026-10-06 18:07:25 +02:00

fix(evidence): enforce common manifest fields and confine file paths ... 079945c5bb

Address PR #117 review findings:
- Enforce common manifest fields for all kinds (subject.repository,
  subject.subject_commit, subject.evaluation_phase, subject.evaluation_date,
  and non-empty files mapping).
- Confine file paths to the evidence family directory to prevent directory
  traversal via relative or escaping paths.
- Add regression tests covering missing subject fields, empty files,
  and path traversal rejection (12 tests total).

clawdie merged commit 34ade81bc7 into main 2026-10-06 18:11:53 +02:00

clawdie deleted branch feat/evidence-manifest-v2-and-plan-evidence 2026-10-06 18:11:53 +02:00

clawdie referenced this pull request from a commit 2026-10-06 18:11:54 +02:00

Merge pull request 'fix(evidence): neutral v2 manifest with kind-scoped fields; plan-review evidence' (#117) from feat/evidence-manifest-v2-and-plan-evidence into main

clawdie referenced this pull request from a commit 2026-10-06 18:24:34 +02:00

docs: close the contract-first checkpoint

clawdie referenced this pull request 2026-10-06 18:24:38 +02:00

docs: close the contract-first checkpoint #118

clawdie referenced this pull request from a commit 2026-10-06 18:24:57 +02:00

docs: close the contract-first checkpoint

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

clawdie/colibri!117

No description provided.