fix(evidence): run folder-agnostic checks on unmanifested folders (stacked on #115)

clawdie commented

2026-10-06 17:18:35 +02:00

Owner

Stacked on #115. Base is docs/evidence-coverage-and-identity-manifest; retarget to main once #115 merges. Review just the last commit.

What

check-evidence.py skipped every folder without a manifest.json before any check, so the folders' JSON files were never secret-scanned. This moves the folder-agnostic checks ahead of the manifest gate so they run on every folder, and adds a placeholder allowlist so benign templates don't fail.

Folder-agnostic checks now run for all folders: JSON syntax, trailing newline, and secret patterns.
Secret scanning covers all text files, not only .json (binary/oversized files are skipped), so a README.md or raw output can no longer hide a credential.
Placeholder allowlist — broad patterns are allowed when the match looks like a template: <your-key>, ${TOKEN}, your-token, example, changeme, redacted, dummy, xxx, ***. The benign ZAI_API_KEY=<your-key> in command-inventory is now allowed rather than failing.

Reporting shows coverage and placeholders:

=== Summary: 2 of 6 set(s) fully checked, 4 unmanifested (folder-agnostic checks only), 1 placeholder(s) allowed, 0 errors ===

Still non-fatal for missing manifests (schema/commit/hash remain manifest-gated); strict mode is a follow-up per Jev.

Verification

Negative test — inject a credential into an unmanifested folder:

Injected value	Result
`ZAI_API_KEY=sk-live-abcdefghijklmnop1234`	FAIL, exit 1 (was silently skipped before)
`ZAI_API_KEY=<your-key>`	allowed (1 placeholder), exit 0

./scripts/ci-checks.sh — All checks passed.
./scripts/check-evidence.py — 2 of 6 fully checked, 4 folder-agnostic only, 1 placeholder allowed, 0 errors.

Notes / trade-off

The placeholder regex is intentionally permissive; a genuine secret containing a placeholder word would be allowed. This is the cost of not failing on documentation templates; the alternative (strict patterns) would flag command-inventory.
This is the "folder-agnostic checks + allowlist" step Jev endorsed. Still deferred: --require-manifests strict mode, backfilling the two Jev folders, and the kind field for non-Jev audits.

> **Stacked on #115.** Base is `docs/evidence-coverage-and-identity-manifest`; retarget to `main` once #115 merges. Review just the last commit. ## What `check-evidence.py` skipped every folder without a `manifest.json` **before any check**, so the folders' JSON files were never secret-scanned. This moves the folder-agnostic checks ahead of the manifest gate so they run on **every** folder, and adds a placeholder allowlist so benign templates don't fail. - **Folder-agnostic checks now run for all folders:** JSON syntax, trailing newline, and secret patterns. - **Secret scanning covers all text files, not only `.json`** (binary/oversized files are skipped), so a `README.md` or raw output can no longer hide a credential. - **Placeholder allowlist** — broad patterns are allowed when the match looks like a template: `<your-key>`, `${TOKEN}`, `your-token`, `example`, `changeme`, `redacted`, `dummy`, `xxx`, `***`. The benign `ZAI_API_KEY=<your-key>` in `command-inventory` is now allowed rather than failing. - **Reporting** shows coverage and placeholders: ``` === Summary: 2 of 6 set(s) fully checked, 4 unmanifested (folder-agnostic checks only), 1 placeholder(s) allowed, 0 errors === ``` - **Still non-fatal for missing manifests** (schema/commit/hash remain manifest-gated); strict mode is a follow-up per Jev. ## Verification Negative test — inject a credential into an **unmanifested** folder: | Injected value | Result | | --- | --- | | `ZAI_API_KEY=sk-live-abcdefghijklmnop1234` | **FAIL**, exit 1 (was silently skipped before) | | `ZAI_API_KEY=<your-key>` | allowed (1 placeholder), exit 0 | - `./scripts/ci-checks.sh` — All checks passed. - `./scripts/check-evidence.py` — 2 of 6 fully checked, 4 folder-agnostic only, 1 placeholder allowed, 0 errors. ## Notes / trade-off - The placeholder regex is intentionally permissive; a genuine secret containing a placeholder word would be allowed. This is the cost of not failing on documentation templates; the alternative (strict patterns) would flag `command-inventory`. - This is the "folder-agnostic checks + allowlist" step Jev endorsed. Still deferred: `--require-manifests` strict mode, backfilling the two Jev folders, and the `kind` field for non-Jev audits.

clawdie added 1 commit

2026-10-06 17:18:35 +02:00

fix(evidence): run folder-agnostic checks on unmanifested folders

CI / markdown (pull_request) Successful in 9s

Details

CI / port (pull_request) Successful in 1s

Details

CI / agent-jail-pkgs (pull_request) Successful in 1s

Details

CI / python-tests (pull_request) Successful in 15s

Details

CI / go (pull_request) Successful in 5s

Details

format-check / prettier (pull_request) Successful in 5s

Details

leak-guard / guard (pull_request) Successful in 2s

Details

ci-gate / gate (pull_request) Has been cancelled

Details

cf96ddd120

The gate skipped unmanifested folders before any check, so their files were
never secret-scanned. Move the folder-agnostic checks (JSON syntax, trailing
newline, secret patterns) ahead of the manifest gate so every folder is
scanned, and allow documentation placeholders (e.g. <your-key>, ${TOKEN},
changeme) that match the broad credential patterns.

Secret scanning now covers all text files in a folder, not only .json, so a
README or raw output can no longer hide a credential. Schema, commit, and hash
checks stay manifest-gated; the summary now reports both coverage and the
number of placeholders allowed.

Gates: ci-checks.sh -- All checks passed. Negative test: an injected
credential in an unmanifested folder fails the gate as expected.

clawdie referenced this pull request

2026-10-06 17:27:08 +02:00

fix(evidence): report unmanifested folders + manifest the identity-decision probe set #115

clawdie commented

2026-10-06 17:27:09 +02:00

Author

Owner

Review — verified in a disposable worktree at cf96ddd (full stack)

The blind spot is closed end-to-end. I injected into an unmanifested, non-JSON file (phase-0-track-b-valhala/raw-output.txt) and reproduced the claim:

ZAI_API_KEY=sk-live-abcdefghijklmnop1234 → FAIL ... contains credential-shaped pattern, exit 1 (was silently skipped before).
ZAI_API_KEY=<your-key> → exit 0, counted as a placeholder.
Clean tree still exit 0 — no false positives across the existing docs today.

Issues / suggestions (non-blocking):

Double-counting. One injected secret line produced 2 FAIL lines / 2 errors, because both \bsk-[A-Za-z0-9_-]{16,} and \b[A-Z0-9_]*_API_KEY\s*[:=]\s*\S+ match. Still fails (fine), but the error count overstates. Consider errors += len(flagged) once per file, or dedupe by line.
Diagnostics. The FAIL message omits which pattern matched and where. Including the pattern (not the value) would speed up triage.
Silent skips remain, in a new place. Binary/oversized (>2 MB) files are skipped with no count reported. Add N file(s) skipped (binary/oversized) to the summary — otherwise a large file is the next uncaught blind spot, exactly the class this PR fixes.
Placeholder permissiveness (you flagged it): sample|fake is matched inside the secret substring, so sk-live-…-sample would be allowed. Acceptable trade-off, but please record it as a decision (script docstring or naming-decisions) so it is not re-litigated.
No unit tests. scripts/tests/ already holds test_check_okf.py; add a table-driven test for scan_secrets / PLACEHOLDER_RE (placeholder allowed · real secret fails · mixed file increments both · omitted newline). This is precisely the logic that regresses silently.
Retarget to main after #115 merges.

**Review — verified in a disposable worktree at `cf96ddd` (full stack)** The blind spot is closed end-to-end. I injected into an **unmanifested, non-JSON** file (`phase-0-track-b-valhala/raw-output.txt`) and reproduced the claim: - `ZAI_API_KEY=sk-live-abcdefghijklmnop1234` → `FAIL ... contains credential-shaped pattern`, **exit 1** (was silently skipped before). - `ZAI_API_KEY=<your-key>` → exit 0, counted as a placeholder. - Clean tree still exit 0 — no false positives across the existing docs today. Issues / suggestions (non-blocking): 1. **Double-counting.** One injected secret line produced **2 FAIL lines / 2 errors**, because both `\bsk-[A-Za-z0-9_-]{16,}` and `\b[A-Z0-9_]*_API_KEY\s*[:=]\s*\S+` match. Still fails (fine), but the error count overstates. Consider `errors += len(flagged)` once per file, or dedupe by line. 2. **Diagnostics.** The FAIL message omits which pattern matched and where. Including the pattern (not the value) would speed up triage. 3. **Silent skips remain, in a new place.** Binary/oversized (>2 MB) files are skipped with no count reported. Add `N file(s) skipped (binary/oversized)` to the summary — otherwise a large file is the next uncaught blind spot, exactly the class this PR fixes. 4. **Placeholder permissiveness** (you flagged it): `sample|fake` is matched inside the *secret substring*, so `sk-live-…-sample` would be allowed. Acceptable trade-off, but please record it as a decision (script docstring or naming-decisions) so it is not re-litigated. 5. **No unit tests.** `scripts/tests/` already holds `test_check_okf.py`; add a table-driven test for `scan_secrets` / `PLACEHOLDER_RE` (placeholder allowed · real secret fails · mixed file increments both · omitted newline). This is precisely the logic that regresses silently. 6. Retarget to `main` after #115 merges.

clawdie force-pushed fix/evidence-folder-agnostic-checks from cf96ddd120

CI / markdown (pull_request) Successful in 9s

Details

CI / port (pull_request) Successful in 1s

Details

CI / agent-jail-pkgs (pull_request) Successful in 1s

Details

CI / python-tests (pull_request) Successful in 15s

Details

CI / go (pull_request) Successful in 5s

Details

format-check / prettier (pull_request) Successful in 5s