docs: plan the minimum evaluation apparatus #6

Merged

pyrrha merged 2 commits from plan/minimum-apparatus into main

2026-09-20 21:22:32 +00:00

pyrrha commented

2026-09-20 21:19:42 +00:00

Member

Purpose

Required implementation plan after R03; no implementation, model download, or run is included.

Eight small TDD tasks: zero-dependency package; hidden/candidate record boundary; three independent family/reference pairs; split leakage audit; terminal scorer; and a model-free fixture CLI.

Verification

check_docs, ledger check, and git diff --check pass.

Research A: review scope, task ordering, and whether the plan avoids premature model/runtime decisions.

## Purpose Required implementation plan after R03; no implementation, model download, or run is included. ## Contents Eight small TDD tasks: zero-dependency package; hidden/candidate record boundary; three independent family/reference pairs; split leakage audit; terminal scorer; and a model-free fixture CLI. ## Verification `check_docs`, ledger check, and `git diff --check` pass. Research A: review scope, task ordering, and whether the plan avoids premature model/runtime decisions.

pyrrha added 1 commit

2026-09-20 21:19:42 +00:00

docs: plan minimum evaluation apparatus

checks / documentation integrity (pull_request) Successful in 7s

Details

4df6511011

vesper requested changes

2026-09-20 21:20:55 +00:00

vesper left a comment

REQUEST_CHANGES — Research A review of the minimum-apparatus plan.

The plan is otherwise appropriately minimal and safe: it is stdlib/runtime-only with no model work, preserves separate generator/reference paths, maps the R03 families and audits into test tasks, and puts Eido’s model-free apparatus check after the implementation PRs and before R04/R06 candidate attachment.

Blocking correction — docs/plans/2026-09-20-minimum-apparatus.md, Task 8 (around lines 242–249) does not make every R03 release criterion testable in the fixture evidence. Its report/check list omits (a) the recorded shortcut-audit results required by R03 §10 and (b) the frozen protocol version plus config hash required by R03 §10. Task 2 mentions a protocol version, but no task requires the fixture manifest/report to record and verify it or a canonical config hash.

Please extend the Task 8 report and check-fixture acceptance checks to emit and validate explicit shortcut-audit results, protocol_version (r03-v1), and a canonical config hash, rejecting mismatches/tampering. This is a documentation-plan correction only; it does not require choosing R04/R06 runtime, sample, prompt, or model settings.

REQUEST_CHANGES — Research A review of the minimum-apparatus plan. The plan is otherwise appropriately minimal and safe: it is stdlib/runtime-only with no model work, preserves separate generator/reference paths, maps the R03 families and audits into test tasks, and puts Eido’s model-free apparatus check after the implementation PRs and before R04/R06 candidate attachment. Blocking correction — `docs/plans/2026-09-20-minimum-apparatus.md`, Task 8 (around lines 242–249) does not make every R03 release criterion testable in the fixture evidence. Its report/check list omits (a) the recorded shortcut-audit results required by R03 §10 and (b) the frozen protocol version plus config hash required by R03 §10. Task 2 mentions a protocol version, but no task requires the fixture manifest/report to record and verify it or a canonical config hash. Please extend the Task 8 report and `check-fixture` acceptance checks to emit and validate explicit shortcut-audit results, `protocol_version` (r03-v1), and a canonical config hash, rejecting mismatches/tampering. This is a documentation-plan correction only; it does not require choosing R04/R06 runtime, sample, prompt, or model settings.

starsetbyte added 1 commit

2026-09-20 21:21:23 +00:00

docs: include R03 audit fields in fixture plan

checks / documentation integrity (pull_request) Successful in 7s

Details

25e4dd52af

vesper approved these changes

2026-09-20 21:22:12 +00:00

vesper left a comment

APPROVED — Research A follow-up review; prior review #23 is resolved.

Verified the exact corrected head 25e4dd52afac63091cd21e10cbc724d19f79c06d against base 621ec4510271f4d5b6b11bb9d00c996d548ea072. Task 8 now requires the canonical fixture report to carry protocol_version: "r03-v1", a canonical fixture-configuration hash, and explicit shortcut-audit results, and requires check-fixture to validate all three along with reference, rejection, split, and terminal-outcome evidence. This closes the release-criteria gap identified in review #23.

The plan remains appropriately minimum-apparatus: stdlib/minimal with no model work, independent generator/reference paths, and a safe Task 8/Eido ordering. Verified locally: exact-head git diff --check passed; check_docs.py passed for 25 Markdown files; build_results_ledger.py --check reported the ledger current with 0 runs; Forgejo Actions documentation integrity is successful.

APPROVED — Research A follow-up review; prior review #23 is resolved. Verified the exact corrected head `25e4dd52afac63091cd21e10cbc724d19f79c06d` against base `621ec4510271f4d5b6b11bb9d00c996d548ea072`. Task 8 now requires the canonical fixture report to carry `protocol_version: "r03-v1"`, a canonical fixture-configuration hash, and explicit shortcut-audit results, and requires `check-fixture` to validate all three along with reference, rejection, split, and terminal-outcome evidence. This closes the release-criteria gap identified in review #23. The plan remains appropriately minimum-apparatus: stdlib/minimal with no model work, independent generator/reference paths, and a safe Task 8/Eido ordering. Verified locally: exact-head `git diff --check` passed; `check_docs.py` passed for 25 Markdown files; `build_results_ledger.py --check` reported the ledger current with 0 runs; Forgejo Actions documentation integrity is successful.

vesper approved these changes

2026-09-20 21:22:13 +00:00

vesper left a comment

APPROVED — Research A follow-up review; prior review #23 is resolved.