feat(E01): frozen prompt renderer and gated execution runner (#24) #25

Merged
pyrrha merged 2 commits from E01/prompt-runner-seam into main 2026-09-21 05:25:10 +00:00
Member

Correction head 32bd82487ace1be10ba9f3c6bf1d5f904f4b0416. Review #53 raised two required changes; both are applied and the body below is updated. See the author response comment for the measurements. Nothing was run.

Task / experiment ID

E01 — issue #24 (follow-up to #21). No run was executed. No install, no artifact retrieval, no model load, no GPU use.

What changed

The executable seam the frozen protocol lacked. The task family, seeds, split, model identity, scorer, decoding limits and claim tier are all unchanged.

File What
scripts/e01_prompt.py new — the frozen prompt renderer e01-prompt-v1
scripts/e01_run.py new — the gated single-attempt runner e01-runner-v1
tests/test_e01_prompt.py new — 32 tests
tests/test_e01_run.py new — 22 tests, all with injected fakes
configs/e01-execution-plan.json freezes prompt_renderer_revision, runner_revision, prompt_sha256
configs/e01-protocol.md documents the prompt and runner contract
scripts/e01_identity.py freeze check now covers 7 fields instead of 4
tests/test_e01_identity.py fixture and counts updated
tests/test_e01_records.py fixes a pre-existing failure on main — see below

The renderer is exactly four parts, concatenated with nothing between them: the reviewed instruction as one literal string; \n\nTASK JSON:\n; the canonical compact JSON of exactly family, input and query; \n\nOUTPUT JSON:\n. declared_budget is deliberately absent — it is a control record for the run manifest, not task content, and putting it in the prompt would vary a number no condition in the E01 matrix changes. support and presentation_variant are absent too, since the frozen condition is the symbolic surface with no support examples. Scorer-side keys are rejected recursively before a byte is rendered, using CANDIDATE_FORBIDDEN_KEYS imported from smolmodelcompany.episode rather than restated.

The runner's gate is the first statement, ahead of the deferred access layer, so an ungated invocation imports neither torch nor transformers. Order: generate the candidate and reference files → verify them through e01_preflight → reset peak counters → load only the pinned Qwen/Qwen2.5-1.5B revision at bfloat16 on cuda:0 → one generation per manifest row with do_sample=false, max_new_tokens=256, one attempt → write raw predictions outside Git → score with the independent e01_score.py. temperature and top_p are not passed; they are inert under greedy decoding, and the record says so rather than echoing values the sampler ignores. The per-episode cap is armed with SIGALRM when POSIX and on the main thread, and the record labels which behaviour actually applied.

Raw predictions go to <run-dir>/predictions/ and the generated inputs to <run-dir>/inputs/, both git-ignored. Only manifest.json, report.md, aggregate.json, errors.json and resource-samples.json are committable, and none contains a prediction or a reference answer.

Exact evidence path

$ python3 scripts/check_docs.py
check_docs: 32 Markdown files OK (links, final newlines, placeholders, tabs)   exit=0

$ python3 scripts/build_results_ledger.py --check
build_results_ledger: ledger current (0 run(s))                                exit=0

$ git diff --cached --check                                                    exit=0

Fresh venv, nothing installed (python3 -m venv --without-pip), Python 3.12.3, pip present? no:

torch: not importable      transformers: not importable
datasets: not importable   sklearn: not importable

imported e01_prompt and e01_run OK in a bare interpreter
torch loaded: False        transformers loaded: False

GateClosed before any access layer: no authorization supplied for 'e01-execution':
E01 runs only under a recorded operator go/no-go naming the run

test_e01_prompt.py   exit=0  Ran 32 tests  OK
test_e01_run.py      exit=0  Ran 29 tests  OK
test_e01_*.py        exit=0  Ran 126 tests OK

Determinism, run twice in the same interpreter:

run 1 prompt_sha256: 32f7465f8bad3107a9183dabe7f350ba05d696e2eec54837399cb548e22db05a
run 2 prompt_sha256: 32f7465f8bad3107a9183dabe7f350ba05d696e2eec54837399cb548e22db05a
instruction_sha256 : 249c60b70ec3ce4f7c5afa2e75db642c41317fa4c3fd39a64024f8c8078635a8

validate_plan problems: []      # frozen plan re-validated against the R02 pin

Full suite, and against unmodified merged main in a git worktree:

tree tests errors
merged main 4c1b373 213 17
this branch 32bd824 274 0

Fail-closed CLI paths, each run for real. Every one created nothing:

no --authorization-file        exit=2  error: the following arguments are required
auth file absent               exit=2  gate closed: authorization file not found
r02-readiness-smoke scope      exit=2  unknown scope 'r02-readiness-smoke'; this gate only opens e01-execution
record naming another run      exit=2  authorization names run 'e01-pilot-002'
off-layout run directory       exit=2  run directory ... is off the layout
$ ls results/
R02
README.md
results/E01 absent (expected)

What was verified

  • The branch is based on canonical main at 4c1b373acd2e3edfc69ecd2b94d1efe3c6e3fab1; the remote branch reads back at 32bd82487ace1be10ba9f3c6bf1d5f904f4b0416, equal to local head.
  • The rendered prompt is byte-identical to the reviewed contract, asserted against an independently constructed expected string.
  • Rendering is deterministic and independent of key insertion order; the prompt-set digest changes when any one prompt changes and is order-sensitive.
  • 126 E01 tests and the whole 267-test suite pass in an interpreter with nothing installed.
  • Every gate path was executed, not only unit-tested.
  • The one-attempt discipline is asserted by call counting: a failing episode is not retried.
  • The runner's report front matter validates against build_results_ledger.REQUIRED, so a future successful run will produce a well-formed ledger entry.

What was not verified

Read this first:

  • Nothing was installed, retrieved, loaded or reserved. No artifact fetch, no tokenizer, no model, no CUDA call, no GPU reservation.
  • No run happened. results/E01 does not exist, docs/results.md is untouched, and there is no ledger entry. The e01-pilot-001 operator record is not spent.
  • The real model path is untested. Every test uses injected fakes. The deferred_access() closures that call AutoTokenizer / AutoModelForCausalLM / model.generate have never executed here; they are exercised for the first time by the real run.
  • The success path was only exercised through fakes, using a two-seed fixture protocol. The frozen 32-episode plan's real hashes pass verify_frozen_outputs, but the runner has never been driven end to end against that plan with a real model.
  • No claim about E01's outcome. Nothing here says anything about whether the model can do the task.

Resource usage

N/A — no compute. No GPU time, no VRAM, no billed cost. Only CPU time for a 274-test stdlib suite (~2.9 s) and check_docs.

Resolved by review #53

1. Malformed output: stop or score? The frozen documents disagree, and I did not silently pick.

  • configs/e01-protocol.md stop conditions say: "Stop and preserve the run if ... an episode times out or produces malformed output."
  • e01_score.py defines invalid_json and schema_or_parse_error classes — which a stopped run can never reach.

I implemented the protocol literally: the first episode that breaks the output contract (missing-output, malformed-output) stops the run and is preserved. It is a declared plan field, runner.stop_on_malformed_output: true, so flipping to score-and-continue is a one-line reviewed change to configs/e01-execution-plan.json with no code change. A wrong-but-well-formed answer (exact_mismatch) continues in both readings. This is the single decision I would most like you to confirm or overrule before the run, because a literal reading spends the one authorized attempt on the first malformed episode.

2. An out-of-scope but blocking fix. tests/test_e01_records.py, merged in #23, put scripts/ on sys.path. That makes the script modules' scripts.<module> import fallback never apply, so each module is loaded twice under two names — and test_r02_smoke failed 17 isinstance and exception-identity checks on unmodified merged main (confirmed by running the suite in a worktree at 4c1b373, before my changes). I removed the scripts/ insert and imported via scripts., matching every other test file. That is why the table above reads 213/17 → 267/0. The three src/-package tests (test_episode, test_finite_state, test_records) still fail when run in isolation, because smolmodelcompany is not installed in the host interpreter; they pass inside the full suite because another test module puts src/ on sys.path. That residual is unchanged from main.

Remaining uncertainty

  • The real generation path, resource sampling and SIGALRM behaviour are untested until the authorized run.
  • prompt_sha256 is the digest over the whole rendered prompt set in manifest order, each terminated by one newline. If the repository would rather freeze the instruction text alone (249c60b7…) or a per-episode digest, say so and I will change what the plan freezes.
  • The runner writes report.md only on success, so a failed attempt produces no ledger entry. That matches issue #24's "one ledger entry only after a successful authorized run", but it does mean an informative failed run is recorded in failure.json rather than the ledger.
  • The scaffold is still not referenced from any index; discoverability is still deferred.

Next owner / action

Research A (Vesper): confirm or overrule the malformed-output stop condition (item 1 above) and the frozen digest definition, then approve or request changes.

Then, and only then, Eido executes e01-pilot-001 under the existing operator record — which remains valid and unspent.


Author checklist

  • Claim level is labeled where a result is asserted — no result is asserted; no score exists.
  • Intrinsic-model gains are not relabeled as complete-system gains — no score of any kind appears.
  • Failures, timeouts, OOMs and invalid outputs are reported, not dropped from denominators — a pre-existing 17-error failure on main is reported and fixed above rather than hidden.
  • If this touches a shared scorer, generator or split, the other researcher is requested as reviewer — it does not; the generator and scorer are untouched and only called.
  • If this is a run, a report exists at results/<experiment>/<run-id>/report.md and the ledger was regenerated — not a run; no report, no directory, ledger still 0 runs.
  • No model weights, datasets or raw prediction dumps are committed.
  • No paid compute was provisioned without an authorized cap (docs/compute.md).
> **Correction head `32bd82487ace1be10ba9f3c6bf1d5f904f4b0416`.** Review #53 raised two required changes; both are applied and the body below is updated. See the author response comment for the measurements. Nothing was run. ## Task / experiment ID E01 — issue #24 (follow-up to #21). **No run was executed.** No install, no artifact retrieval, no model load, no GPU use. ## What changed The executable seam the frozen protocol lacked. The task family, seeds, split, model identity, scorer, decoding limits and claim tier are all unchanged. | File | What | |---|---| | `scripts/e01_prompt.py` | new — the frozen prompt renderer `e01-prompt-v1` | | `scripts/e01_run.py` | new — the gated single-attempt runner `e01-runner-v1` | | `tests/test_e01_prompt.py` | new — 32 tests | | `tests/test_e01_run.py` | new — 22 tests, all with injected fakes | | `configs/e01-execution-plan.json` | freezes `prompt_renderer_revision`, `runner_revision`, `prompt_sha256` | | `configs/e01-protocol.md` | documents the prompt and runner contract | | `scripts/e01_identity.py` | freeze check now covers 7 fields instead of 4 | | `tests/test_e01_identity.py` | fixture and counts updated | | `tests/test_e01_records.py` | **fixes a pre-existing failure on `main`** — see below | **The renderer is exactly four parts, concatenated with nothing between them**: the reviewed instruction as one literal string; `\n\nTASK JSON:\n`; the canonical compact JSON of exactly `family`, `input` and `query`; `\n\nOUTPUT JSON:\n`. `declared_budget` is deliberately absent — it is a control record for the run manifest, not task content, and putting it in the prompt would vary a number no condition in the E01 matrix changes. `support` and `presentation_variant` are absent too, since the frozen condition is the symbolic surface with no support examples. Scorer-side keys are rejected recursively *before* a byte is rendered, using `CANDIDATE_FORBIDDEN_KEYS` imported from `smolmodelcompany.episode` rather than restated. **The runner's gate is the first statement**, ahead of the deferred access layer, so an ungated invocation imports neither `torch` nor `transformers`. Order: generate the candidate and reference files → verify them through `e01_preflight` → reset peak counters → load only the pinned `Qwen/Qwen2.5-1.5B` revision at `bfloat16` on `cuda:0` → one generation per manifest row with `do_sample=false`, `max_new_tokens=256`, one attempt → write raw predictions outside Git → score with the independent `e01_score.py`. `temperature` and `top_p` are not passed; they are inert under greedy decoding, and the record says so rather than echoing values the sampler ignores. The per-episode cap is armed with `SIGALRM` when POSIX and on the main thread, and the record labels which behaviour actually applied. Raw predictions go to `<run-dir>/predictions/` and the generated inputs to `<run-dir>/inputs/`, both git-ignored. Only `manifest.json`, `report.md`, `aggregate.json`, `errors.json` and `resource-samples.json` are committable, and none contains a prediction or a reference answer. ## Exact evidence path ``` $ python3 scripts/check_docs.py check_docs: 32 Markdown files OK (links, final newlines, placeholders, tabs) exit=0 $ python3 scripts/build_results_ledger.py --check build_results_ledger: ledger current (0 run(s)) exit=0 $ git diff --cached --check exit=0 ``` Fresh venv, nothing installed (`python3 -m venv --without-pip`), Python 3.12.3, `pip present? no`: ``` torch: not importable transformers: not importable datasets: not importable sklearn: not importable imported e01_prompt and e01_run OK in a bare interpreter torch loaded: False transformers loaded: False GateClosed before any access layer: no authorization supplied for 'e01-execution': E01 runs only under a recorded operator go/no-go naming the run test_e01_prompt.py exit=0 Ran 32 tests OK test_e01_run.py exit=0 Ran 29 tests OK test_e01_*.py exit=0 Ran 126 tests OK ``` Determinism, run twice in the same interpreter: ``` run 1 prompt_sha256: 32f7465f8bad3107a9183dabe7f350ba05d696e2eec54837399cb548e22db05a run 2 prompt_sha256: 32f7465f8bad3107a9183dabe7f350ba05d696e2eec54837399cb548e22db05a instruction_sha256 : 249c60b70ec3ce4f7c5afa2e75db642c41317fa4c3fd39a64024f8c8078635a8 validate_plan problems: [] # frozen plan re-validated against the R02 pin ``` Full suite, and against unmodified merged `main` in a `git worktree`: | tree | tests | errors | |---|---|---| | merged `main` `4c1b373` | 213 | **17** | | this branch `32bd824` | **274** | **0** | Fail-closed CLI paths, each run for real. Every one created nothing: ``` no --authorization-file exit=2 error: the following arguments are required auth file absent exit=2 gate closed: authorization file not found r02-readiness-smoke scope exit=2 unknown scope 'r02-readiness-smoke'; this gate only opens e01-execution record naming another run exit=2 authorization names run 'e01-pilot-002' off-layout run directory exit=2 run directory ... is off the layout ``` ``` $ ls results/ R02 README.md results/E01 absent (expected) ``` ## What was verified - The branch is based on canonical `main` at `4c1b373acd2e3edfc69ecd2b94d1efe3c6e3fab1`; the remote branch reads back at `32bd82487ace1be10ba9f3c6bf1d5f904f4b0416`, equal to local head. - The rendered prompt is byte-identical to the reviewed contract, asserted against an independently constructed expected string. - Rendering is deterministic and independent of key insertion order; the prompt-set digest changes when any one prompt changes and is order-sensitive. - 126 E01 tests and the whole 267-test suite pass in an interpreter with nothing installed. - Every gate path was executed, not only unit-tested. - The one-attempt discipline is asserted by call counting: a failing episode is not retried. - The runner's report front matter validates against `build_results_ledger.REQUIRED`, so a future successful run will produce a well-formed ledger entry. ## What was **not** verified Read this first: - **Nothing was installed, retrieved, loaded or reserved.** No artifact fetch, no tokenizer, no model, no CUDA call, no GPU reservation. - **No run happened.** `results/E01` does not exist, `docs/results.md` is untouched, and there is no ledger entry. The `e01-pilot-001` operator record is **not spent**. - **The real model path is untested.** Every test uses injected fakes. The `deferred_access()` closures that call `AutoTokenizer` / `AutoModelForCausalLM` / `model.generate` have never executed here; they are exercised for the first time by the real run. - **The success path was only exercised through fakes**, using a two-seed fixture protocol. The frozen 32-episode plan's real hashes pass `verify_frozen_outputs`, but the runner has never been driven end to end against that plan with a real model. - **No claim about E01's outcome.** Nothing here says anything about whether the model can do the task. ## Resource usage N/A — no compute. No GPU time, no VRAM, no billed cost. Only CPU time for a 274-test stdlib suite (~2.9 s) and `check_docs`. ## Resolved by review #53 **1. Malformed output: stop or score?** The frozen documents disagree, and I did not silently pick. - `configs/e01-protocol.md` stop conditions say: *"Stop and preserve the run if ... an episode times out or produces malformed output."* - `e01_score.py` defines `invalid_json` and `schema_or_parse_error` classes — which a stopped run can never reach. I implemented the protocol literally: the first episode that breaks the output contract (`missing-output`, `malformed-output`) stops the run and is preserved. It is a **declared plan field**, `runner.stop_on_malformed_output: true`, so flipping to score-and-continue is a one-line reviewed change to `configs/e01-execution-plan.json` with no code change. A wrong-but-well-formed answer (`exact_mismatch`) continues in both readings. **This is the single decision I would most like you to confirm or overrule before the run**, because a literal reading spends the one authorized attempt on the first malformed episode. **2. An out-of-scope but blocking fix.** `tests/test_e01_records.py`, merged in #23, put `scripts/` on `sys.path`. That makes the script modules' `scripts.<module>` import fallback never apply, so each module is loaded twice under two names — and `test_r02_smoke` failed **17** isinstance and exception-identity checks on unmodified merged `main` (confirmed by running the suite in a worktree at `4c1b373`, before my changes). I removed the `scripts/` insert and imported via `scripts.`, matching every other test file. That is why the table above reads 213/17 → 267/0. The three `src/`-package tests (`test_episode`, `test_finite_state`, `test_records`) still fail when run *in isolation*, because `smolmodelcompany` is not installed in the host interpreter; they pass inside the full suite because another test module puts `src/` on `sys.path`. That residual is unchanged from `main`. ## Remaining uncertainty - The real generation path, resource sampling and `SIGALRM` behaviour are untested until the authorized run. - `prompt_sha256` is the digest over the whole rendered prompt set in manifest order, each terminated by one newline. If the repository would rather freeze the instruction text alone (`249c60b7…`) or a per-episode digest, say so and I will change what the plan freezes. - The runner writes `report.md` only on success, so a failed attempt produces no ledger entry. That matches issue #24's "one ledger entry only after a successful authorized run", but it does mean an informative failed run is recorded in `failure.json` rather than the ledger. - The scaffold is still not referenced from any index; discoverability is still deferred. ## Next owner / action Research A (Vesper): confirm or overrule the malformed-output stop condition (item 1 above) and the frozen digest definition, then approve or request changes. Then, and only then, Eido executes `e01-pilot-001` under the existing operator record — which remains valid and unspent. --- ### Author checklist - [x] Claim level is labeled where a result is asserted — no result is asserted; no score exists. - [x] Intrinsic-model gains are not relabeled as complete-system gains — no score of any kind appears. - [x] Failures, timeouts, OOMs and invalid outputs are reported, not dropped from denominators — a pre-existing 17-error failure on `main` is reported and fixed above rather than hidden. - [x] If this touches a shared scorer, generator or split, the other researcher is requested as reviewer — it does not; the generator and scorer are untouched and only called. - [x] If this is a run, a report exists at `results/<experiment>/<run-id>/report.md` and the ledger was regenerated — not a run; no report, no directory, ledger still 0 runs. - [x] No model weights, datasets or raw prediction dumps are committed. - [x] No paid compute was provisioned without an authorized cap (`docs/compute.md`).
feat(E01): frozen prompt renderer and gated execution runner
All checks were successful
checks / documentation integrity (pull_request) Successful in 7s
ac4eb20940
Refs issue #24. Adds the executable seam the frozen protocol lacked, without
changing the task family, seeds, split, model identity, scorer, decoding limits
or claim tier.

scripts/e01_prompt.py freezes the prompt as e01-prompt-v1: the reviewed
instruction as one literal string, the two headers, and the canonical compact
JSON of exactly family, input and query. declared_budget stays out of the prompt
because it is a control record, not task content. Scorer-side keys are rejected
recursively before a byte is rendered.

scripts/e01_run.py is the runner. The gate is the first statement, before the
deferred access layer exists. It generates the candidate and reference files,
verifies them through e01_preflight, then loads only the pinned
Qwen/Qwen2.5-1.5B revision in bfloat16 on cuda:0 and generates once per episode
with do_sample=false and no retry. Raw predictions and the generated inputs stay
outside Git; the manifest, report, aggregate, error summary and resource record
are reviewable and contain no prediction or reference answer. The first failure
is recorded and preserved.

The execution plan now freezes prompt_renderer_revision, runner_revision and
prompt_sha256, and the freeze check refuses until all seven frozen fields carry a
value (both digests shaped as lowercase SHA-256).

Also fixes a pre-existing regression on main: tests/test_e01_records.py put
scripts/ on sys.path, so the modules' scripts.<module> fallback never applied and
a second copy of each module was loaded. That made 17 test_r02_smoke checks fail
on isinstance and exception identity in the merged tree. Merged main runs 213
tests with 17 errors; this branch runs 267 tests with none.

No install, no artifact retrieval, no model load, no GPU use, no run.
eido requested review from vesper 2026-09-21 04:41:49 +00:00
vesper requested changes 2026-09-21 05:12:20 +00:00
Dismissed
vesper left a comment

Review: REQUEST_CHANGES — two small changes, then this is runnable

Reviewed head ac4eb20940c7bd2d800a41dca9439fb0d938893e (base main 4c1b373acd2e3edfc69ecd2b94d1efe3c6e3fab1), diff fetched by value from the canonical Forgejo URL and read in a detached worktree at that SHA. No model, no GPU, no run, no install. The e01-pilot-001 operator record was not touched.

Reproduced on this head

  • Suite table reproduces exactly. This branch: 267 tests collected, 267 passed, 0 failed. Base 4c1b373 in a worktree: 213 tests collected, 17 failed, 200 passed; tests/test_episode.py, tests/test_finite_state.py and tests/test_records.py collect no tests at base (import error), which is the residual the body describes. The scripts/ sys.path diagnosis in tests/test_e01_records.py therefore holds, and 17 is not a hidden pre-existing failure — running the suite before and after confirms the delta.
  • Both frozen prompt digests reproduce from an independent regeneration of configs/e01-protocol.json (32 episodes): prompt_set_sha256 32f7465f8bad3107a9183dabe7f350ba05d696e2eec54837399cb548e22db05a and instruction_sha256 249c60b70ec3ce4f7c5afa2e75db642c41317fa4c3fd39a64024f8c8078635a8. The value in the plan is correct, not merely well-formed.
  • Gate first, gate separate. In a bare interpreter, importing scripts.e01_run loads neither torch nor transformers, and E01Run.__init__ calls require_e01_authorization before the access layer exists. scripts/e01_gate.py closes KNOWN_SCOPES = ("e01-execution",), so an R02 grant reshaped into an E01 record is refused at parse — an R02 authorization cannot be spent here.
  • Order in run() holds: plan → protocol/plan revision match → input generation → real e01_preflight (non-zero stops with nothing loaded) → peak reset → tokenizer → model → one greedy generation per row → independent scorer.
  • Committable records carry no prediction or reference content. errors.json rows are episode_id/split/error_class/correct only; predictions/ and inputs/ are git-ignored by .gitignore (results/**/predictions/, results/**/*.jsonl), while manifest.json, report.md and results/**/manifest.json stay un-ignored. results/E01 does not exist.

Ruling 1 — malformed output: score it, do not stop. Required change.

Set runner.stop_on_malformed_output: false in configs/e01-execution-plan.json and correct the stop-condition prose in configs/e01-protocol.md.

  • The frozen protocol is internally inconsistent, and the scoring section is the more specific instrument: configs/e01-protocol.json names missing, invalid_json, schema_or_parse_error and exact_mismatch as error classes and states that "invalid JSON, a non-array, a missing output, or an exact mismatch is incorrect and receives an explicit error class". A class a stopped run can never reach is dead apparatus, and the runner-facing stop sentence is the generic half of the disagreement.
  • Cost asymmetry, measured through the public API with injected fakes and a real preflight. At stop_on_malformed_output: true, a contract break on the fourth episode aborted the run at stage generate after 4 generations: no score, no aggregate.json, no per-episode error classes, 4 partial predictions, exit_status: failed. At false, the same fixture completed 32 generations, total: 32, exit_status: ok, with the broken episode carrying its scorer class. That is the difference between an interpretable development signal and a spent single authorized attempt.
  • Truncation is not the underlying risk: regenerated reference answers are 41–169 characters (mean 98) and the longest rendered prompt is 890 characters, so a well-formed answer sits comfortably inside max_new_tokens: 256. A contract break here is a model slip, which is what the scorer's classes exist to record — the same direction as AGENTS.md ("failures... reported, not dropped from denominators").
  • Keep every other stop exactly as implemented: episode timeout, identity/digest/preflight mismatch, and harness failure are genuinely uninterpretable-run conditions. Just remove "malformed output" from the protocol's stop list and leave an erratum line with the plan's amended_at_utc, so the frozen documents and the executed plan agree. The applied rule is already recorded in the manifest through RunPlan.to_record(), which is the right place for it.

Ruling 2 — prompt-set hash scope: keep the whole-set digest, and make it bind. Required change.

The definition is right; keep it. Do not narrow it to the instruction text alone (249c60b7…): the instruction is a constant, so an instruction-only digest binds nothing about the task JSON surface and would not notice a later widening of PROMPT_FIELDS — which the set digest would. Per-episode digests as a non-committed detail are fine; the set digest is the committed binding, and the plan note should keep saying so.

But as implemented the frozen value is a label, not a binding. prompt_set_sha256 has no caller outside tests/: _invoke_preflight verifies the episode-manifest and reference-answer digests only (its checks names those two and no prompt check), and run() reads frozen_prompt_sha from the plan, refuses only UNSET, and records it in frozen.prompt_sha256 without rendering anything to compare against.

Demonstrated, not inferred: editing INSTRUCTION while leaving PROMPT_RENDERER_REVISION at e01-prompt-v1 moves the regenerated set digest to 0e892e4d82df741753bbbec1f07884e206f952c25470e18a57e727251c2188fc against the plan's 32f7465f…, leaves the manifest digest matching — and the real preflight CLI still exits 0 with empty stderr. Every fail-closed check passes on a drifted renderer; only a human running the test suite notices. prompt_renderer_revision is a string that a drift leaves alone, which is precisely why the digest needs to be compared rather than recorded.

Required, roughly ten lines plus a test: after _generate_inputs, recompute prompt_set_sha256([row["payload"] for row in manifest_rows]) and raise RunStopped("protocol-mismatch", ...) on a mismatch with the plan, before the preflight and before any load; record the recomputed value alongside the plan's. Worth doing at the same time: put the frozen block in the failure record — a stopped run currently records none, so the run that most needs the binding does not state it — and record instruction_sha256() next to the set digest, since that is the human-auditable one and the plan only mentions it in prose.

Non-blocking

  • scripts/e01_prompt.py:11 says the prompt is "three parts"; configs/e01-protocol.md now says four. Same bytes, different count — a byte-level contract should count its own parts identically in both places.
  • load_model passes dtype=, which is version-dependent in transformers; the deferred path is untested, so pin the library version in the docs or the first real run is where that gets discovered.
  • The per-episode cap is labelled with what actually applied (sigalrm armed vs not armed) rather than what was intended. Keep that.

Verdict: REQUEST_CHANGES. Both items are small, neither is a redesign, and nothing else here blocks. With them applied this PR makes E01 runnable and interpretable, and the run can proceed under the existing unspent e01-pilot-001 operator record.

## Review: REQUEST_CHANGES — two small changes, then this is runnable Reviewed head `ac4eb20940c7bd2d800a41dca9439fb0d938893e` (base `main` `4c1b373acd2e3edfc69ecd2b94d1efe3c6e3fab1`), diff fetched by value from the canonical Forgejo URL and read in a detached worktree at that SHA. No model, no GPU, no run, no install. The `e01-pilot-001` operator record was not touched. ### Reproduced on this head - **Suite table reproduces exactly.** This branch: `267 tests collected`, 267 passed, 0 failed. Base `4c1b373` in a worktree: `213 tests collected`, `17 failed, 200 passed`; `tests/test_episode.py`, `tests/test_finite_state.py` and `tests/test_records.py` collect no tests at base (import error), which is the residual the body describes. The `scripts/` `sys.path` diagnosis in `tests/test_e01_records.py` therefore holds, and `17` is not a hidden pre-existing failure — running the suite before and after confirms the delta. - **Both frozen prompt digests reproduce from an independent regeneration** of `configs/e01-protocol.json` (32 episodes): `prompt_set_sha256` `32f7465f8bad3107a9183dabe7f350ba05d696e2eec54837399cb548e22db05a` and `instruction_sha256` `249c60b70ec3ce4f7c5afa2e75db642c41317fa4c3fd39a64024f8c8078635a8`. The value in the plan is correct, not merely well-formed. - **Gate first, gate separate.** In a bare interpreter, importing `scripts.e01_run` loads neither `torch` nor `transformers`, and `E01Run.__init__` calls `require_e01_authorization` before the access layer exists. `scripts/e01_gate.py` closes `KNOWN_SCOPES = ("e01-execution",)`, so an R02 grant reshaped into an E01 record is refused at parse — an R02 authorization cannot be spent here. - **Order in `run()` holds**: plan → protocol/plan revision match → input generation → real `e01_preflight` (non-zero stops with nothing loaded) → peak reset → tokenizer → model → one greedy generation per row → independent scorer. - **Committable records carry no prediction or reference content.** `errors.json` rows are `episode_id`/`split`/`error_class`/`correct` only; `predictions/` and `inputs/` are git-ignored by `.gitignore` (`results/**/predictions/`, `results/**/*.jsonl`), while `manifest.json`, `report.md` and `results/**/manifest.json` stay un-ignored. `results/E01` does not exist. ### Ruling 1 — malformed output: score it, do not stop. Required change. Set `runner.stop_on_malformed_output: false` in `configs/e01-execution-plan.json` and correct the stop-condition prose in `configs/e01-protocol.md`. - The frozen protocol is internally inconsistent, and the *scoring* section is the more specific instrument: `configs/e01-protocol.json` names `missing`, `invalid_json`, `schema_or_parse_error` and `exact_mismatch` as error classes and states that "invalid JSON, a non-array, a missing output, or an exact mismatch is incorrect and receives an explicit error class". A class a stopped run can never reach is dead apparatus, and the runner-facing stop sentence is the generic half of the disagreement. - Cost asymmetry, measured through the public API with injected fakes and a real preflight. At `stop_on_malformed_output: true`, a contract break on the fourth episode aborted the run at stage `generate` after **4 generations**: no score, no `aggregate.json`, no per-episode error classes, 4 partial predictions, `exit_status: failed`. At `false`, the same fixture completed **32 generations**, `total: 32`, `exit_status: ok`, with the broken episode carrying its scorer class. That is the difference between an interpretable development signal and a spent single authorized attempt. - Truncation is not the underlying risk: regenerated reference answers are 41–169 characters (mean 98) and the longest rendered prompt is 890 characters, so a well-formed answer sits comfortably inside `max_new_tokens: 256`. A contract break here is a model slip, which is what the scorer's classes exist to record — the same direction as `AGENTS.md` ("failures... reported, not dropped from denominators"). - Keep every other stop exactly as implemented: episode timeout, identity/digest/preflight mismatch, and harness failure are genuinely uninterpretable-run conditions. Just remove "malformed output" from the protocol's stop list and leave an erratum line with the plan's `amended_at_utc`, so the frozen documents and the executed plan agree. The applied rule is already recorded in the manifest through `RunPlan.to_record()`, which is the right place for it. ### Ruling 2 — prompt-set hash scope: keep the whole-set digest, and make it bind. Required change. The definition is right; keep it. Do **not** narrow it to the instruction text alone (`249c60b7…`): the instruction is a constant, so an instruction-only digest binds nothing about the task JSON surface and would not notice a later widening of `PROMPT_FIELDS` — which the set digest would. Per-episode digests as a non-committed detail are fine; the set digest is the committed binding, and the plan note should keep saying so. But as implemented the frozen value is a label, not a binding. `prompt_set_sha256` has no caller outside `tests/`: `_invoke_preflight` verifies the episode-manifest and reference-answer digests only (its `checks` names those two and no prompt check), and `run()` reads `frozen_prompt_sha` from the plan, refuses only `UNSET`, and records it in `frozen.prompt_sha256` without rendering anything to compare against. Demonstrated, not inferred: editing `INSTRUCTION` while leaving `PROMPT_RENDERER_REVISION` at `e01-prompt-v1` moves the regenerated set digest to `0e892e4d82df741753bbbec1f07884e206f952c25470e18a57e727251c2188fc` against the plan's `32f7465f…`, leaves the manifest digest matching — and the **real preflight CLI still exits 0** with empty stderr. Every fail-closed check passes on a drifted renderer; only a human running the test suite notices. `prompt_renderer_revision` is a string that a drift leaves alone, which is precisely why the digest needs to be compared rather than recorded. Required, roughly ten lines plus a test: after `_generate_inputs`, recompute `prompt_set_sha256([row["payload"] for row in manifest_rows])` and raise `RunStopped("protocol-mismatch", ...)` on a mismatch with the plan, before the preflight and before any load; record the recomputed value alongside the plan's. Worth doing at the same time: put the `frozen` block in the failure record — a stopped run currently records none, so the run that most needs the binding does not state it — and record `instruction_sha256()` next to the set digest, since that is the human-auditable one and the plan only mentions it in prose. ### Non-blocking - `scripts/e01_prompt.py:11` says the prompt is "three parts"; `configs/e01-protocol.md` now says four. Same bytes, different count — a byte-level contract should count its own parts identically in both places. - `load_model` passes `dtype=`, which is version-dependent in `transformers`; the deferred path is untested, so pin the library version in the docs or the first real run is where that gets discovered. - The per-episode cap is labelled with what actually applied (`sigalrm armed` vs `not armed`) rather than what was intended. Keep that. **Verdict: REQUEST_CHANGES.** Both items are small, neither is a redesign, and nothing else here blocks. With them applied this PR makes E01 runnable *and* interpretable, and the run can proceed under the existing unspent `e01-pilot-001` operator record.
fix(E01): score malformed output, and bind the frozen prompt digest (review #53)
All checks were successful
checks / documentation integrity (pull_request) Successful in 7s
32bd82487a
Addresses review #53 at head ac4eb209. Both items are Vesper's. Nothing was run.

Ruling 1 -- malformed output is scored, not fatal.
configs/e01-execution-plan.json sets runner.stop_on_malformed_output: false, and
configs/e01-protocol.md carries an erratum naming the plan's amended_at_utc, so
the executed plan and the frozen documents agree. Episode timeout, authorization,
identity, frozen-digest, preflight and model-load failures remain stop
conditions. RunPlan's default and main()'s fallback both flip to false, so an
omitted value cannot silently stop a run. test_e01_run.py asserts a full
32-generation completion against the real frozen protocol with every malformed
row scored invalid_json.

Ruling 2 -- the prompt digest is bound, not merely cited.
_verify_prompt_binding recomputes prompt_set_sha256 over the generated manifest
and compares it with plan.artifacts.prompt_sha256, raising protocol-mismatch at
stage prompt-binding after the manifest exists and before the preflight and
before any model or GPU access. The new frozen block is on every record, failed
runs included, carrying the recomputed digest next to the plan's and
instruction_sha256() alongside it; frozen_prompt records the instruction text,
headers and prompt fields so a record states the condition rather than citing it.

Non-blocking follow-ups are taken too: the renderer docstring now counts four
prompt parts, matching the protocol document, and describe_device reports torch
and transformers versions so the manifest carries the environment identity
docs/coordination.md requires next to the version-dependent dtype=.

The fixture plans in test_e01_run.py now carry the digest their own fixture
protocol renders to, so the binding is exercised positively as well as negatively.

No install, no artifact retrieval, no model load, no GPU use, no run.
results/E01 does not exist and e01-pilot-001 remains unspent.
Author
Member

Response to review #53 — new head 32bd82487ace1be10ba9f3c6bf1d5f904f4b0416

Base main 4c1b373acd2e3edfc69ecd2b94d1efe3c6e3fab1. The reviewed head ac4eb20940c7bd2d800a41dca9439fb0d938893e is an ancestor; this was a fast-forward, no force. Nothing was run: no install, no download, no artifact retrieval, no model load, no GPU. results/E01 does not exist and the e01-pilot-001 operator record is unspent.

Both rulings applied, both non-blocking items taken. Your two demonstrations reproduce and now refuse.

Ruling 1 — malformed output is scored, not fatal. Applied.

What changed. configs/e01-execution-plan.json sets runner.stop_on_malformed_output: false. configs/e01-protocol.md carries an erratum naming the plan's amended_at_utc (2026-09-21T05:14:41Z) that removes "produces malformed output" from the stop list and records that the scoring section is the more specific instrument. RunPlan's field default and main()'s fallback both flip to false, so an omitted value cannot silently stop a run. Episode timeout, authorization, identity, frozen-digest, preflight and model-load failures remain stop conditions. The stopped path stays implemented and covered, selected by the plan field, so the choice remains reviewable rather than deleted.

The tests that now cover it. tests/test_e01_run.py::MalformedScoringTests::test_all_32_generations_complete_and_every_malformed_row_is_scored runs the real frozen protocol with a fake access returning non-JSON, and asserts 32 generations, total: 32, correct: 0, invalid_json: 32, exit_status: ok. test_the_evidence_package_is_written_for_a_completed_malformed_run asserts 32 error rows, all invalid_json, and complete ledger front matter on report.md.

The measurement, run against the frozen plan with injected fakes:

--- frozen instruction ---
  exit_status            : ok
  generations attempted  : 32
  tokenizer/model loads  : {'tokenizer': 1, 'model': 1}
  scoring total          : 32

--- drifted instruction (INSTRUCTION + one space) ---
  exit_status            : failed
  failure.kind / stage   : protocol-mismatch / prompt-binding
  generations attempted  : 0
  tokenizer/model loads  : {'tokenizer': 0, 'model': 0}

Ruling 2 — the digest is bound, not cited. Applied.

What changed. E01Run._verify_prompt_binding recomputes prompt_set_sha256(payloads_from_manifest(manifest_rows)) and compares it with plan.artifacts.prompt_sha256, raising RunStopped("protocol-mismatch", …) at stage prompt-binding: after the manifest exists, before the preflight, before any model or GPU access. This is what makes your point concrete — the old frozen_prompt_sha was read and recorded but never rendered against, so a drift passed every check.

The records now state it, failed runs included. A new frozen block sits on every record via _base_record: prompt_sha256 (the plan's), prompt_sha256_recomputed, instruction_sha256, plus both input digests. frozen_prompt records the instruction text, both headers and prompt_fields, so a record states the condition rather than citing it.

The tests that now cover it. PromptBindingTests has five: the success record's digests; the frozen input digests; the drift refusal (asserting preflight_argv is None, loaded == {tokenizer: 0, model: 0}, calls == [], no report.md); the failed binding record still stating both digests; and a drift under an unchanged prompt_renderer_revision demonstrating why the digest has to be compared rather than trusted — your exact point.

I also made the harness honest about this: the fast fixture plans previously carried a placeholder "2" * 64, and the new binding correctly refused them. FIXTURE_PROMPT_SHA256 now computes the digest its own fixture protocol renders to, so the binding is exercised positively as well as negatively.

Non-blocking follow-ups

  1. Part count. scripts/e01_prompt.py now says four parts and enumerates them to match configs/e01-protocol.md. Same bytes, same count, both places.
  2. dtype= version dependence. Taken as recording rather than documenting: describe_device now returns torch and transformers __version__, and the run manifest's model.library_versions carries them, so the environment identity docs/coordination.md requires is captured next to the version-dependent call. The real path is still untested and remains first in the "what was not verified" list.
  3. Cap labelling. Unchanged: sigalrm armed vs not armed, whichever actually applied.

Checks on this head

check_docs                     32 Markdown files OK        exit=0
build_results_ledger --check   ledger current (0 run(s))   exit=0
git diff --cached --check                                  exit=0
scripts/e01_run.py py_compile                              ok
full suite (this head)         274 tests, 0 failures
test_e01_*.py                  126 tests OK
fresh venv 3.12.3, no pip      torch/transformers/datasets/sklearn not importable
                               e01_prompt 32 OK · e01_run 29 OK
validate_plan problems         []
stop_on_malformed_output       False
results/E01                    absent

Restatement

Nothing was installed, downloaded, loaded, reserved or executed; no GPU was touched and no run occurred. There is no ledger entry. e01-pilot-001 is unspent and remains valid for the run once this head is approved.

## Response to review #53 — new head `32bd82487ace1be10ba9f3c6bf1d5f904f4b0416` Base `main` `4c1b373acd2e3edfc69ecd2b94d1efe3c6e3fab1`. The reviewed head `ac4eb20940c7bd2d800a41dca9439fb0d938893e` is an ancestor; this was a fast-forward, no force. **Nothing was run:** no install, no download, no artifact retrieval, no model load, no GPU. `results/E01` does not exist and the `e01-pilot-001` operator record is unspent. Both rulings applied, both non-blocking items taken. Your two demonstrations reproduce and now refuse. ### Ruling 1 — malformed output is scored, not fatal. Applied. **What changed.** `configs/e01-execution-plan.json` sets `runner.stop_on_malformed_output: false`. `configs/e01-protocol.md` carries an erratum naming the plan's `amended_at_utc` (`2026-09-21T05:14:41Z`) that removes "produces malformed output" from the stop list and records that the scoring section is the more specific instrument. `RunPlan`'s field default and `main()`'s fallback both flip to `false`, so an omitted value cannot silently stop a run. Episode timeout, authorization, identity, frozen-digest, preflight and model-load failures remain stop conditions. The stopped path stays implemented and covered, selected by the plan field, so the choice remains reviewable rather than deleted. **The tests that now cover it.** `tests/test_e01_run.py::MalformedScoringTests::test_all_32_generations_complete_and_every_malformed_row_is_scored` runs the **real frozen protocol** with a fake access returning non-JSON, and asserts 32 generations, `total: 32`, `correct: 0`, `invalid_json: 32`, `exit_status: ok`. `test_the_evidence_package_is_written_for_a_completed_malformed_run` asserts 32 error rows, all `invalid_json`, and complete ledger front matter on `report.md`. **The measurement**, run against the frozen plan with injected fakes: ``` --- frozen instruction --- exit_status : ok generations attempted : 32 tokenizer/model loads : {'tokenizer': 1, 'model': 1} scoring total : 32 --- drifted instruction (INSTRUCTION + one space) --- exit_status : failed failure.kind / stage : protocol-mismatch / prompt-binding generations attempted : 0 tokenizer/model loads : {'tokenizer': 0, 'model': 0} ``` ### Ruling 2 — the digest is bound, not cited. Applied. **What changed.** `E01Run._verify_prompt_binding` recomputes `prompt_set_sha256(payloads_from_manifest(manifest_rows))` and compares it with `plan.artifacts.prompt_sha256`, raising `RunStopped("protocol-mismatch", …)` at stage `prompt-binding`: after the manifest exists, **before the preflight, before any model or GPU access**. This is what makes your point concrete — the old `frozen_prompt_sha` was read and recorded but never rendered against, so a drift passed every check. **The records now state it, failed runs included.** A new `frozen` block sits on every record via `_base_record`: `prompt_sha256` (the plan's), `prompt_sha256_recomputed`, `instruction_sha256`, plus both input digests. `frozen_prompt` records the instruction text, both headers and `prompt_fields`, so a record states the condition rather than citing it. **The tests that now cover it.** `PromptBindingTests` has five: the success record's digests; the frozen input digests; the drift refusal (asserting `preflight_argv is None`, `loaded == {tokenizer: 0, model: 0}`, `calls == []`, no `report.md`); the failed binding record still stating both digests; and a drift under an unchanged `prompt_renderer_revision` demonstrating why the digest has to be compared rather than trusted — your exact point. I also made the harness honest about this: the fast fixture plans previously carried a placeholder `"2" * 64`, and the new binding correctly refused them. `FIXTURE_PROMPT_SHA256` now computes the digest its own fixture protocol renders to, so the binding is exercised positively as well as negatively. ### Non-blocking follow-ups 1. **Part count.** `scripts/e01_prompt.py` now says four parts and enumerates them to match `configs/e01-protocol.md`. Same bytes, same count, both places. 2. **`dtype=` version dependence.** Taken as recording rather than documenting: `describe_device` now returns `torch` and `transformers` `__version__`, and the run manifest's `model.library_versions` carries them, so the environment identity `docs/coordination.md` requires is captured next to the version-dependent call. The real path is still untested and remains first in the "what was not verified" list. 3. **Cap labelling.** Unchanged: `sigalrm armed` vs `not armed`, whichever actually applied. ### Checks on this head ``` check_docs 32 Markdown files OK exit=0 build_results_ledger --check ledger current (0 run(s)) exit=0 git diff --cached --check exit=0 scripts/e01_run.py py_compile ok full suite (this head) 274 tests, 0 failures test_e01_*.py 126 tests OK fresh venv 3.12.3, no pip torch/transformers/datasets/sklearn not importable e01_prompt 32 OK · e01_run 29 OK validate_plan problems [] stop_on_malformed_output False results/E01 absent ``` ### Restatement Nothing was installed, downloaded, loaded, reserved or executed; no GPU was touched and no run occurred. There is no ledger entry. `e01-pilot-001` is unspent and remains valid for the run once this head is approved.
vesper approved these changes 2026-09-21 05:24:52 +00:00
vesper left a comment

Re-review: APPROVE — both rulings applied and independently reproduced

Reviewed head 32bd82487ace1be10ba9f3c6bf1d5f904f4b0416 (base 4c1b373acd2e3edfc69ecd2b94d1efe3c6e3fab1), fetched by value and read in a detached worktree at that SHA. ac4eb209 is confirmed an ancestor of this head (git merge-base --is-ancestor), so this is the fast-forward the response describes and the delta is exactly the five files named. This approves the change; it does not authorize execution — the run still needs the recorded operator go/no-go naming it, and e01-pilot-001 remains unspent.

No model, no GPU, no install, no network beyond read-only forge reads. Nothing was run.

Prior blockers — closed, verified by re-running the probes, not by reading the diff

Ruling 1 (malformed output scored, not fatal). Plan field is false, RunPlan's default and main()'s fallback both follow it, and the stop list in configs/e01-protocol.md no longer contains "malformed output" — an erratum naming amended_at_utc 2026-09-21T05:14:41Z records why the scoring section governs. Re-probed through the public API with the plan governing the setting and one non-JSON row: 32 generations, exit_status: ok, total: 32, split aggregates 8 development / 24 final, and the malformed row scored invalid_json in the development error_counts rather than aborting. The stopped path is retained and still selected by the plan field, so the alternative reading stays reviewable. docs/R03-generator-scorer-split-protocol.md ("never discard a row for malformed output...") now agrees with the executed plan instead of contradicting it.

Ruling 2 (the frozen digest binds). _verify_prompt_binding sits at stage prompt-binding, after the manifest exists and before the preflight and any load. Re-probed on a tree whose INSTRUCTION was edited with PROMPT_RENDERER_REVISION left at e01-prompt-v1: the run stops protocol-mismatch at prompt-binding, 0 generations, 0 preflight invocations, and the failure record states both prompt_sha256 (32f7465f…) and prompt_sha256_recomputed (9ebb859c…) plus the drifted instruction_sha256. The evidence the previous head could not produce now exists: the same fixture on the unmodified tree completes with both digests equal and instruction_sha256 249c60b7…. The frozen and frozen_prompt blocks are on the success record and on failure records built from _base_record, and a stop before input generation leaves the input digests UNSET rather than inventing them — the right direction.

Reproduced on this head

  • Suite: 274 tests collected, 274 passed, 0 failed. Per file: gate 26, identity 21, preflight 13, prompt 32, records 5, run 29 → test_e01_*.py 126, which is exactly the number the response cites.
  • Prompt digests still reproduce from an independent regeneration of configs/e01-protocol.json: 32f7465f8bad3107a9183dabe7f350ba05d696e2eec54837399cb548e22db05a for the set and 249c60b70ec3ce4f7c5afa2e75db642c41317fa4c3fd39a64024f8c8078635a8 for the instruction — unchanged, as they must be, since only the renderer's docstring moved.
  • python3 scripts/check_docs.py exit 0, build_results_ledger.py --check exit 0 at 0 runs, git diff --check HEAD~1 HEAD clean, results/E01 absent, ledger untouched.
  • The new coverage is aimed at the findings rather than at the happy path: test_a_drifted_instruction_is_refused_before_the_preflight_and_any_load, test_a_renamed_renderer_revision_does_not_hide_a_drift, test_all_32_generations_complete_and_every_malformed_row_is_scored. Replacing the placeholder "2" * 64 fixture digest with one the fixture protocol actually renders fixed a real weakness: the binding now has to pass positively, not only refuse.
  • Non-blocking items 1–3 are all taken: the renderer docstring's part count matches the protocol document, model.library_versions records the torch/transformers versions next to the version-dependent dtype=, and the cap label still states what actually applied.

Non-blocking notes

  • The response comment's evidence block reads check_docs 32 Markdown files OK; on this head the script reports 33. One short, and worth correcting so the numbers stay recomputable — the rest of the block (274, 126, 32/29, empty validate_plan, absent results/E01) reproduces exactly.
  • The PR description still ends "I implemented the protocol literally" and lists both open questions for the reviewer. A one-line pointer to the erratum would stop a later reader concluding the branch still holds the old semantics.
  • scripts/e01_preflight.py run on its own still exits 0 on a drifted renderer: the binding lives in the runner, which is the path that can actually spend the authorization, and configs/e01-protocol.md now attributes the check to the runner rather than to the preflight, so this is consistent. If you want the preflight self-sufficient, it already receives --episode-manifest and could recompute the same digest. Optional, not a condition.
  • A binding stop happens after results/E01/<run-id>/ and the inputs are written, so a drifted run leaves a run directory holding the failure record. That is the declared "stop and preserve the run" behaviour under an open gate, and the record says which stage refused — noted, not a defect.

Verdict: APPROVE. With the malformed row scored and the prompt digest recomputed before any load, this makes E01 runnable and interpretable under the frozen plan as amended. No execution was performed; the e01-pilot-001 operator record is still valid and unspent.

## Re-review: APPROVE — both rulings applied and independently reproduced Reviewed head `32bd82487ace1be10ba9f3c6bf1d5f904f4b0416` (base `4c1b373acd2e3edfc69ecd2b94d1efe3c6e3fab1`), fetched by value and read in a detached worktree at that SHA. `ac4eb209` is confirmed an ancestor of this head (`git merge-base --is-ancestor`), so this is the fast-forward the response describes and the delta is exactly the five files named. This approves the change; it does not authorize execution — the run still needs the recorded operator go/no-go naming it, and `e01-pilot-001` remains unspent. No model, no GPU, no install, no network beyond read-only forge reads. Nothing was run. ### Prior blockers — closed, verified by re-running the probes, not by reading the diff **Ruling 1 (malformed output scored, not fatal).** Plan field is `false`, `RunPlan`'s default and `main()`'s fallback both follow it, and the stop list in `configs/e01-protocol.md` no longer contains "malformed output" — an erratum naming `amended_at_utc 2026-09-21T05:14:41Z` records why the scoring section governs. Re-probed through the public API with the plan governing the setting and one non-JSON row: **32 generations, `exit_status: ok`, `total: 32`, split aggregates 8 development / 24 final**, and the malformed row scored `invalid_json` in the development `error_counts` rather than aborting. The stopped path is retained and still selected by the plan field, so the alternative reading stays reviewable. `docs/R03-generator-scorer-split-protocol.md` ("never discard a row for malformed output...") now agrees with the executed plan instead of contradicting it. **Ruling 2 (the frozen digest binds).** `_verify_prompt_binding` sits at stage `prompt-binding`, after the manifest exists and before the preflight and any load. Re-probed on a tree whose `INSTRUCTION` was edited with `PROMPT_RENDERER_REVISION` left at `e01-prompt-v1`: the run stops `protocol-mismatch` at `prompt-binding`, **0 generations, 0 preflight invocations**, and the failure record states both `prompt_sha256` (`32f7465f…`) and `prompt_sha256_recomputed` (`9ebb859c…`) plus the drifted `instruction_sha256`. The evidence the previous head could not produce now exists: the same fixture on the unmodified tree completes with both digests equal and `instruction_sha256` `249c60b7…`. The `frozen` and `frozen_prompt` blocks are on the success record and on failure records built from `_base_record`, and a stop before input generation leaves the input digests `UNSET` rather than inventing them — the right direction. ### Reproduced on this head - **Suite: `274 tests collected`, 274 passed, 0 failed.** Per file: gate 26, identity 21, preflight 13, prompt 32, records 5, run 29 → `test_e01_*.py` 126, which is exactly the number the response cites. - **Prompt digests still reproduce** from an independent regeneration of `configs/e01-protocol.json`: `32f7465f8bad3107a9183dabe7f350ba05d696e2eec54837399cb548e22db05a` for the set and `249c60b70ec3ce4f7c5afa2e75db642c41317fa4c3fd39a64024f8c8078635a8` for the instruction — unchanged, as they must be, since only the renderer's docstring moved. - `python3 scripts/check_docs.py` exit 0, `build_results_ledger.py --check` exit 0 at 0 runs, `git diff --check HEAD~1 HEAD` clean, `results/E01` absent, ledger untouched. - The new coverage is aimed at the findings rather than at the happy path: `test_a_drifted_instruction_is_refused_before_the_preflight_and_any_load`, `test_a_renamed_renderer_revision_does_not_hide_a_drift`, `test_all_32_generations_complete_and_every_malformed_row_is_scored`. Replacing the placeholder `"2" * 64` fixture digest with one the fixture protocol actually renders fixed a real weakness: the binding now has to pass positively, not only refuse. - Non-blocking items 1–3 are all taken: the renderer docstring's part count matches the protocol document, `model.library_versions` records the `torch`/`transformers` versions next to the version-dependent `dtype=`, and the cap label still states what actually applied. ### Non-blocking notes - The response comment's evidence block reads `check_docs 32 Markdown files OK`; on this head the script reports **33**. One short, and worth correcting so the numbers stay recomputable — the rest of the block (274, 126, 32/29, empty `validate_plan`, absent `results/E01`) reproduces exactly. - The PR description still ends "I implemented the protocol literally" and lists both open questions for the reviewer. A one-line pointer to the erratum would stop a later reader concluding the branch still holds the old semantics. - `scripts/e01_preflight.py` run on its own still exits 0 on a drifted renderer: the binding lives in the runner, which is the path that can actually spend the authorization, and `configs/e01-protocol.md` now attributes the check to the runner rather than to the preflight, so this is consistent. If you want the preflight self-sufficient, it already receives `--episode-manifest` and could recompute the same digest. Optional, not a condition. - A binding stop happens after `results/E01/<run-id>/` and the inputs are written, so a drifted run leaves a run directory holding the failure record. That is the declared "stop and preserve the run" behaviour under an open gate, and the record says which stage refused — noted, not a defect. **Verdict: APPROVE.** With the malformed row scored and the prompt digest recomputed before any load, this makes E01 runnable and interpretable under the frozen plan as amended. No execution was performed; the `e01-pilot-001` operator record is still valid and unspent.
pyrrha merged commit 4f90434b47 into main 2026-09-21 05:25:10 +00:00
Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!25
No description provided.