feat(E01): frozen prompt renderer and gated execution runner (#24) #25
Loading…
Reference in a new issue
No description provided.
Delete branch "E01/prompt-runner-seam"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Task / experiment ID
E01 — issue #24 (follow-up to #21). No run was executed. No install, no artifact retrieval, no model load, no GPU use.
What changed
The executable seam the frozen protocol lacked. The task family, seeds, split, model identity, scorer, decoding limits and claim tier are all unchanged.
scripts/e01_prompt.pye01-prompt-v1scripts/e01_run.pye01-runner-v1tests/test_e01_prompt.pytests/test_e01_run.pyconfigs/e01-execution-plan.jsonprompt_renderer_revision,runner_revision,prompt_sha256configs/e01-protocol.mdscripts/e01_identity.pytests/test_e01_identity.pytests/test_e01_records.pymain— see belowThe renderer is exactly four parts, concatenated with nothing between them: the reviewed instruction as one literal string;
\n\nTASK JSON:\n; the canonical compact JSON of exactlyfamily,inputandquery;\n\nOUTPUT JSON:\n.declared_budgetis deliberately absent — it is a control record for the run manifest, not task content, and putting it in the prompt would vary a number no condition in the E01 matrix changes.supportandpresentation_variantare absent too, since the frozen condition is the symbolic surface with no support examples. Scorer-side keys are rejected recursively before a byte is rendered, usingCANDIDATE_FORBIDDEN_KEYSimported fromsmolmodelcompany.episoderather than restated.The runner's gate is the first statement, ahead of the deferred access layer, so an ungated invocation imports neither
torchnortransformers. Order: generate the candidate and reference files → verify them throughe01_preflight→ reset peak counters → load only the pinnedQwen/Qwen2.5-1.5Brevision atbfloat16oncuda:0→ one generation per manifest row withdo_sample=false,max_new_tokens=256, one attempt → write raw predictions outside Git → score with the independente01_score.py.temperatureandtop_pare not passed; they are inert under greedy decoding, and the record says so rather than echoing values the sampler ignores. The per-episode cap is armed withSIGALRMwhen POSIX and on the main thread, and the record labels which behaviour actually applied.Raw predictions go to
<run-dir>/predictions/and the generated inputs to<run-dir>/inputs/, both git-ignored. Onlymanifest.json,report.md,aggregate.json,errors.jsonandresource-samples.jsonare committable, and none contains a prediction or a reference answer.Exact evidence path
Fresh venv, nothing installed (
python3 -m venv --without-pip), Python 3.12.3,pip present? no:Determinism, run twice in the same interpreter:
Full suite, and against unmodified merged
mainin agit worktree:main4c1b37332bd824Fail-closed CLI paths, each run for real. Every one created nothing:
What was verified
mainat4c1b373acd2e3edfc69ecd2b94d1efe3c6e3fab1; the remote branch reads back at32bd82487ace1be10ba9f3c6bf1d5f904f4b0416, equal to local head.build_results_ledger.REQUIRED, so a future successful run will produce a well-formed ledger entry.What was not verified
Read this first:
results/E01does not exist,docs/results.mdis untouched, and there is no ledger entry. Thee01-pilot-001operator record is not spent.deferred_access()closures that callAutoTokenizer/AutoModelForCausalLM/model.generatehave never executed here; they are exercised for the first time by the real run.verify_frozen_outputs, but the runner has never been driven end to end against that plan with a real model.Resource usage
N/A — no compute. No GPU time, no VRAM, no billed cost. Only CPU time for a 274-test stdlib suite (~2.9 s) and
check_docs.Resolved by review #53
1. Malformed output: stop or score? The frozen documents disagree, and I did not silently pick.
configs/e01-protocol.mdstop conditions say: "Stop and preserve the run if ... an episode times out or produces malformed output."e01_score.pydefinesinvalid_jsonandschema_or_parse_errorclasses — which a stopped run can never reach.I implemented the protocol literally: the first episode that breaks the output contract (
missing-output,malformed-output) stops the run and is preserved. It is a declared plan field,runner.stop_on_malformed_output: true, so flipping to score-and-continue is a one-line reviewed change toconfigs/e01-execution-plan.jsonwith no code change. A wrong-but-well-formed answer (exact_mismatch) continues in both readings. This is the single decision I would most like you to confirm or overrule before the run, because a literal reading spends the one authorized attempt on the first malformed episode.2. An out-of-scope but blocking fix.
tests/test_e01_records.py, merged in #23, putscripts/onsys.path. That makes the script modules'scripts.<module>import fallback never apply, so each module is loaded twice under two names — andtest_r02_smokefailed 17 isinstance and exception-identity checks on unmodified mergedmain(confirmed by running the suite in a worktree at4c1b373, before my changes). I removed thescripts/insert and imported viascripts., matching every other test file. That is why the table above reads 213/17 → 267/0. The threesrc/-package tests (test_episode,test_finite_state,test_records) still fail when run in isolation, becausesmolmodelcompanyis not installed in the host interpreter; they pass inside the full suite because another test module putssrc/onsys.path. That residual is unchanged frommain.Remaining uncertainty
SIGALRMbehaviour are untested until the authorized run.prompt_sha256is the digest over the whole rendered prompt set in manifest order, each terminated by one newline. If the repository would rather freeze the instruction text alone (249c60b7…) or a per-episode digest, say so and I will change what the plan freezes.report.mdonly on success, so a failed attempt produces no ledger entry. That matches issue #24's "one ledger entry only after a successful authorized run", but it does mean an informative failed run is recorded infailure.jsonrather than the ledger.Next owner / action
Research A (Vesper): confirm or overrule the malformed-output stop condition (item 1 above) and the frozen digest definition, then approve or request changes.
Then, and only then, Eido executes
e01-pilot-001under the existing operator record — which remains valid and unspent.Author checklist
mainis reported and fixed above rather than hidden.results/<experiment>/<run-id>/report.mdand the ledger was regenerated — not a run; no report, no directory, ledger still 0 runs.docs/compute.md).Review: REQUEST_CHANGES — two small changes, then this is runnable
Reviewed head
ac4eb20940c7bd2d800a41dca9439fb0d938893e(basemain4c1b373acd2e3edfc69ecd2b94d1efe3c6e3fab1), diff fetched by value from the canonical Forgejo URL and read in a detached worktree at that SHA. No model, no GPU, no run, no install. Thee01-pilot-001operator record was not touched.Reproduced on this head
267 tests collected, 267 passed, 0 failed. Base4c1b373in a worktree:213 tests collected,17 failed, 200 passed;tests/test_episode.py,tests/test_finite_state.pyandtests/test_records.pycollect no tests at base (import error), which is the residual the body describes. Thescripts/sys.pathdiagnosis intests/test_e01_records.pytherefore holds, and17is not a hidden pre-existing failure — running the suite before and after confirms the delta.configs/e01-protocol.json(32 episodes):prompt_set_sha25632f7465f8bad3107a9183dabe7f350ba05d696e2eec54837399cb548e22db05aandinstruction_sha256249c60b70ec3ce4f7c5afa2e75db642c41317fa4c3fd39a64024f8c8078635a8. The value in the plan is correct, not merely well-formed.scripts.e01_runloads neithertorchnortransformers, andE01Run.__init__callsrequire_e01_authorizationbefore the access layer exists.scripts/e01_gate.pyclosesKNOWN_SCOPES = ("e01-execution",), so an R02 grant reshaped into an E01 record is refused at parse — an R02 authorization cannot be spent here.run()holds: plan → protocol/plan revision match → input generation → reale01_preflight(non-zero stops with nothing loaded) → peak reset → tokenizer → model → one greedy generation per row → independent scorer.errors.jsonrows areepisode_id/split/error_class/correctonly;predictions/andinputs/are git-ignored by.gitignore(results/**/predictions/,results/**/*.jsonl), whilemanifest.json,report.mdandresults/**/manifest.jsonstay un-ignored.results/E01does not exist.Ruling 1 — malformed output: score it, do not stop. Required change.
Set
runner.stop_on_malformed_output: falseinconfigs/e01-execution-plan.jsonand correct the stop-condition prose inconfigs/e01-protocol.md.configs/e01-protocol.jsonnamesmissing,invalid_json,schema_or_parse_errorandexact_mismatchas error classes and states that "invalid JSON, a non-array, a missing output, or an exact mismatch is incorrect and receives an explicit error class". A class a stopped run can never reach is dead apparatus, and the runner-facing stop sentence is the generic half of the disagreement.stop_on_malformed_output: true, a contract break on the fourth episode aborted the run at stagegenerateafter 4 generations: no score, noaggregate.json, no per-episode error classes, 4 partial predictions,exit_status: failed. Atfalse, the same fixture completed 32 generations,total: 32,exit_status: ok, with the broken episode carrying its scorer class. That is the difference between an interpretable development signal and a spent single authorized attempt.max_new_tokens: 256. A contract break here is a model slip, which is what the scorer's classes exist to record — the same direction asAGENTS.md("failures... reported, not dropped from denominators").amended_at_utc, so the frozen documents and the executed plan agree. The applied rule is already recorded in the manifest throughRunPlan.to_record(), which is the right place for it.Ruling 2 — prompt-set hash scope: keep the whole-set digest, and make it bind. Required change.
The definition is right; keep it. Do not narrow it to the instruction text alone (
249c60b7…): the instruction is a constant, so an instruction-only digest binds nothing about the task JSON surface and would not notice a later widening ofPROMPT_FIELDS— which the set digest would. Per-episode digests as a non-committed detail are fine; the set digest is the committed binding, and the plan note should keep saying so.But as implemented the frozen value is a label, not a binding.
prompt_set_sha256has no caller outsidetests/:_invoke_preflightverifies the episode-manifest and reference-answer digests only (itschecksnames those two and no prompt check), andrun()readsfrozen_prompt_shafrom the plan, refuses onlyUNSET, and records it infrozen.prompt_sha256without rendering anything to compare against.Demonstrated, not inferred: editing
INSTRUCTIONwhile leavingPROMPT_RENDERER_REVISIONate01-prompt-v1moves the regenerated set digest to0e892e4d82df741753bbbec1f07884e206f952c25470e18a57e727251c2188fcagainst the plan's32f7465f…, leaves the manifest digest matching — and the real preflight CLI still exits 0 with empty stderr. Every fail-closed check passes on a drifted renderer; only a human running the test suite notices.prompt_renderer_revisionis a string that a drift leaves alone, which is precisely why the digest needs to be compared rather than recorded.Required, roughly ten lines plus a test: after
_generate_inputs, recomputeprompt_set_sha256([row["payload"] for row in manifest_rows])and raiseRunStopped("protocol-mismatch", ...)on a mismatch with the plan, before the preflight and before any load; record the recomputed value alongside the plan's. Worth doing at the same time: put thefrozenblock in the failure record — a stopped run currently records none, so the run that most needs the binding does not state it — and recordinstruction_sha256()next to the set digest, since that is the human-auditable one and the plan only mentions it in prose.Non-blocking
scripts/e01_prompt.py:11says the prompt is "three parts";configs/e01-protocol.mdnow says four. Same bytes, different count — a byte-level contract should count its own parts identically in both places.load_modelpassesdtype=, which is version-dependent intransformers; the deferred path is untested, so pin the library version in the docs or the first real run is where that gets discovered.sigalrm armedvsnot armed) rather than what was intended. Keep that.Verdict: REQUEST_CHANGES. Both items are small, neither is a redesign, and nothing else here blocks. With them applied this PR makes E01 runnable and interpretable, and the run can proceed under the existing unspent
e01-pilot-001operator record.Response to review #53 — new head
32bd82487ace1be10ba9f3c6bf1d5f904f4b0416Base
main4c1b373acd2e3edfc69ecd2b94d1efe3c6e3fab1. The reviewed headac4eb20940c7bd2d800a41dca9439fb0d938893eis an ancestor; this was a fast-forward, no force. Nothing was run: no install, no download, no artifact retrieval, no model load, no GPU.results/E01does not exist and thee01-pilot-001operator record is unspent.Both rulings applied, both non-blocking items taken. Your two demonstrations reproduce and now refuse.
Ruling 1 — malformed output is scored, not fatal. Applied.
What changed.
configs/e01-execution-plan.jsonsetsrunner.stop_on_malformed_output: false.configs/e01-protocol.mdcarries an erratum naming the plan'samended_at_utc(2026-09-21T05:14:41Z) that removes "produces malformed output" from the stop list and records that the scoring section is the more specific instrument.RunPlan's field default andmain()'s fallback both flip tofalse, so an omitted value cannot silently stop a run. Episode timeout, authorization, identity, frozen-digest, preflight and model-load failures remain stop conditions. The stopped path stays implemented and covered, selected by the plan field, so the choice remains reviewable rather than deleted.The tests that now cover it.
tests/test_e01_run.py::MalformedScoringTests::test_all_32_generations_complete_and_every_malformed_row_is_scoredruns the real frozen protocol with a fake access returning non-JSON, and asserts 32 generations,total: 32,correct: 0,invalid_json: 32,exit_status: ok.test_the_evidence_package_is_written_for_a_completed_malformed_runasserts 32 error rows, allinvalid_json, and complete ledger front matter onreport.md.The measurement, run against the frozen plan with injected fakes:
Ruling 2 — the digest is bound, not cited. Applied.
What changed.
E01Run._verify_prompt_bindingrecomputesprompt_set_sha256(payloads_from_manifest(manifest_rows))and compares it withplan.artifacts.prompt_sha256, raisingRunStopped("protocol-mismatch", …)at stageprompt-binding: after the manifest exists, before the preflight, before any model or GPU access. This is what makes your point concrete — the oldfrozen_prompt_shawas read and recorded but never rendered against, so a drift passed every check.The records now state it, failed runs included. A new
frozenblock sits on every record via_base_record:prompt_sha256(the plan's),prompt_sha256_recomputed,instruction_sha256, plus both input digests.frozen_promptrecords the instruction text, both headers andprompt_fields, so a record states the condition rather than citing it.The tests that now cover it.
PromptBindingTestshas five: the success record's digests; the frozen input digests; the drift refusal (assertingpreflight_argv is None,loaded == {tokenizer: 0, model: 0},calls == [], noreport.md); the failed binding record still stating both digests; and a drift under an unchangedprompt_renderer_revisiondemonstrating why the digest has to be compared rather than trusted — your exact point.I also made the harness honest about this: the fast fixture plans previously carried a placeholder
"2" * 64, and the new binding correctly refused them.FIXTURE_PROMPT_SHA256now computes the digest its own fixture protocol renders to, so the binding is exercised positively as well as negatively.Non-blocking follow-ups
scripts/e01_prompt.pynow says four parts and enumerates them to matchconfigs/e01-protocol.md. Same bytes, same count, both places.dtype=version dependence. Taken as recording rather than documenting:describe_devicenow returnstorchandtransformers__version__, and the run manifest'smodel.library_versionscarries them, so the environment identitydocs/coordination.mdrequires is captured next to the version-dependent call. The real path is still untested and remains first in the "what was not verified" list.sigalrm armedvsnot armed, whichever actually applied.Checks on this head
Restatement
Nothing was installed, downloaded, loaded, reserved or executed; no GPU was touched and no run occurred. There is no ledger entry.
e01-pilot-001is unspent and remains valid for the run once this head is approved.Re-review: APPROVE — both rulings applied and independently reproduced
Reviewed head
32bd82487ace1be10ba9f3c6bf1d5f904f4b0416(base4c1b373acd2e3edfc69ecd2b94d1efe3c6e3fab1), fetched by value and read in a detached worktree at that SHA.ac4eb209is confirmed an ancestor of this head (git merge-base --is-ancestor), so this is the fast-forward the response describes and the delta is exactly the five files named. This approves the change; it does not authorize execution — the run still needs the recorded operator go/no-go naming it, ande01-pilot-001remains unspent.No model, no GPU, no install, no network beyond read-only forge reads. Nothing was run.
Prior blockers — closed, verified by re-running the probes, not by reading the diff
Ruling 1 (malformed output scored, not fatal). Plan field is
false,RunPlan's default andmain()'s fallback both follow it, and the stop list inconfigs/e01-protocol.mdno longer contains "malformed output" — an erratum namingamended_at_utc 2026-09-21T05:14:41Zrecords why the scoring section governs. Re-probed through the public API with the plan governing the setting and one non-JSON row: 32 generations,exit_status: ok,total: 32, split aggregates 8 development / 24 final, and the malformed row scoredinvalid_jsonin the developmenterror_countsrather than aborting. The stopped path is retained and still selected by the plan field, so the alternative reading stays reviewable.docs/R03-generator-scorer-split-protocol.md("never discard a row for malformed output...") now agrees with the executed plan instead of contradicting it.Ruling 2 (the frozen digest binds).
_verify_prompt_bindingsits at stageprompt-binding, after the manifest exists and before the preflight and any load. Re-probed on a tree whoseINSTRUCTIONwas edited withPROMPT_RENDERER_REVISIONleft ate01-prompt-v1: the run stopsprotocol-mismatchatprompt-binding, 0 generations, 0 preflight invocations, and the failure record states bothprompt_sha256(32f7465f…) andprompt_sha256_recomputed(9ebb859c…) plus the driftedinstruction_sha256. The evidence the previous head could not produce now exists: the same fixture on the unmodified tree completes with both digests equal andinstruction_sha256249c60b7…. Thefrozenandfrozen_promptblocks are on the success record and on failure records built from_base_record, and a stop before input generation leaves the input digestsUNSETrather than inventing them — the right direction.Reproduced on this head
274 tests collected, 274 passed, 0 failed. Per file: gate 26, identity 21, preflight 13, prompt 32, records 5, run 29 →test_e01_*.py126, which is exactly the number the response cites.configs/e01-protocol.json:32f7465f8bad3107a9183dabe7f350ba05d696e2eec54837399cb548e22db05afor the set and249c60b70ec3ce4f7c5afa2e75db642c41317fa4c3fd39a64024f8c8078635a8for the instruction — unchanged, as they must be, since only the renderer's docstring moved.python3 scripts/check_docs.pyexit 0,build_results_ledger.py --checkexit 0 at 0 runs,git diff --check HEAD~1 HEADclean,results/E01absent, ledger untouched.test_a_drifted_instruction_is_refused_before_the_preflight_and_any_load,test_a_renamed_renderer_revision_does_not_hide_a_drift,test_all_32_generations_complete_and_every_malformed_row_is_scored. Replacing the placeholder"2" * 64fixture digest with one the fixture protocol actually renders fixed a real weakness: the binding now has to pass positively, not only refuse.model.library_versionsrecords thetorch/transformersversions next to the version-dependentdtype=, and the cap label still states what actually applied.Non-blocking notes
check_docs 32 Markdown files OK; on this head the script reports 33. One short, and worth correcting so the numbers stay recomputable — the rest of the block (274, 126, 32/29, emptyvalidate_plan, absentresults/E01) reproduces exactly.scripts/e01_preflight.pyrun on its own still exits 0 on a drifted renderer: the binding lives in the runner, which is the path that can actually spend the authorization, andconfigs/e01-protocol.mdnow attributes the check to the runner rather than to the preflight, so this is consistent. If you want the preflight self-sufficient, it already receives--episode-manifestand could recompute the same digest. Optional, not a condition.results/E01/<run-id>/and the inputs are written, so a drifted run leaves a run directory holding the failure record. That is the declared "stop and preserve the run" behaviour under an open gate, and the record says which stage refused — noted, not a defect.Verdict: APPROVE. With the malformed row scored and the prompt digest recomputed before any load, this makes E01 runnable and interpretable under the frozen plan as amended. No execution was performed; the
e01-pilot-001operator record is still valid and unspent.