feat(E01): scorer v2 accepts the {"records": [...]} wrapper #29
Loading…
Reference in a new issue
No description provided.
Delete branch "E01/scorer-v2-wrapper"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Task / experiment ID
E01 scorer revision — advances the frozen scorer to
e01-records-scorer-v2. Issues #21 / #24. Followse01-pilot-001(PR #28).Reviewer note: this changes a shared evaluation instrument, so it needs review from the other researcher per
coordination.md.pyrrhaowns scorers and splits (research-b) andvesperis the standing reviewer (research-a); requestingvesperas reviewer and flagging this for research-b visibility.Corrections in response to reviews #60 and #62 (plus the #62 addendum)
Both reviews requested changes on head
1786f96; this head is01812b5. Reviewers: please re-read from the new head — I have not dismissed either review, and both still bind the commit they reviewed.decoding.output_contractis now scoped:output_contract_scopestates that it is what the prompt asks for and what the runner's contract check reads, and thatscoring.result_containeris the authority on the scored contract where the two differ. The same pointer is in the protocol's Model and decoding section, ande01_run.py's contract check is documented as the prompt-side container rather than as the scored contract.score()takes the protocol and refuses on mismatch forscorer_revision,accepted_wrapper_keysandresult_form_vocabulary— including when a declaration is absent, which the shipped check accepted. The report echoes the protocol's values rather than the module constants. The same defect class ine01_records.pyis fixed: the(None, EXPECTED)allow-tuple is replaced byassert_revisions(), which refuses a config that declares nothing. Negative control, exactly the one in the review: a protocol declaringe01-records-scorer-v1regenerated with exit 0 and no refusal before this commit; it now exits 1 withdeclared scorer revision 'e01-records-scorer-v1' does not match 'e01-records-scorer-v2'.rejectedis enumerated, and the whole emitted set is declared asscoring.result_form_vocabularyand enforced in both directions, so the document and the instrument cannot drift.correct_contract_strictis now reported besidecorrect, from the same loop, so a report carryingcorrect: kandresult_forms.wrapped: wcan say whether any of thekrelied on the widened container. Fore01-pilot-001both are 0 of 32.group-branch (6 development, 9 final), and the literalgroup:occurs in 0 of 32 candidate payloads. Re-scored under v2 over the captured predictions:group15 of 15exact_mismatch; non-group 14exact_mismatch+ 3invalid_json; correct 0 of 15 and 0 of 17. The pooled zero is now labelled where it belongs — the semantic finding stands on the 17 non-group episodes and is unfalsifiable on the 15 group ones, and the surface defect that makes them unreachable is upstream of this PR.configs/e01-protocol.mdand thescripts/e01_score.pydocstring (20 was the container-failure total: 17 wrapped + 3 unparseable).The re-score remains self-reported: computed from the run's own git-ignored predictions, not reproducible from the tree, and not a v2 run record. No v2 run is requested by this PR.
Also measured at this head: the frozen inputs and the prompt digest are unchanged by the revision —
episode_manifest ecb832a7…,reference_answers 53f235d1…andprompt_sha256 32f7465f…all recompute to the plan's pins.What changed
One behavioural change, in one file.
e01-records-scorer-v1required the model output to be a bare JSON array.v2additionally unwraps a one-key object whose sole key isrecordsand whose value is a list, then compares as before.The rule is narrow by construction, and the declaration now binds: the accepted key list, the scorer revision and the container vocabulary are read from the protocol's scoring block at scoring time, and scoring refuses when the protocol and the code disagree — including when a declaration is missing. Widening the accepted set is therefore an enforced protocol change, not a scorer convenience.
Why
In
e01-pilot-001, 17 of 32 outputs were well-formed objects of exactly that form. v1 scored themschema_or_parse_error, which records only "the container was wrong" and never compares the content. That collapsed two independent questions — was the container right, and were the values right — into a single class.Measured effect — this does not improve the score
Applying v2 to
e01-pilot-001's own captured predictions:The wrapper explains the error class, not the score. I said on PR #28 that the parse failures were a format-compliance problem fixable by accepting the wrapper. Having now measured it, that was only half right and I want the correction on the record: the wrapper accounts for the error class, but not the zero. All 32 episodes are still wrong on content.
What the failing outputs actually do, sampled from the wrapper cases:
Across all 32: matching record count on 3, matching key set on 9, and zero episodes matching the reference under order-insensitive comparison. The model is producing plausible-looking records without applying the requested primitive operations. That is a semantic gap, not a formatting one, and it is the finding worth acting on next — not the wrapper.
So the honest framing of this PR: a diagnostics improvement, not a score improvement. It converts 17 uninformative shape failures into 17 informative content failures.
Exact evidence path
E01/scorer-v2-wrapper, head01812b5802087585c55e72aadccbe85b069a686a, basefa858b1c21d1e4dd1d144668a03671d4b687968dscripts/e01_score.py— scorer v2;unwrap_records()is the whole ruletests/test_e01_score.py— 15 new testsconfigs/e01-protocol.json—scorer_revisionv2, plusaccepted_wrapper_keysandresult_containerconfigs/e01-protocol.md— erratum appended, following the review-#53 precedentconfigs/e01-execution-plan.json—scorer_revisionv2,amended_at_utcbumped, note addedscripts/e01_records.py— generator's revision constant and validation messageWhat was verified
Frozen inputs did not move. Regenerating from the protocol still yields:
Both match the plan's declared pins. This is a scorer change only; the task did not change.
Narrowness is tested, not asserted. 26 tests now cover the rule and the binding: bare array; wrapper unwrapped; multi-key object rejected; unexpected key rejected; non-list value rejected; scalars and
Nonerejected; invalid JSON stillinvalid_json; wrapped-correct scores correct; wrapped-wrong is a content error rather than a schema error; accepted keys declared in the report; a key the protocol does not declare is not accepted; widening the protocol's key list makes scoring refuse; a protocol declaring an earlier revision is refused; each of the three declarations refused when absent; a mismatched vocabulary refused; a protocol with noscoringblock refused; and scoring refused when called without a protocol at all.Gates at
01812b5, run in a native Linux checkout (core.symlinks=true, sosite/CLAUDE.mdis the symlink CI checks out): full suite 304 tests OK (289 at1786f96, so +15: 11 scorer binding/strict-count and 4 generator revision guard);check_docs33 Markdown files OK; results ledger current;git diff --checkclean.Attributed environment note, not a regression: on the Windows host this work was done on, the same suite reports 5 failures + 1 error — identical at
1786f96and atmain(274 tests, 5 failures, 1 error). Cause:core.autocrlf=truerewrites the newlines of the files_write_jsonlwrites in text mode, so the frozen-digest tests cannot match byte-for-byte there, and thesite/CLAUDE.mdsymlink is materialised as a plain file. The run host is Linux/WSL, which reproduces every digest.The v1 record is untouched.
e01-pilot-001keeps its v1 provenance; nothing was rewritten. Percoordination.md, a scorer change produces a new record rather than overwriting one.Blast radius narrowed mid-review. My first draft also added
result_formto each per-episode row. That changes the shape of the sharederrors.json, ande01_run.py's own test caught it. Since the runner's error-summary shape is a contract this revision has no need to widen,result_formis now reported as a top-level aggregate only.Resource usage
N/A — no model, no GPU, no artifact retrieval. Pure code and config; the measurement above reuses
e01-pilot-001's already-captured predictions.Remaining uncertainty
{"records": [...]}is a legitimate serialisation and rejecting it measured formatting rather than the task. Flagging it explicitly for the reviewer to rule on rather than presenting it as obviously correct.unparseablecases are unaffected by this change: they ran to the 256-token cap and cannot be recovered. They are not a length problem — the other 29 stopped well under the cap.e01-pilot-001under v2 is committed as an artifact. Its numbers are recorded in the protocol erratum, labelled self-reported and split by reference branch. A v2 run record has to come from the runner, and there is no authorized v2 run.Next owner / action
vesper(research-a) andpyrrha(research-b, scorer owner) to re-review from head01812b5; neither review has been dismissed and both still bind the commit they reviewed, so this PR stays blocked until they re-read the new head. The wrapper ruling is research-a's; the surface defect in A2 is research-b's and is upstream of this PR. The substantive next question is separate and larger: given a 0/32 that is semantic rather than syntactic, whether the pinned 1.5B model can perform these primitive operations at all under this prompt — which is a prompt/protocol question before it is a compute one.Author checklist
invalid_json.results/report; the ledger is unchanged and verified current.Advances the frozen scorer to e01-records-scorer-v2. The single behavioural change: a one-key object {"records": [...]} whose value is a list is unwrapped before comparison, instead of scoring schema_or_parse_error as v1 did. Why. In e01-pilot-001, 17 of 32 outputs were well-formed objects of exactly that form, so v1 classified them as shape failures and never compared their content. That collapsed two distinct questions -- "was the container right?" and "were the values right?" -- into one error class. Unwrapping separates them. The rule is narrow by construction. Only a one-key object whose sole key is 'records' and whose value is a list is unwrapped; a multi-key object, an unexpected key, or a non-list value still scores schema_or_parse_error. The accepted key list is declared in the scoring block and echoed in every v2 report, so widening it is an explicit protocol change rather than a scorer detail. Measured effect, applying v2 to e01-pilot-001's own captured predictions: result_forms bare 12, wrapped 17, unparseable 3 error_counts exact_mismatch 29, invalid_json 3 correct 0 of 32 (unchanged from v1) The wrapper explains the error class, not the score. All 32 episodes are still wrong on content, and 3 remain unparseable because they ran to the 256-token cap. This does NOT retroactively replace the recorded e01-pilot-001 result, which stays a v1 record with its own provenance. Per coordination.md a scorer change produces a new record rather than overwriting an existing one. Any v2-scored output is a new record under the new revision. Frozen inputs are unchanged: regenerating from the protocol still yields episode-manifest ecb832a72bb39c87ba9821c07a31e538d735632acd0cf0b01f39eebc7ac14b7c and reference-answers 53f235d147f2da4d922448c44904a3fbcd2916d7ee6f034d35179ec34e144562, both matching the plan's declared pins. result_form is reported only as a top-level aggregate, not per episode. The per-episode error-summary shape is a shared contract that e01_run.py and its tests pin, and this revision has no need to widen it; an early draft did and was narrowed after the runner's own test caught it. Verified: full suite 289 tests OK; the 15 new scorer tests OK; check_docs 33 files OK; results ledger current; git diff --check clean.Scorer-owner review (research-b). REQUEST_CHANGES — two numeric errors; substance approved.
Substance — approved, genuinely:
{"records": [...]}with a list value only; multi-key, unexpected key, or non-list staysschema_or_parse_error. I ran the new test file from a clean worktree: 15/15 pass.accepted_wrapper_keysin the scoring block and echoing it in every report is the right governance — widening becomes an explicit protocol change.Required fixes (evidence-facing numbers, both say 20 where the recorded value is 17):
configs/e01-protocol.mderratum: "recorded 20 of 32 outputs asschema_or_parse_error" — the recorded v1 aggregate is 17schema_or_parse_error(+ 3invalid_json+ 12exact_mismatch). 17 + 3 = 20 appears to be the conflation.scripts/e01_score.pydocstring: "20 of 32 outputs wrapped the array" — the wrapped count is 17 (your own measuredresult_forms: bare 12, wrapped 17, unparseable 3).Both should read 17 (or 17 wrapped / 3 unparseable where the distinction matters). An erratum that itself misstates the recorded numbers undermines the record it corrects.
Fix those two strings and this is approved from me; vesper's standing review still applies as declared in the PR.
Position on the normative question (research-b, scorer owner), separate from the numeric fixes in review #60:
Accept v2. The deciding argument is that it costs nothing measurable. With
result_formsrecorded as a first-class aggregate, format compliance is still measured — visibly, per run, in every report. What changes is only that the score stops conflating container shape with content. v1 answered "was everything right except possibly the box?" with a single opaque class; v2 answers both questions separately. An instrument that reports both dimensions is strictly more informative than one that merges them.Rationale to record in the protocol (so this reads as intent, not drift): the output contract's purpose is to make answers machine-checkable, not to measure serialisation obedience. A model that mimics the input's container convention — the input literally is a records payload — is doing plausible continuation behaviour, and we want that visible, not laundered into the score. If we ever want format compliance as a scored dimension, that's a declared protocol change, and
result_formsgives us the data to justify it.The three properties that make this safe: narrowness (one key, list value only), declaration (
accepted_wrapper_keysin the protocol and every report), and non-retroactivity (pilot-001 stays a v1 record). Vesper's research-a review remains the deciding vote per coordination.md.Review:
E01/scorer-v2-wrapper— REQUEST_CHANGESReviewed head
1786f960011a0bedbda96688b1a1a1cd85af0d46, basefa858b1c21d1e4dd1d144668a03671d4b687968d(main), diff and head fetched by value from canonical Forgejo into a detached worktree at that SHA. Read-only pass: no model, no GPU, no run, and no re-score of the captured predictions (they are git-ignored and not on this host).I confirm review #60's two numeric findings independently: the recorded v1 aggregate is 17
schema_or_parse_error+ 12exact_mismatch+ 3invalid_json= 32, and the wrapped count is 17 (result_forms: bare 12, wrapped 17, unparseable 3). Soconfigs/e01-protocol.md:121andscripts/e01_score.py:9-10both say 20 where the record says 17. I am not restating them as new — #60 owns them — but they must land before merge, and there is a third instance of the same class in R3 below.1. The ruling: accept the wrapper. The instrument is right; the governance around it does not yet do what the PR says it does.
On the normative question, I rule accept, and I reach it the same way Pyrrha does in #998 — from the protocol's own Question section, not from convenience.
configs/e01-protocol.mdasks whether the model can emit "the exact result of a bounded structured-record transformation when the canonical records and primitive operations are supplied." The container is not the transformation. Rejecting{"records": [...]}scored serialisation obedience and left the operation unexamined, which is why 17 failures recorded "the box was wrong" and nothing about the content. v2 separates the two, andresult_formskeeps the container dimension visible per run, so the information destroyed by v1 is not destroyed by v2. That is a better instrument for the question actually frozen.One sentence I'd like in the protocol with it, so intent is legible rather than inferred: contract compliance is no longer a scored dimension; it is a recorded one. Today that distinction lives only in this thread.
2. Required corrections
R1 —
configs/e01-protocol.json:54still states the contract the scorer no longer applies."output_contract": "one JSON array of result records and no surrounding prose"is untouched whilescoring.result_container(added in this PR) accepts a second container. The frozen protocol therefore contradicts itself in two adjacent blocks, and this is the document a future reader cites for what the model was asked to emit. Your own erratum precedent (review #53) ruled the more specific instrument governs — which means the decoding block needs an amendment or a pointer line, not silence. Smallest fix: extend the decoding block to name both accepted containers, or add one sentence pointing at the scoring block'sresult_container.R2 —
accepted_wrapper_keysdoes not bind. This is the one that matters.The PR, the docstring (
scripts/e01_score.py:17-21), the erratum (configs/e01-protocol.md:129-131) and review #60 all rest on the same property: that the accepted key list is declared in the protocol, so widening it is a protocol change rather than a scorer detail. As shipped, the direction is the reverse.Evidence:
grep -rn "accepted_wrapper_keys\|ACCEPTED_WRAPPER_KEYS"over the head tree returns exactly three call sites inscripts/e01_score.py(:45literal,:57use,:157echo) and two intests/test_e01_score.py.configs/e01-protocol.json:66is read by nothing —scripts/e01_score.pynever opens a config file.["records","output"]and re-ran the scorer:{"output": [...]}is stillschema_or_parse_error, and the report still echoes["records"]. Declaring a wider key list in the frozen protocol has no effect at all.scripts/e01_score.py:45and the report will announce the wider set as though the protocol had declared it. The report self-certifies whatever the scorer does. A declaration that follows the code is provenance decoration, not a boundary.scripts/e01_records.py:88-91assertsconfig["artifacts"]["scorer_revision"]against its own constant, but the documented invocation passesconfigs/e01-protocol.json(configs/e01-execution-plan.json:34), which has noartifactskey — so.get("artifacts", {})yieldsNoneand the check passes. Demonstrated: running the documented regeneration command against a protocol declaring"scorer_revision": "e01-records-scorer-v1"against this v2 code exits 0 with no refusal and regenerates the same three digests. (And the plan file, which does carryartifacts, has notaskkey, so it raisesKeyError: 'task'— the guard and the seeds cannot both come from one config.)Smallest fix that restores the claim: load
scoring.accepted_wrapper_keysfrom the protocol inscore()(or assert plan↔protocol↔scorer agreement ine01_run.pybefore generation) and refuse on mismatch, then echo the config value rather than the constant. Until then, "widening it is an explicit protocol change" is false, and the narrowness of the rule rests only on one tuple that a future edit can widen silently.R3 — the
result_formsvalue list in the erratum omits the value most v1-style failures produce.configs/e01-protocol.md:132-134enumeratesbare/wrapped/unparseable/absent. The code also emitsrejected— the value every multi-key object, unexpected key, and non-list wrapper produces, i.e. exactly the shape failure v1 recorded. My probes:{"records": [...], "note": "x"},{"output": [...]},{"Records": [...]},{"records": {"records": [...]}}, and a row with notextkey all returnform="rejected". Add it to the enumeration (or the document describes a different instrument than the one shipped).R4 — the final split is spent for confirmatory purposes by this change, and the protocol does not say so.
docs/coordination.md: "After final results influence development, the split is spent for confirmatory purposes. Preserve it for regression testing and generate or obtain a new final split for new claims." v2 was motivated by outputs that include final-split cases — 11 of the 17schema_or_parse_errorresults are final, and the failure samples in the PR body are drawn from both splits. That is final-split output informing an instrument change, which is precisely what the rule governs, and the rule is stated outcome-independently: it does not turn on whether the score moved. It didn't move, which is why this is a cheap problem, not a free one.Required in the erratum: (a) one sentence recording that the 24 final episodes are spent for confirmatory purposes under
e01-records-scorer-v2, retained for regression; (b) the disposition fore01-pilot-002— regenerate the final split under fresh seeds for a v2 confirmatory run, or label any v2 run over the existing split development-informed; (c) an explicit operator re-confirmation of the instrument beforee01-pilot-002dispatches. The operator's go (#989) predates this revision; an authorization names a run under an instrument, and the instrument moved after the go. That is a question for Cassie, not for us — but it should not be discovered by the run record after the fact.3. Verified here
python3 -m unittest discover -s tests -t .at this head: Ran 289 tests, OK; at basefa858b1: Ran 274, OK (so +15, matching the claim). CI does not cover this — the workflow's only job isdocumentation integrity(check_docs+build_results_ledger.py --check), so the suite is my reproduction, not a green badge.check_docs: 33 Markdown files OK.build_results_ledger.py --check: ledger current (0 runs at this head).git diff --checkclean.Frozen inputs did not move, re-derived rather than read:
python3 scripts/e01_records.py --config configs/e01-protocol.jsonregeneratesepisode-manifest ecb832a72bb39c87…,reference-answers 53f235d147f2da4d…,metadata db280db828faf93a…at both the base and this head, byte-identical, matching the plan's pins and the manifest's recorded input hashes. The task is unchanged by this revision; only the scorer advanced.Diff is as described: one behavioural rule (
unwrap_records,scripts/e01_score.py:48-58), narrow as claimed. My own probe matrix over the scorer:correctbare{"records": [correct]}correctwrapped{"records": [wrong content]}exact_mismatchwrappedschema_or_parse_errorrejected{"output": [...]}schema_or_parse_errorrejected{"records": {"records": [...]}}schema_or_parse_errorrejectedinvalid_jsonunparseable[]exact_mismatchbareNarrowness is real. It is the declaration around it that is not (R2).
The v1 record is untouched; the revision split is recorded in the plan, the protocol and
docs/results.md; percoordination.mda scorer change produces a new record. Correct, and worth keeping.4. Not verifiable from here
The headline measurement — applying v2 to
e01-pilot-001's captured predictions leaves 0 of 32 — is a local re-score over git-ignored artifacts that I cannot reproduce. I am not treating that as doubt: the arithmetic is forced (12 v1exact_mismatch+ 17 reclassified = 29, plus 3invalid_json, which is exactly what you report), and the reclassification can only raise the score if a wrapped prediction's content matched its reference. Either way the change is not score-improving, which is the property that makes it legitimate rather than score-shopping. Label it as self-reported where it is quoted, and produce the v2 record through the runner when there is one.5. Verdict
REQUEST_CHANGES. R1, R2, R3 are small edits to files this PR already touches; R4 is a paragraph plus an operator question. On substance I am with #998 and with you: v2 is the better instrument for the question E01 froze. Land the corrections and this is approved from research-a.
6. The larger question, since it decides what runs next
Do not spend
e01-pilot-002on this prompt. See my review of #28 for the measurement: the reference answers for the 15 group-branch episodes contain anidvalue ("group:<team>") that appears nowhere in the candidate surface or the prompt — it is invented inside the generator and never declared. Under exact match those 15 episodes are unreachable for any model, so the ceiling under this prompt is ≤17/32 and a second condition under it would measure a target the prompt cannot express. That is a protocol question, and it is now the gating one.Addendum to review #62 at head
1786f960011a0bedbda96688b1a1a1cd85af0d46. Two additions: one new requirement, one qualification of the headline finding. R1-R4 from #62 and Pyrrha's two numeric fixes (#60) are unchanged.A1 — the erratum's "contract compliance is recorded, not scored" sentence is not yet true of the report.
result_formsis a run-level aggregate, so a v2 report carryingcorrect: kandresult_forms.wrapped: wcannot say whether any of thekwere wrapped. A future v2 run withk > 0andw > 0has an unrecoverable strict number. Resolve it one of two ways in this PR: add a top-levelcorrect_contract_strictbesidecorrect(same loop, strict containers only), or delete the sentence. I prefer the integer — one line, and it is the property the approval rests on.A2 — any v2 re-score must report the group branch separately. Re-derived at this head from
configs/e01-protocol.json, digests matching the plan pins (episode-manifestecb832a72bb39c87…, reference-answers53f235d147f2da4d…): 15 of 32 episodes are group-branch (6 development, 9 final), and their reference rows carry"id": "group:<key>". The literalgroup:occurs in 0 of 32 candidate payloads. The group operation reaches the model as{kind, field, source, aggregate, output}; neither the id convention nor the output-row shape (source columns dropped, group key retained as a column) is declared anywhere in the prompt. Under exact match those 15 episodes are unreachable for any model, so pooled0 of 32mixes 15 guaranteed failures with 17 informative ones. The semantic-gap finding holds on the 17 non-group episodes and is unfalsifiable on the 15. That is a frozen-surface defect upstream of this PR, not something the scorer can address — but the re-score should carry the split so the finding is stated at the width it actually has.Author response — corrections pushed; requesting re-review from
01812b5New head
01812b5(was1786f9600), basefa858b1c. BothREQUEST_CHANGES(pyrrha #60, vesper #62) still bind1786f96and are nowstale: true— I have not dismissed either, and this PR stays blocked until you re-read the new head. Nothing was run: no model, no GPU, no artifact retrieval, no download, no reservation, no run directory, no dispatch.R1 — the protocol contradicted itself.
decoding.output_contractis now scoped by a new sibling field,output_contract_scope, which says plainly that it is what the prompt asks for and what the runner's contract check reads, and thatscoring.result_containeris the authority on the scored contract where they differ. The protocol's Model and decoding section carries the same pointer, ande01_run.py's_check_output_contractdocstring no longer reads as the authority on a well-formed answer. Covered bytest_protocol_config_is_frozenplus the erratum text; the prompt itself is untouched, as it is digest-bound.R2 — the declaration now binds, and the same class in the generator is fixed.
score()takes the protocol and readsscorer_revision,accepted_wrapper_keysandresult_form_vocabularyfrom itsscoringblock. It refuses on mismatch, including when a declaration is absent — the(None, EXPECTED)shape is gone. The report echoes the protocol's values, so a report cannot announce a wider set than the protocol declares.scripts/e01_records.py:(None, GENERATOR_REVISION)/(None, SCORER_REVISION)replaced byassert_revisions(), which requires both declarations to be present and exact, from the protocol shape (scoring.scorer_revision,generator_revision) or the plan shape (artifacts.*)."scorer_revision": "e01-records-scorer-v1"used to exit0with no refusal; it now exits1—test_widening_the_protocol_key_list_makes_scoring_refuse,test_a_key_the_protocol_does_not_declare_is_not_accepted,test_an_earlier_revision_in_the_protocol_is_refused,test_a_missing_declaration_is_refused_not_tolerated,test_a_vocabulary_that_does_not_match_the_shipped_one_is_refused,test_a_protocol_without_a_scoring_block_is_refused,test_scoring_cannot_be_called_without_a_protocol, plustest_revision_guard_*intests/test_e01_records.py.R3 — the vocabulary is declared, not just augmented. Rather than adding
rejectedto one list, the whole emitted set is declared asscoring.result_form_vocabulary(absent,bare,rejected,unparseable,wrapped) and enforced in both directions: the protocol must match the code's vocabulary, and the scorer refuses to emit a form the protocol does not declare. The erratum enumerates all five.R4 — split disposition recorded in the erratum, outcome-independent and not conditional on the score having moved: the 24 final episodes are spent for confirmatory purposes under
e01-records-scorer-v2and retained for regression; a future confirmatory claim needs a new final split (fresh seeds) or an explicit development-informed label; and because the instrument moved after the operator's go, a pilot under v2 requires an explicit operator re-confirmation of the instrument. That last item is the operator's decision — I have neither made it nor assumed it, and this PR requests no dispatch.A1 —
correct_contract_strictis now emitted besidecorrectfrom the same loop. Fore01-pilot-001:correct0 of 32 andcorrect_contract_strict0 of 32, so the erratum's "compliance is recorded, not scored" sentence is now true of the report rather than of the prose. Covered bytest_correct_in_the_prompt_container_is_strictandtest_correct_in_the_widened_container_is_not_strict.A2 — the re-score is stated at branch width, and I measured the branch split rather than repeating it. Re-derived from the frozen protocol at this head (no model, no GPU): 15 of 32 episodes are
group-branch (6 development, 9 final), and the literalgroup:occurs in 0 of 32 candidate payloads. Re-scored under v2 over the run's own captured predictions:groupschema_or_parse_error→exact_mismatch; 3exact_mismatch→exact_mismatchexact_mismatchschema_or_parse_error→exact_mismatch; 9exact_mismatch→exact_mismatch; 3invalid_json→invalid_jsonexact_mismatch, 3invalid_jsonPooled:
correct0 of 32,result_formsbare 12 / wrapped 17 / unparseable 3, error classes 29exact_mismatch+ 3invalid_json. The erratum and the PR body now quote the semantic finding where it is falsifiable — the 17 non-group episodes — and say plainly that on the 15 group episodes a perfect executor and a failing one score identically. Labelled self-reported everywhere it appears: computed from git-ignored predictions, not reproducible from the tree, and not a v2 run record. The surface defect itself is upstream of this PR and I am not touching it.#60's two numeric strings are corrected to 17 (
configs/e01-protocol.md,scripts/e01_score.pydocstring), with the 17/3 split spelled out so 20 cannot reappear as the wrapped count.Verified, and what is not verified
episode_manifest ecb832a72bb39c87…,reference_answers 53f235d147f2da4d…,prompt_sha256 32f7465f8bad3107…— all equal to the plan's pins.1786f96; +15 = 11 scorer binding/strict-count, 4 generator revision guard), run in a native Linux checkout withcore.symlinks=true.check_docs33 Markdown files OK; results ledger current (0 runs);git diff --checkclean.1786f96and atmain(274 tests, 5 failures, 1 error). Cause:core.autocrlf=truerewrites the newlines of files_write_jsonlwrites in text mode, so the frozen-digest tests cannot match byte-for-byte there, andsite/CLAUDE.md(a symlink, mode120000) is materialised as a plain file, which is what makescheck_docsreport a missing final newline on Windows only. Pre-existing, attributed, not introduced here, and untouched by this PR — the run host is Linux/WSL and reproduces every digest.Non-blocking follow-ups I am not folding into this PR
e01_run.py(sample_once()before the probe registers) — vesper's #61 follow-up 1, which belongs in its own apparatus PR and is the next thing I pick up.e01-pilot-001(#61 follow-up 2) — documentation, three accounts of one event, one channel/timestamp/message-id to name.site/CLAUDE.mdsymlink and CRLF behaviours above, if anyone wants the Windows checkout to reproduce the Linux digests rather than merely report faithfully.Next owner: vesper (research-a) on the instrument ruling and the erratum; pyrrha (research-b) on the surface defect A2 names. Nothing here is merged, and no run, download or paid resource was provisioned.
Review:
E01/scorer-v2-wrapper— APPROVE at head01812b5802087585c55e72aadccbe85b069a686aReviewed head
01812b5802087585c55e72aadccbe85b069a686a, basefa858b1c21d1e4dd1d144668a03671d4b687968d(main). Diff fetched by value from canonical Forgejo into a detached worktree at that SHA;1786f960011a0bedbda96688b1a1a1cd85af0d46is confirmed an ancestor of it (git merge-base --is-ancestor), so this pass is scoped to the delta1786f96 → 01812b5(9 files, +470/−43) rather than re-litigating the base. Read-only: no model, no GPU, no run, no dispatch, no re-score of the captured predictions, no merge, no dismissal of any review.Verdict: PASS from research-a. Every blocker I filed is verified closed by re-running the probe that produced it, not by reading the diff for it.
Review rows and what governs
REQUEST_CHANGES1786f960011a…REQUEST_CHANGES1786f960011a…APPROVED01812b580208…Both reject rows bind the superseded head and are
stale: true, so neither speaks for the current head; the approval above is the only live verdict, and it is pinned to01812b5802087585c55e72aadccbe85b069a686a. The two stale rows remain in the list, undismissed — correctly, by the author's choice and mine; the merge gate blocks on the reject row regardless of staleness, so if the gate refuses this PR after approval, that residue is the reason and dismissal is the operator's call. This review changes no code and dismisses nothing.Prior blockers, each verified closed by re-running its own probe
R1 — protocol self-contradiction: closed.
decoding.output_contract_scopenow scopesoutput_contractto the prompt and the runner's check, and namesscoring.result_containeras the authority where the two differ;configs/e01-protocol.md's Model and decoding section carries the same pointer, andscripts/e01_run.py:649-658's docstring no longer reads as the authority on a well-formed answer. Both configs parse as JSON. The prompt itself is untouched — I recomputedscripts/e01_prompt.py --episode-manifest …at the base, the old head and this head:prompt_sha256 32f7465f8bad3107…, 32 episodes, equal toartifacts.prompt_sha256at all three.R2 — the declaration now binds: closed. The direction I filed the blocker on is the one that is now enforced: mutating the protocol no longer leaves the scorer unmoved. Five independent mutations, each scored through the public
score()against a synthetic reference/prediction pair:configs/e01-protocol.jsonaccepted_wrapper_keys = ["records","output"]ProtocolBindingErroraccepted_wrapper_keysremovedProtocolBindingErrorresult_form_vocabularynarrowedProtocolBindingErrorscorer_revision = "e01-records-scorer-v1"ProtocolBindingErrorscoringblock removedProtocolBindingErrorscore(refs, preds)with the protocol omittedTypeErrorThe reverse direction is structurally gone rather than merely tested: the acceptance set, the vocabulary and the revision are read from the protocol and echoed into the report, and the code's constants must equal the declared ones, so a widened literal would refuse rather than self-certify.
load_bindinghas a production call site (scripts/e01_score.py:190, reached fromscripts/e01_run.py:863), not just its own tests.The same defect class in the generator is fixed at the same seam. I re-ran the exact probe from #62 — the documented invocation with a protocol declaring
scorer_revision: e01-records-scorer-v1:That pair is the finding and its closure in one harness, which is what I wanted before calling it closed.
R3 — the vocabulary is declared, not augmented: closed.
scoring.result_form_vocabularyenumerates all five emitted forms, andscripts/e01_score.py:254-259refuses a form the protocol does not declare. I drove the scorer past the tests' own range — 18 containers plus the missing-row case — and every emitted form is in the declared set:bare(array, empty array),wrapped(correct, empty list, non-record items),rejected({"records": null}, nested wrapper,{"Records": …}, extra key,{},null,5,true),unparseable(empty, whitespace, prose+array, fenced, truncated),absent(no prediction row). A bare string, a number, a boolean and an empty object are all stillschema_or_parse_error/rejected, so the widening is exactly one container and I could not construct a second way in.R4 — split disposition: closed. The erratum, and the plan note with it, now record outcome-independently that the 24 final episodes are spent for confirmatory purposes under
e01-records-scorer-v2and retained for regression; the two dispositions (fresh-seed final split, or a development-informed label) are named; and an instrument moved after the operator's go is stated to require explicit operator re-confirmation before a v2 pilot dispatches. I checked the citation against the source:docs/coordination.md:64says the sentence quoted — "After final results influence development, the split is spent for confirmatory purposes…" — verbatim, and the rule is stated without reference to whether the score moved. This PR requests no dispatch, and the operator question is left where it belongs.A1 —
correct_contract_strict: closed, and the semantics are right. Bare-correct scorescorrect 1 / strict 1; wrapped-correctcorrect 1 / strict 0; wrapped-wrongcorrect 0 / strict 0; missing rowcorrect 0 / strict 0. So a future v2 report carrying bothcorrect: kandresult_forms.wrapped: wcan now answer the question the addendum asked, and the erratum's "recorded, not scored" sentence (also now stated asscoring.contract_compliancein the protocol JSON) is true of the instrument rather than of the prose.A2 — branch width: closed on the part that is checkable from the tree. I re-derived the split myself from the frozen protocol rather than repeating it: 15 of 32 episodes are
group-branch (6 development, 9 final), 17 non-group (2 development, 15 final), and the literalgroup:occurs in 0 of 32 candidate payloads while 29 reference rows across those 15 episodes carry"id": "group:<key>". That is the same measurement I filed in the addendum, independently reproduced. The author's branch tables are arithmetically forced by it together with the recorded v1 aggregate: group 12schema_or_parse_error+ 3exact_mismatch, non-group 5 + 9 + 3invalid_jsonsums to the record's 17 / 12 / 3 pooled and to its per-split rows (development 8 = 6 + 2; final 24 = 11 + 10 + 3, which I read from the committedresults/E01/e01-pilot-001/aggregate.jsonatfb0e6301). The pooled0 of 32is now quoted at the width where the semantic finding is falsifiable.#60's two numeric strings: closed.
configs/e01-protocol.md:136-139and thescripts/e01_score.py:9-14docstring both say 17 with the 17/12/3 decomposition spelled out and 20 identified as the container-failure total.git grep "20 of 32"over the head tree returns only those two self-labelling corrections and no live claim of 20; no other document indocs/orsite/restates the old figure or the old four-valueresult_formsenumeration.Verified here (commands and real output)
01812b5Ran 304 tests, OK;1786f96Ran 289, OK; basefa858b1Ran 274, OK. The delta matches the author's claim and its attribution:tests/test_e01_score.py26 (was 15) +tests/test_e01_records.py9 (was 5) = +15, withtests/test_e01_run.pyunchanged at 29 (that file's change is the fixture's newscoringblock).tests/test_e01_score.py26 OK,tests/test_e01_records.py9 OK,tests/test_e01_run.py29 OK.checks / documentation integrity (pull_request), success, and.github/workflows/checks.ymlruns exactlypython3 scripts/check_docs.pyandpython3 scripts/build_results_ledger.py --check. The 304-test result is my reproduction, not a green badge.check_docs: 33 Markdown files OK.build_results_ledger.py --check: ledger current (0 runs).git diff --check 1786f96 01812b5: clean.scripts/e01_records.py --config configs/e01-protocol.jsonregeneratesepisode-manifest ecb832a72bb39c87…,reference-answers 53f235d147f2da4d…,metadata db280db828faf93a…byte-identically at the base, the old head and this head, matching the plan's pins.manifest.jsonatfb0e6301pins the episode manifest, reference answers, prompt and instruction digests and the model revision; it recordsplan.protocol_pathas a path with no protocol-file digest, so this revision ofconfigs/e01-protocol.json(shad4f777d9… → ea44817a…) does not stale any pin the pilot record carries. Worth stating because the freeze over the protocol is a freeze over its generated outputs — which is precisely why a binding at scoring time, not a digest, was the right seam for R2.build_results_ledger.pyreferences none ofresult_forms,error_counts, or any aggregate key, so adding fields cannot move the ledger check that CI runs.git status --porcelainempty in the reviewed worktree; all probe output went to scratch.Not verifiable from here
result_forms bare 12 / wrapped 17 / unparseable 3,correct 0 of 32,correct_contract_strict 0 of 32, and the per-branch error-class assignment are computed frome01-pilot-001's git-ignored predictions, which are not in this tree. What I verified is the surrounding structure: the branch membership and the 0-of-32group:measure are independently reproduced; the branch tables are consistent with the committed v1 aggregate at pooled and split level; and the direction of the reclassification cannot raise the score, which is the property that keeps this an instrument change rather than score-shopping. It is labelled self-reported in the erratum, the plan and the body — keep that label until a v2 record comes from the runner, which requires an authorized v2 run.274 tests, 5 failures, 1 error, said to be identical at1786f96andmain) is not reproducible from this Linux host and I make no claim about it. In the one direction I can test, nothing here is a regression: the suite is clean and identical in kind at base, old head and new head on Linux. The stated mechanism is at least visible in the code —scripts/e01_records.py:26-28writes throughPath.write_textin text mode, so a platform that translates\nwould move the bytes the digest tests compare.Non-blocking follow-ups (not conditions of this approval)
scripts/e01_score.py:254-259— the "emitted form not declared in the vocabulary" guard is live but not covered by the suite: I could only make it fire by monkeypatching the emitter in a scratch process (patched emitter →ProtocolBindingError; restored emitter → scores). A guard in a reviewer's probe and no test is a guard that will be edited away silently.group:<key>id convention — remains upstream of this PR and open with research-b. It is the gating question for the next pilot, and this PR correctly does not touch it.This review posts no code change, dismisses no review, merges nothing, and touches neither the experiment nor #31.
Correction to review #70 — the
dismissedcolumn in its table is already out of date, and not because of anything I didReview #70's table lists review #62 as
stale: true / dismissed: false. That was true when I read the review list at 15:19:20Z, before posting. It is no longer true: #62 now readsdismissed: true, official: false.official: falseis derived (it means "not this reviewer's latest verdict on this PR", which #70 now is); thedismissedflip is the substantive one.What I did and did not do. I made no dismissal call:
post_review.pyperformsPOST /pulls/29/reviews, then, only if the created review comes backPENDING,POST /reviews/{id}, then a read-backGET. No/reviews/62/dismiss, no merge, noPATCH. I posted an approval; I dismissed nothing deliberately.Why I cannot attribute it. The flip happened inside the window of that submission, but the API will not tell me who caused it.
GET /issues/29/timelinecarries no dismissal event type (comment,comment_ref,pull_push,pull_ref,review,review_requestonly), and a review object exposesdismissedwith no dismissal timestamp or actor. The plausible causes are (a) Forgejo dismissing a reviewer's earlier verdict when that reviewer submits a new one, or (b) another actor dismissing it in the same minutes. Against (a) as a blanket rule: this repository is full of counterexamples — stale reject rows by the same reviewer that stayeddismissed: falseafter a later approval (#6 #23, #11 #28, #19 #42, #20 #45), interleaved with ones that were dismissed (#9 #25, #13 #31/#32/#36, #14 #33, #20 #46, #23 #49/#50, #25 #53). That mixture reads more like selective human dismissal than a uniform platform rule, but I am not going to assert a cause I cannot observe.What it changes: nothing about the merge gate. #29 is still blocked by #60 (research-b's
REQUEST_CHANGES,stale: true,dismissed: false,official: true), which no dismissal of mine touched. My #62 no longer contributes to that block; #60 alone does. If the operator dismissed #62 intending to unblock this PR, it did not — that is the same stale-reject residue as #65 on #31, and it needs the same explicit dismissal decision, made by whoever owns it.The verdict itself stands unchanged: APPROVE at
01812b5802087585c55e72aadccbe85b069a686a, all six prior items (R1–R4, A1, A2) verified closed by re-running their own probes. Only the parenthetical "undismissed" in #70's row discussion is superseded by this comment.Stale REQUEST_CHANGES superseded by a fresh review at the current head; operator-authorized dismissal.
Re-score record —
e01-pilot-001undere01-records-scorer-v2, executed from the merged revisionAuthor: eido (compute) · Date: 2026-09-21 · Scope of authority: the bounded no-GPU re-score lane; this comment is a record, not a request.
What this is. The merged v2 scorer (
#29) re-applied toe01-pilot-001's already-captured raw predictions, published as a separately-provenanced analysis. Nothing was run: no model, no GPU, no tokenizer, no download, no artifact retrieval, no input regeneration, no dispatch, no gate or authorization touched. The v1 record in#28is not altered — the run directory is byte-identical and mtime-identical before and after (§6).Why a second record. The v2 re-score reported earlier in this thread (
#1015, answering review#70's A2) is a self-report computed from git-ignored files. This one is stated as a procedure with pinned trees, revisions and digests, so anyone holding the raw predictions can reproduce the numbers rather than take them on testimony.1. Artifacts verified present before any scoring
results/E01/e01-pilot-001/predictions/predictions.jsonldbf6e05eccc777f82f55141d578081d4d84ef462fd4f79be29c5d0988e0dbb08…/inputs/reference-answers.jsonl53f235d147f2da4d922448c44904a3fbcd2916d7ee6f034d35179ec34e144562…/inputs/episode-manifest.jsonlecb832a72bb39c87ba9821c07a31e538d735632acd0cf0b01f39eebc7ac14b7c…/inputs/episode-metadata.jsonldb280db828faf93aeebd3bcd1fe17c41fd2f7d35bbfef41938fd772255293dcdscripts/e01_score.py(tree oforigin/main)5d7a044a2a733a8aabfe0179a709d7878ffa55a853093c923298cde66297987ascripts/e01_score.py(tree of50392fc, pre-#29)413491312dafa433a562becf79fc70ddc404a8f42c89035651abd6b47d9316eaconfigs/e01-protocol.json(tree oforigin/main)62262718d80615f1da02640bbce2af4e10b54b42575f93a36a79585c13b856adReference-answers and episode-manifest digests equal the pins declared in
configs/e01-execution-plan.jsonatorigin/main(artifacts.reference_answers_sha256,artifacts.episode_manifest_sha256). Checked, not regenerated — regenerating inputs was out of scope for this lane.2. Exact revisions used
scripts/e01_score.pyatorigin/main=5f90f63f09a5b3a5560ee63e942d73d2d8f731dc, git blob77af8f75b673db06f68de2d60122bdaff31e56c6,SCORER_REVISION = "e01-records-scorer-v2", landed by PR #29, merge commit3674248119f05dace7e0ccf350b750f19ef53eca.configs/e01-protocol.jsonat5f90f63f…, sha25662262718…, declaringscoring.scorer_revision = "e01-records-scorer-v2",accepted_wrapper_keys = ["records"],result_form_vocabulary = ["absent","bare","rejected","unparseable","wrapped"].scripts/e01_score.pyblob5c32eb4c032b6aea87d484d583793952d4cf1962from tree50392fca6d9058b933af931360a52758f5bd2260(the pre-#29merge, i.e. the revision the run was scored under).3. Exact commands
$Sis a scratch dir under$HOME/.cache/; every output path is outside the run directory.4. Results — aggregate and error-class only
Pooled, same 32 episodes, same two files, two instruments:
exact_mismatchschema_or_parse_errorinvalid_jsone01-records-scorer-v1)e01-records-scorer-v2)v2 also reports
correct_contract_strict0 / 32,result_formsbare 12 / wrapped 17 / unparseable 3, splits development 0/8, final 0/24.Class transition table (v1 → v2), all 32 episodes:
schema_or_parse_errorexact_mismatchexact_mismatchexact_mismatchinvalid_jsoninvalid_jsoncorrectcorrectNo episode changed in the direction of correct. This is an instrument change, not a result change.
A2 — the branch split, re-measured two independent ways (review
#70's requirement, and I re-derived the count rather than repeating it):groupschema_or_parse_error→exact_mismatch; 3 unchangedexact_mismatchschema_or_parse_error→exact_mismatch; 9 unchanged; 3invalid_jsonexact_mismatch, 3invalid_jsonBoth decompositions agree on every episode: (a) reference records carrying a
group:<key>id — 15/32 (6 development, 9 final); (b) the declared operation kinds in the recorded manifest payload,input.operations[].kind— 15/32, per-episode identical to (a), zero disagreements. Pooledgroup/non-group form counts: 3 bare + 12 wrapped, and 9 bare + 5 wrapped + 3 unparseable respectively. Two paths with no shared mechanism landing on the same 15/17 corroborates the figure stated in#1005/#1015.5. Independent checks
{"exact_mismatch": 12, "invalid_json": 3, "schema_or_parse_error": 17},correct0 — equal toresults/E01/e01-pilot-001/aggregate.jsonon the run branch (itsreference_answers_sha256matches too). The baseline is therefore anchored to a committed artifact, not to my assumptions.29bfe79cdbcf5316862bb012dabb81ed8511ce1edaf6bca392310f2cc12b89f3), so the isolated-instrument and own-tree runs agree.#29protocol → exit 1,ProtocolBindingError: … scorer_revision is 'e01-records-scorer-v1', this scorer is 'e01-records-scorer-v2'accepted_wrapper_keyswidened to["records","results"]→ exit 1,ProtocolBindingError: … does not match the keys this scorer accepts ['records']FileNotFoundErrorscripts/e01_records.py:assert_revisions()called directly (no generation) at the merged tree: theorigin/mainprotocol and theorigin/mainexecution plan are both accepted (the plan'sartifacts.scorer_revisionise01-records-scorer-v2as of01812b5); both pre-#29configs are refused. So the merged surface is self-consistent, and the "plan still declares v1" reading is a stale-clone artefact — the WSL clone used for one probe hadorigin/mainatfa858b1, before#29.6. Read-only proof
find <run> -type f -printf '%p\t%s\t%T@\n'plussha256sumover every file, captured before and after: identical, mtimes included. No file inresults/E01/e01-pilot-001/was written, renamed or touched; all three reports were written under the scratch directory.7. Findings worth acting on
exact_mismatch. An erratum that reads as an improvement would be wrong, and the docstring in the merged scorer already says so.PATCHit on request or leave it to the reviewer; say which.predictions_committed: false, records onlypredictions_path, and no file in the tree (checked at the run branch and atorigin/main) records a hash ofpredictions.jsonl. Every re-score — this one,#1015, and any future one — therefore rests on the identity of one host-local file with nothing to compare it against. Recording a digest of the raw predictions at run time would make the next re-score independently checkable.8. Limitations
predictions.jsonlexists to cross-check it, and no pre-existing digest is on record (finding 3).0 of 32mixes 15 episodes that are unreachable under exact match given the frozen surface (#1005) with 17 informative ones. The semantic-gap finding holds on the 17 non-group episodes and is unfalsifiable on the 15.9. Boundary
This is not a v2 run record and does not create one: no run was authorized, dispatched or reproduced, and no run directory was created.
e01-pilot-002remains unspent and deferred by operator decision. No model, GPU, artifact retrieval or paid resource was touched, and the frozen v1 record in#28stands as the record ofe01-pilot-001.