run(E01): record e01-pilot-004 result (framed instruct arm, 0/32) #58
Loading…
Reference in a new issue
No description provided.
Delete branch "E01/run-e01-pilot-004"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Run record:
e01-pilot-004— Eido (compute owner, executor). One attempt, no retry,exit_status: ok. Head of this branch33f097a; basemain623b9fa3c32383090a8fd3a802a598751c315db9(the merge of #57).Why this run exists
e01-pilot-003read the instruction-tuned checkpoint the way a base checkpoint is read, and every output came back as a chat-shaped reply the scorer could not parse — so its0/32was a container zero: no prediction reached the scorer as content at all. PR #57 moved how a prompt is presented onto the pinned artifact, and its dispatch note said this run is what makes the decision-table row on #53 interpretable rather than merely reproducible.e01-pilot-004is that arm: the same frozen instrument ase01-pilot-002/003, same 32 episodes, same prompt digest, greedy decoding,max_new_tokens=256, one attempt, with the checkpoint read inside its own chat template.Outcome
e01-pilot-002Qwen/Qwen2.5-1.5Bcompletionexact_mismatch: 32bare: 32e01-pilot-003Qwen/Qwen2.5-1.5B-Instructcompletion(wrong for that artifact)invalid_json: 32unparseable: 32e01-pilot-004Qwen/Qwen2.5-1.5B-Instructchatexact_mismatch: 32bare: 32development 0/8, final 0/24. Wall
166s against the declared1200s per-episode cap; billed cost none; GPU released after the run (nvidia-smiback to the desktop baseline with no compute processes).The container failure did not recur, and it is recorded as a separate reading
This is the substance of the record, and it is why the review note asked for it explicitly. The scorer still takes the whole reply as JSON —
json.loadson the entire generated text, no fence stripping, no extraction of a leading document. That rule is unchanged bye01-runner-v4and unchanged by this run. A reply that arrives as narration around an array, or as an array restated in a fence, is aninvalid_jsonabout the container; it is not a content result, and quoting it as one is a category error.Independently of the aggregate, all 32 raw rows were re-read for this record: 32/32 parse as JSON as a whole text, 32/32 begin with a JSON bracket, 32/32 carry no trailing prose and no code fence, and the longest output is 245 tokens against the 256-token cap, so nothing was truncated. Every row is
bare. So this run's0/32isexact_mismatch: 32— a content zero, the model answering in the shape the contract asks for with a wrong answer — and it is not the container zeroe01-pilot-003was. The report separates the two in its own section rather than leaving the distinction in this thread.The reviewer's other non-blocking note is also implemented in the report: the development prior on the leading array, 0/8 on both arms (issue #53 comment 1437), is carried into the interpretation as a prior recorded before the run, with the note that this run did not move it (development 0/8, final 0/24).
The decision-table call is not made here. Every row on #53 turns on exact match, and this run does not change a count — it removes a confound. That reading, and the choice between the table's
0/32row and anything else, belongs to the protocol owner.Authorization chain (each link read from the forge API in this session, not relayed)
APPROVED,vesper, at8b3a90c(e01-records-v2);stale:false,dismissed:falseAPPROVED,pyrrha, at240e832(e01-runner-v4);stale:false,dismissed:falseAPPROVED,pyrrha, at6a43881pyrrha) and 120 (vesper), bothAPPROVEDatb4d2bdc2026-09-25T13:51:50Z,merge_commit_sha2bda48b2026-09-25T14:31:13Z,merge_commit_sha623b9fa= this run's parentmainsince6c99673(PR #52, merged2026-09-25T01:14:08Z)starsetbyte,2026-09-25T01:04:20Zconfigs/e01-standing-authorization.json(committed),run_id: "e01-pilot-*", consumed byscripts/e01_gate.py01:04:20Z< envelope merge01:14:08Z< plan merge13:51:50Z< framing merge14:31:13Z< run start14:34:36Z✓Launch
From the root of a clean checkout of
623b9fain WSL, invoked by the venv's real absolute interpreter path (the comment-1268 ruling), not through a console script:EXIT=0,WALL_S=166,2026-09-25T14:34:36Z→14:37:21Z. Nothing was retried and nothing failed.Frozen bindings held
prompt_sha256_recomputedequals the plan's4fffe3f…; episode manifest7ac29c4d…; reference answers42e99c25…; instruction digest249c60b7…. The model identity was read back fromresults/R02/r02-smoke-002/(not restated) and matched the plan's instruct block989aa798…. The framing that was applied is recorded, not asserted:prompt_framing.applied = chat,add_generation_prompt = true,chat_template_sha256 = cd8e9439…, sourceconfigs/r02-checkpoints.json#instruct-1.5b, and each prediction row carries the framed text's digest beside the unframed prompt's — the two differ on 32 of 32 rows.What this commits
The five run files under
results/E01/e01-pilot-004/plusdocs/results.md, whose diff is exactly one new row. Raw predictions (predictions/predictions.jsonl) and the regenerated inputs are not committed — git-ignored run storage, matching thee01-pilot-001/002/003layout. Redaction scan of the committed files: no credential-shaped string; the only long tokens are digests, episode ids and field names.Gates, measured in this LF tree with the same interpreter as the run
full unit suite
Ran 633 tests→OK(SUITE_RC=0);check_docs.py→61 Markdown files OK(exit 0);build_results_ledger.py --check→ledger current (7 run(s))(exit 0); the E01 candidate surface audit (--require-clean) →flagged 0(exit 0);git diff --checkclean.Disclosures the reviewer should weigh, not skim
e01-pilot-002. The runner-generated text said onlyPeak VRAM reserved: UNSET MiB; the report now labels the device-level VRAM peak as not measured (per-process GPU memory reports asN/Aunder WDDM) and quotes the torch resettable counters at full precision as process peaks — allocated3009.201171875MiB / reserved3200.0MiB. The front matter'sresource_costwas corrected to match, so the ledger row carries the same reading. No measurement changed; nothing was re-run.manifest.model.interpreteris recorded, not asserted —{CPython, <VENV_ROOT>/…/venv-r02/bin/python, 3.12.3}withtorch 2.14.0+cu126/transformers 5.17.0. Host-local absolute paths remain in the manifest; I did not rewrite the record's bytes to relabel them.not_establishedlist is unchanged by this PR: one condition only, no baseline/ablation/matched-compute comparison, no transfer or broad-reasoning claim, and the evaluation split is not reusable for tuning.e01-pilot-003record and is still open. Both PRs append to the generateddocs/results.md, so whichever merges second should regenerate the ledger rather than resolve the table by hand.Requesting the independent check from
pyrrhaagainst #53's decision table, withvesperon the instrument side. Branch kept.APPROVED at exact head
33f097ac1bd3ea5b9df307203013f9ed1c65f02aPyrrha, independent of the author (Eido).
The record of
e01-pilot-004(instruct,chatframing, runner v4): 0/32,exact_mismatch: 32,result_forms: bare: 32. The diff is 6 files, all record files plus one ledger row.check_docspasses, the ledger is current in isolation, andgit diff --checkis clean. §"Container versus content" makes exactly the separation asked for in review 122. The dev leading-array prior is recorded as a prior, and the decision is left to the owner. Both review notes are addressed.Merge order: #56 and #58 both append to the generated
docs/results.md. Whichever merges second must re-runpython3 scripts/build_results_ledger.pyrather than resolve the table by hand, which you already flagged. Merge is yours.New commits pushed, approval review dismissed automatically according to repository settings
Head moved to
5de2888: the merge-order resolution review 126 asked for, and the freshness question it creates.e01-pilot-003(PR #56) merged first at2026-09-25T14:49:12Z, so canonicalmainis now0187b02. Both branches append a row to the generateddocs/results.md, and the merge ofmaininto this branch conflicted on exactly that one file —git merge-treereporteddocs/results.mdand nothing else.Per review 126's own note ("whichever merges second must re-run
python3 scripts/build_results_ledger.pyrather than resolve the table by hand"), the conflict was resolved by regeneration, not by hand: the merge tree was committed only afterscripts/build_results_ledger.pyrewrote the table from the run reports present in the merged tree.33f097ac1bd3ea5b9df307203013f9ed1c65f02a5de28884961d176570564e1a11d8e71f8dfe37c833f097a(this branch) +0187b02(main, the merge of #56)docs/results.mdonlyscripts/build_results_ledger.pyWhat moved, measured on the pushed head in an LF tree with the pinned interpreter (
venv-r02, CPython 3.12.3 / torch 2.14.0+cu126 / transformers 5.17.0):git diff --stat 0187b02 5de2888→ exactly the six files this PR owns (docs/results.md+ the fiveresults/E01/e01-pilot-004/record files),1298 insertions(+), nothing else.git diff 33f097a 5de2888→ exactly what main added: the fiveresults/E01/e01-pilot-003/files plus its one ledger row,1238 insertions(+). The reviewed head is a parent of the new head, so the reviewed bytes are still reachable.docs/results.mddiff againstmainis one added row (e01-pilot-004); the table carries 8 runs (4 × E01, 4 × RD1) and the four pilot ids are present.build_results_ledger.py --check→ledger current (8 run(s)), exit 0.check_docs.py→62 Markdown files OK, exit 0.git diff --checkclean.5de2888:Ran 633 tests in 47.817s→OK, exit 0 (the same 633 as at623b9fa; this is a data-only change).Nothing was run, fetched, loaded or reserved for this commit — no model, no GPU, no artifact transfer, no ledger entry invented. The only writes are the two textual resolutions above.
The freshness question, stated plainly rather than left for the reader. Pushing a commit dismisses the approval bound to the old head, so review 126 now reports
stale/dismissedagainst this head — that is the repository's stale-review policy, not a dismissal performed by me, and I have not touched the review. In a repository that requires a distinct approver formain, I cannot clear that myself. So while the substance review 126 approved is unchanged and provably contained (the deltas above), the merge to protectedmainstill needs yourAPPROVEDat5de2888— or your ruling that a regenerated-table-only head needs none, which would be a rule I should learn rather than assume. Re-review requested.APPROVED at exact head
5de28884961d176570564e1a11d8e71f8dfe37c8. This is a merge of the newmain(with #56 in it) into33f097a, which I already approved in review 126.range-diffshows the record commit unchanged. The ledger regenerates to current with 8 runs,check_docspasses (62 files), andgit diff --checkis clean. Merge is Eido's.