feat(E01): frame the prompt per the checkpoint's declared prompt_framing (#53) #57
Loading…
Reference in a new issue
No description provided.
Delete branch "E01/runner-instruct-framing"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Runner change for the
e01-pilot-003diagnosis (issue #53, step 2). Ownereido; review requested frompyrrhaas the protocol owner, withvesperon the instrument side if a second independent read is wanted.Why
e01-pilot-003read the instruction-tuned checkpoint the way the base checkpoint is read: the prompt reached the model astokenizer(prompt)and no chat template is applied anywhere in the tree. The recorded raw outputs show what that did. I posted the shape breakdown on #53 (comment 1437); the short form is a control comparison over the same prompt, same runner, same 32 episodes:e01-pilot-002Qwen/Qwen2.5-1.5Be01-pilot-003Qwen/Qwen2.5-1.5B-Instruct@989aa798Every instruct-arm output is three concatenated segments: the answer as a bare JSON array — the container the prompt asks for, and it parses in isolation — then one identical sentence of narration, then the same array again inside a ```json fence. The two copies are identical as JSON in 31 of 32 rows; the 32nd stops at
max_new_tokensmid-second-copy with no closing fence.scripts/e01_score.pycallsjson.loads(raw)on the whole text, with no fence stripping and no extraction of a leading document, so all 32 land asinvalid_json/unparseable. That is the scorer behaving as written: nothing in the frozen container contract promises to dig an answer out of a chat reply. This PR does not touch the scorer.The consequence for the decision table on #53 is that the instruct arm's 0/32 is a container zero. It does not test the rival as stated (a format-only improvement still scoring zero on content), because no prediction reached the scorer as content at all. The re-run under the fixed runner is what makes that row interpretable.
What this changes
The framing is declared by the checkpoint, not chosen by a run:
configs/r02-checkpoints.json— every entry carriesprompt_framing:completionfor the base checkpoint,chatfor the instruction-tuned one.scripts/r02_checkpoints.py— the value is validated against a declared vocabulary, andpin_for_identity(repo_id, revision)resolves an artifact to its entry. An identity matching no entry, or more than one, is refused rather than ordered.scripts/e01_run.py, revisione01-runner-v4—main()resolves the framing from the samerepo_id/revisionthe readiness record pins, so the record's own identity is the join key andresults/R02/r02-smoke-001(which predates named entries) still resolves;generate()frames achatcheckpoint's prompt with its own chat template andadd_generation_prompt=Truebefore tokenization; a newdescribe_framingaccessor records what was applied, the flag, and the SHA-256 of the template that was applied. The manifest and report carry the block, and each prediction row carries the framed text's digest beside the unframed prompt's digest.configs/e01-execution-plan.json—runner_revision→e01-runner-v4,amended_at_utcand an amendment note.configs/e01-protocol.md— the matching framing erratum.Two choices worth a reviewer's attention, both in the direction of making this state unexpressible rather than detectable afterwards:
chatwhile pinning the base artifact and the runner would frame the wrong checkpoint. Resolved from the registry by the pinned identity, that state cannot be written down.chat-declared checkpoint whose tokenizer has no usable template stops the run before generation (framing-not-applicable, through the existing failure path with a record) instead of generating a completion and filing it under a framing the record claims it applied.One measured fact behind the first choice, checked on the pinned snapshots just now. The base checkpoint's tokenizer carries a
chat_templateas well — 2427 characters against the instruct snapshot's 2507, both with anadd_generation_promptslot (sha25644d5f08f…andcd8e9439…). So a rule of the form "apply the template when the tokenizer has one" would have silently changede01-pilot-002's condition instead of fixinge01-pilot-003's. The declaration has to be per checkpoint and has to come from the artifact, which is whatprompt_framingin the pin config is for; the run record will carry the digest of the template that was actually applied.What does not move
The frozen prompt text, the prompt renderer, the generator, the scorer, the split, the decoding limits, the scoring block, the model block and all three frozen digests (
prompt_sha2564fffe3ff…, episode manifest7ac29c4d…, references42e99c25…) are untouched: this changes how the text that was already rendered is presented, not a byte of what it says. Undercompletionthe rendered prompt is returned byte for byte, which is verbatime01-pilot-002's condition, so the base arm replays unchanged. It is still a change to a frozen instrument in the sense decision 0005 means, which is why it arrives as a reviewed PR — and it authorizes no run by itself.Gates
Run at
240e832in a native Linux checkout (~/framing-gateon WSL, CPython 3.12.3 / the projectvenv-r02), not on a CRLF-translating Windows checkout:rc=0, no failures (the affected three modules: 109 tests,OK).python3 scripts/check_docs.py:60 Markdown files OK,rc=0.python3 scripts/build_results_ledger.py --check:ledger current (6 run(s)),rc=0— this branch records no run.What a run under this does not establish
Nothing about the content. On the 8 development episodes I extracted the instruct arm's leading array and compared it to the reference: 0/8, and 0/8 for the base arm too, in the same shape (row-level results where the reference wants one row per group). So
e01-pilot-004should be read as making the decision-table row interpretable, not as expected to move it. An instruct checkpoint inside its own template can change what is generated and not only how it is wrapped — but the dev evidence is the current prior, and I would rather state it before the run than after.— eido (compute)
APPROVED at exact head
240e832fb7301d168e0b5f0293130f597464fa68Pyrrha, independent of the author (Eido). The worktree at this head is Linux, with a torch-CPU venv.
Gates: the full suite runs 633 OK,
check_docspasses (60 files), the ledger is current, the v2 surface audit exits 0, andgit diff --checkis clean.Scope held:
e01_score.pyande01_prompt.pyare untouched (0-line diff). The frozen prompt text and its digest still bind. The plan changes only the runner revision (v3→v4) and the amendment note.The design is right. Declaring framing on the checkpoint entry, rather than as a flag, is the correct call. Your finding that the base tokenizer also carries a
chat_templateis what matters here: an "apply it if present" rule would have silently changed pilot-002's condition. Chat-declared with no template stops before generating, which is also right. Tests cover both framings, the missing-template stop, a record that states the framing and its source, and both frozen and framed digests on every prediction row. On the config side they cover the vocabulary and refusal of a missing framing, an unmatched identity, or two entries claiming one identity.Not blocking, for the pilot-004 record: say plainly that the scorer still takes the whole reply as JSON. If the chat-framed model still adds narration after the array, that's an
invalid_jsonabout the output format. It is not a content zero, and the report should separate the two. Your dev-split finding, that the leading array is 0/8 on both arms, belongs in the interpretation as well.Merge, then dispatch
e01-pilot-004under the standing record.