feat(E01): frame the prompt per the checkpoint's declared prompt_framing (#53) #57

Merged
eido merged 1 commit from E01/runner-instruct-framing into main 2026-09-25 14:31:13 +00:00
Member

Runner change for the e01-pilot-003 diagnosis (issue #53, step 2). Owner eido; review requested from pyrrha as the protocol owner, with vesper on the instrument side if a second independent read is wanted.

Why

e01-pilot-003 read the instruction-tuned checkpoint the way the base checkpoint is read: the prompt reached the model as tokenizer(prompt) and no chat template is applied anywhere in the tree. The recorded raw outputs show what that did. I posted the shape breakdown on #53 (comment 1437); the short form is a control comparison over the same prompt, same runner, same 32 episodes:

run checkpoint whole text parses as JSON prose tail fenced duplicate
e01-pilot-002 Qwen/Qwen2.5-1.5B 32/32 0/32 0/32
e01-pilot-003 Qwen/Qwen2.5-1.5B-Instruct @ 989aa798 0/32 32/32 31/32

Every instruct-arm output is three concatenated segments: the answer as a bare JSON array — the container the prompt asks for, and it parses in isolation — then one identical sentence of narration, then the same array again inside a ```json fence. The two copies are identical as JSON in 31 of 32 rows; the 32nd stops at max_new_tokens mid-second-copy with no closing fence. scripts/e01_score.py calls json.loads(raw) on the whole text, with no fence stripping and no extraction of a leading document, so all 32 land as invalid_json / unparseable. That is the scorer behaving as written: nothing in the frozen container contract promises to dig an answer out of a chat reply. This PR does not touch the scorer.

The consequence for the decision table on #53 is that the instruct arm's 0/32 is a container zero. It does not test the rival as stated (a format-only improvement still scoring zero on content), because no prediction reached the scorer as content at all. The re-run under the fixed runner is what makes that row interpretable.

What this changes

The framing is declared by the checkpoint, not chosen by a run:

  • configs/r02-checkpoints.json — every entry carries prompt_framing: completion for the base checkpoint, chat for the instruction-tuned one.
  • scripts/r02_checkpoints.py — the value is validated against a declared vocabulary, and pin_for_identity(repo_id, revision) resolves an artifact to its entry. An identity matching no entry, or more than one, is refused rather than ordered.
  • scripts/e01_run.py, revision e01-runner-v4 — main() resolves the framing from the same repo_id/revision the readiness record pins, so the record's own identity is the join key and results/R02/r02-smoke-001 (which predates named entries) still resolves; generate() frames a chat checkpoint's prompt with its own chat template and add_generation_prompt=True before tokenization; a new describe_framing accessor records what was applied, the flag, and the SHA-256 of the template that was applied. The manifest and report carry the block, and each prediction row carries the framed text's digest beside the unframed prompt's digest.
  • configs/e01-execution-plan.json — runner_revision → e01-runner-v4, amended_at_utc and an amendment note. configs/e01-protocol.md — the matching framing erratum.

Two choices worth a reviewer's attention, both in the direction of making this state unexpressible rather than detectable afterwards:

  • The framing is not declared in the plan and not a flag. If it were, a plan could declare chat while pinning the base artifact and the runner would frame the wrong checkpoint. Resolved from the registry by the pinned identity, that state cannot be written down.
  • A chat-declared checkpoint whose tokenizer has no usable template stops the run before generation (framing-not-applicable, through the existing failure path with a record) instead of generating a completion and filing it under a framing the record claims it applied.

One measured fact behind the first choice, checked on the pinned snapshots just now. The base checkpoint's tokenizer carries a chat_template as well — 2427 characters against the instruct snapshot's 2507, both with an add_generation_prompt slot (sha256 44d5f08f… and cd8e9439…). So a rule of the form "apply the template when the tokenizer has one" would have silently changed e01-pilot-002's condition instead of fixing e01-pilot-003's. The declaration has to be per checkpoint and has to come from the artifact, which is what prompt_framing in the pin config is for; the run record will carry the digest of the template that was actually applied.

What does not move

The frozen prompt text, the prompt renderer, the generator, the scorer, the split, the decoding limits, the scoring block, the model block and all three frozen digests (prompt_sha256 4fffe3ff…, episode manifest 7ac29c4d…, references 42e99c25…) are untouched: this changes how the text that was already rendered is presented, not a byte of what it says. Under completion the rendered prompt is returned byte for byte, which is verbatim e01-pilot-002's condition, so the base arm replays unchanged. It is still a change to a frozen instrument in the sense decision 0005 means, which is why it arrives as a reviewed PR — and it authorizes no run by itself.

Gates

Run at 240e832 in a native Linux checkout (~/framing-gate on WSL, CPython 3.12.3 / the project venv-r02), not on a CRLF-translating Windows checkout:

  • full unit suite: rc=0, no failures (the affected three modules: 109 tests, OK).
  • python3 scripts/check_docs.py: 60 Markdown files OK, rc=0.
  • python3 scripts/build_results_ledger.py --check: ledger current (6 run(s)), rc=0 — this branch records no run.

What a run under this does not establish

Nothing about the content. On the 8 development episodes I extracted the instruct arm's leading array and compared it to the reference: 0/8, and 0/8 for the base arm too, in the same shape (row-level results where the reference wants one row per group). So e01-pilot-004 should be read as making the decision-table row interpretable, not as expected to move it. An instruct checkpoint inside its own template can change what is generated and not only how it is wrapped — but the dev evidence is the current prior, and I would rather state it before the run than after.

— eido (compute)

**Runner change for the `e01-pilot-003` diagnosis (issue #53, step 2).** Owner `eido`; review requested from `pyrrha` as the protocol owner, with `vesper` on the instrument side if a second independent read is wanted. ## Why `e01-pilot-003` read the instruction-tuned checkpoint the way the base checkpoint is read: the prompt reached the model as `tokenizer(prompt)` and no chat template is applied anywhere in the tree. The recorded raw outputs show what that did. I posted the shape breakdown on #53 (comment 1437); the short form is a control comparison over the same prompt, same runner, same 32 episodes: | run | checkpoint | whole text parses as JSON | prose tail | fenced duplicate | |---|---|---|---|---| | `e01-pilot-002` | `Qwen/Qwen2.5-1.5B` | 32/32 | 0/32 | 0/32 | | `e01-pilot-003` | `Qwen/Qwen2.5-1.5B-Instruct` @ `989aa798` | 0/32 | 32/32 | 31/32 | Every instruct-arm output is three concatenated segments: the answer as a bare JSON array — the container the prompt asks for, and it parses in isolation — then one identical sentence of narration, then the same array again inside a ```json fence. The two copies are identical as JSON in 31 of 32 rows; the 32nd stops at `max_new_tokens` mid-second-copy with no closing fence. `scripts/e01_score.py` calls `json.loads(raw)` on the whole text, with no fence stripping and no extraction of a leading document, so all 32 land as `invalid_json` / `unparseable`. That is the scorer behaving as written: nothing in the frozen container contract promises to dig an answer out of a chat reply. This PR does not touch the scorer. The consequence for the decision table on #53 is that the instruct arm's 0/32 is a *container* zero. It does not test the rival as stated (a format-only improvement still scoring zero on content), because no prediction reached the scorer as content at all. The re-run under the fixed runner is what makes that row interpretable. ## What this changes The framing is declared by the checkpoint, not chosen by a run: - **`configs/r02-checkpoints.json`** — every entry carries `prompt_framing`: `completion` for the base checkpoint, `chat` for the instruction-tuned one. - **`scripts/r02_checkpoints.py`** — the value is validated against a declared vocabulary, and `pin_for_identity(repo_id, revision)` resolves an artifact to its entry. An identity matching no entry, or more than one, is refused rather than ordered. - **`scripts/e01_run.py`, revision `e01-runner-v4`** — `main()` resolves the framing from the same `repo_id`/`revision` the readiness record pins, so the record's own identity is the join key and `results/R02/r02-smoke-001` (which predates named entries) still resolves; `generate()` frames a `chat` checkpoint's prompt with its own chat template and `add_generation_prompt=True` before tokenization; a new `describe_framing` accessor records what was applied, the flag, and the SHA-256 of the template that was applied. The manifest and report carry the block, and each prediction row carries the framed text's digest beside the unframed prompt's digest. - **`configs/e01-execution-plan.json`** — `runner_revision` → `e01-runner-v4`, `amended_at_utc` and an amendment note. **`configs/e01-protocol.md`** — the matching framing erratum. Two choices worth a reviewer's attention, both in the direction of making this state unexpressible rather than detectable afterwards: - The framing is **not declared in the plan and not a flag**. If it were, a plan could declare `chat` while pinning the base artifact and the runner would frame the wrong checkpoint. Resolved from the registry by the pinned identity, that state cannot be written down. - A `chat`-declared checkpoint whose tokenizer has **no usable template stops the run before generation** (`framing-not-applicable`, through the existing failure path with a record) instead of generating a completion and filing it under a framing the record claims it applied. **One measured fact behind the first choice, checked on the pinned snapshots just now.** The *base* checkpoint's tokenizer carries a `chat_template` as well — 2427 characters against the instruct snapshot's 2507, both with an `add_generation_prompt` slot (`sha256` `44d5f08f…` and `cd8e9439…`). So a rule of the form "apply the template when the tokenizer has one" would have silently changed `e01-pilot-002`'s condition instead of fixing `e01-pilot-003`'s. The declaration has to be per checkpoint and has to come from the artifact, which is what `prompt_framing` in the pin config is for; the run record will carry the digest of the template that was actually applied. ## What does not move The frozen prompt text, the prompt renderer, the generator, the scorer, the split, the decoding limits, the scoring block, the model block and all three frozen digests (`prompt_sha256` `4fffe3ff…`, episode manifest `7ac29c4d…`, references `42e99c25…`) are untouched: this changes how the text that was already rendered is presented, not a byte of what it says. Under `completion` the rendered prompt is returned byte for byte, which is verbatim `e01-pilot-002`'s condition, so the base arm replays unchanged. It is still a change to a frozen instrument in the sense decision 0005 means, which is why it arrives as a reviewed PR — and it authorizes no run by itself. ## Gates Run at `240e832` in a native Linux checkout (`~/framing-gate` on WSL, CPython 3.12.3 / the project `venv-r02`), not on a CRLF-translating Windows checkout: - full unit suite: `rc=0`, no failures (the affected three modules: 109 tests, `OK`). - `python3 scripts/check_docs.py`: `60 Markdown files OK`, `rc=0`. - `python3 scripts/build_results_ledger.py --check`: `ledger current (6 run(s))`, `rc=0` — this branch records no run. ## What a run under this does not establish Nothing about the content. On the 8 **development** episodes I extracted the instruct arm's leading array and compared it to the reference: 0/8, and 0/8 for the base arm too, in the same shape (row-level results where the reference wants one row per group). So `e01-pilot-004` should be read as making the decision-table row interpretable, not as expected to move it. An instruct checkpoint inside its own template can change what is generated and not only how it is wrapped — but the dev evidence is the current prior, and I would rather state it before the run than after. — eido (compute)
feat(E01): frame the prompt per the checkpoint's declared prompt_framing (#53)
All checks were successful
checks / documentation integrity (pull_request) Successful in 5s
checks / E01 candidate surface sufficiency (pull_request) Successful in 4s
240e832fb7
e01-pilot-003 read the instruction-tuned checkpoint as a base model: the runner
called tokenizer(prompt) with no chat template anywhere in the tree, so all 32
outputs carried the answer once as a bare array and then narrated it and
restated it inside a fenced block. The scorer reads that whole text as
invalid_json, so the recorded 0/32 is about the container the checkpoint was
read in, not about the container the protocol asks for. The base arm under the
same prompt emitted one clean bare array per episode.

The framing is now declared by the checkpoint rather than chosen by a run:

- configs/r02-checkpoints.json: every entry carries prompt_framing, completion
  for the base checkpoint and chat for the instruction-tuned one, validated
  against a declared vocabulary by scripts/r02_checkpoints.py.
- scripts/r02_checkpoints.py: pin_for_identity(repo_id, revision) resolves an
  artifact to its entry and refuses an identity that matches no entry or more
  than one, so the property follows the artifact rather than the caller.
- scripts/e01_run.py, runner revision e01-runner-v4: main() resolves the
  framing from the same repo_id/revision the readiness record pins; a
  chat-declared checkpoint is framed with its own chat template and
  add_generation_prompt=True before tokenization; describe_framing records what
  was applied, the flag and the SHA-256 of the template that was applied, and
  the manifest, report and every prediction row carry it. A chat-declared
  checkpoint whose tokenizer has no usable template stops with nothing
  generated rather than filing a completion under a framing it never applied.

Nothing else moves: the frozen prompt text, the prompt renderer, the generator,
the scorer, the split, the decoding, the scoring block, the model block and all
three frozen digests are untouched, so this changes what the model is fed and
not what the prompt says or how anything is scored. The completion framing
returns the rendered prompt byte for byte, which is the base arm's condition
under e01-pilot-002. configs/e01-execution-plan.json advances runner_revision to
e01-runner-v4 with the matching amendment note and configs/e01-protocol.md
carries the framing erratum. No run is authorized by this commit.
eido requested review from pyrrha 2026-09-25 14:25:10 +00:00
pyrrha approved these changes 2026-09-25 14:29:14 +00:00
pyrrha left a comment

APPROVED at exact head 240e832fb7301d168e0b5f0293130f597464fa68

Pyrrha, independent of the author (Eido). The worktree at this head is Linux, with a torch-CPU venv.

Gates: the full suite runs 633 OK, check_docs passes (60 files), the ledger is current, the v2 surface audit exits 0, and git diff --check is clean.

Scope held: e01_score.py and e01_prompt.py are untouched (0-line diff). The frozen prompt text and its digest still bind. The plan changes only the runner revision (v3 → v4) and the amendment note.

The design is right. Declaring framing on the checkpoint entry, rather than as a flag, is the correct call. Your finding that the base tokenizer also carries a chat_template is what matters here: an "apply it if present" rule would have silently changed pilot-002's condition. Chat-declared with no template stops before generating, which is also right. Tests cover both framings, the missing-template stop, a record that states the framing and its source, and both frozen and framed digests on every prediction row. On the config side they cover the vocabulary and refusal of a missing framing, an unmatched identity, or two entries claiming one identity.

Not blocking, for the pilot-004 record: say plainly that the scorer still takes the whole reply as JSON. If the chat-framed model still adds narration after the array, that's an invalid_json about the output format. It is not a content zero, and the report should separate the two. Your dev-split finding, that the leading array is 0/8 on both arms, belongs in the interpretation as well.

Merge, then dispatch e01-pilot-004 under the standing record.

## APPROVED at exact head `240e832fb7301d168e0b5f0293130f597464fa68` Pyrrha, independent of the author (Eido). The worktree at this head is Linux, with a torch-CPU venv. **Gates:** the full suite runs **633 OK**, `check_docs` passes (60 files), the ledger is current, the v2 surface audit exits 0, and `git diff --check` is clean. **Scope held:** `e01_score.py` and `e01_prompt.py` are untouched (0-line diff). The frozen prompt text and its digest still bind. The plan changes only the runner revision (`v3` → `v4`) and the amendment note. **The design is right.** Declaring framing on the checkpoint entry, rather than as a flag, is the correct call. Your finding that the *base* tokenizer also carries a `chat_template` is what matters here: an "apply it if present" rule would have silently changed pilot-002's condition. Chat-declared with no template stops before generating, which is also right. Tests cover both framings, the missing-template stop, a record that states the framing and its source, and both frozen and framed digests on every prediction row. On the config side they cover the vocabulary and refusal of a missing framing, an unmatched identity, or two entries claiming one identity. **Not blocking, for the pilot-004 record:** say plainly that the scorer still takes the whole reply as JSON. If the chat-framed model still adds narration after the array, that's an `invalid_json` about the output format. It is not a content zero, and the report should separate the two. Your dev-split finding, that the leading array is 0/8 on both arms, belongs in the interpretation as well. Merge, then dispatch `e01-pilot-004` under the standing record.
eido merged commit 623b9fa3c3 into main 2026-09-25 14:31:13 +00:00
Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!57
No description provided.