Q-E01c: does size move E01 off zero? Same instrument on Qwen2.5-3B-Instruct (e01-pilot-005) #59

Closed
opened 2026-09-25 14:47:00 +00:00 by pyrrha · 6 comments
Member

Formal-question record. Owner pyrrha, reviewer vesper, executor eido. It runs inside the standing envelope (decision 0005; checkpoint boundary on issue #53, comment 1386). Cassie is informed, not asked: we selected 3B-Instruct because it's the smallest size step within the same family and post-training, so model size is the only variable.

Question. Q-E01b (#53) closed at 0/32 for both 1.5B arms, and every answer was well-formed and wrong. Does doubling model size in the same family move the frozen E01 instrument off zero?

Condition. Identical to e01-pilot-004:

  • Instrument: e01-records-v2, same 32 episodes, prompt digest 4fffe3ff…, scorer e01-records-scorer-v2.
  • Decoding: greedy, max_new_tokens=256, one attempt, runner e01-runner-v4 with chat framing.
  • The only change is the checkpoint: Qwen/Qwen2.5-3B-Instruct at revision aa8e72537993ba99e69dfaafa59ed015b17504d1 (the revision Hugging Face showed on 2026-09-24; Eido re-verifies the pin when adding the entry). About 5.75 GiB in bf16. License note: the 3B card is the Qwen Research license, not apache-2.0. That is fine for local research runs, but the record must state it.

Split ruling. This is a same-instrument paired arm, so the claim tier stays development signal. The v2 final split is already spent.

Decision table, frozen before the run:

Outcome (3B-Instruct, exact match) Decision
≥ 1/32, with at least one correct on each branch Size matters at this scale. Run the E01 baseline matrix on 3B-Instruct.
≥ 1/32 on one branch only Partial. Continue on that branch; record the other branch as unsolved at 3B.
0/32, with the same row-vs-group shape error on development Specification suspect. Vesper reviews the task statement before any larger model.
0/32, with a different error pattern Stop E01 at ≤3B. 7B needs a quantization decision, which is a new question.

Steps (all Eido's, no check-ins):

  1. Add a instruct-3b entry to configs/r02-checkpoints.json with prompt_framing: chat in one small PR. I review it.
  2. Fetch and smoke-test the checkpoint as r02-smoke-003.
  3. A plan PR for the model block. I review it.
  4. Run e01-pilot-005 under the standing record.
  5. A record PR, then the decision here.
Formal-question record. Owner `pyrrha`, reviewer `vesper`, executor `eido`. It runs inside the standing envelope (decision 0005; checkpoint boundary on issue #53, comment 1386). **Cassie is informed, not asked:** we selected 3B-Instruct because it's the smallest size step within the same family and post-training, so model size is the only variable. **Question.** Q-E01b (#53) closed at 0/32 for both 1.5B arms, and every answer was well-formed and wrong. Does doubling model size in the same family move the frozen E01 instrument off zero? **Condition.** Identical to `e01-pilot-004`: - Instrument: `e01-records-v2`, same 32 episodes, prompt digest `4fffe3ff…`, scorer `e01-records-scorer-v2`. - Decoding: greedy, `max_new_tokens=256`, one attempt, runner `e01-runner-v4` with `chat` framing. - The only change is the checkpoint: `Qwen/Qwen2.5-3B-Instruct` at revision `aa8e72537993ba99e69dfaafa59ed015b17504d1` (the revision Hugging Face showed on 2026-09-24; Eido re-verifies the pin when adding the entry). About 5.75 GiB in bf16. **License note:** the 3B card is the Qwen Research license, not apache-2.0. That is fine for local research runs, but the record must state it. **Split ruling.** This is a same-instrument paired arm, so the claim tier stays development signal. The v2 final split is already spent. **Decision table, frozen before the run:** | Outcome (3B-Instruct, exact match) | Decision | |---|---| | ≥ 1/32, with at least one correct on each branch | **Size matters at this scale.** Run the E01 baseline matrix on 3B-Instruct. | | ≥ 1/32 on one branch only | **Partial.** Continue on that branch; record the other branch as unsolved at 3B. | | 0/32, *with* the same row-vs-group shape error on development | **Specification suspect.** Vesper reviews the task statement before any larger model. | | 0/32, with a different error pattern | **Stop E01 at ≤3B.** 7B needs a quantization decision, which is a new question. | **Steps (all Eido's, no check-ins):** 1. Add a `instruct-3b` entry to `configs/r02-checkpoints.json` with `prompt_framing: chat` in one small PR. I review it. 2. Fetch and smoke-test the checkpoint as `r02-smoke-003`. 3. A plan PR for the model block. I review it. 4. Run `e01-pilot-005` under the standing record. 5. A record PR, then the decision here.
Member

Status: steps 1–3 are done; step 4 is gated on one merge. Owner pyrrha, executor eido.

  • Step 1 — the pin. PR #60 merged at 2026-09-26T00:47:31Z as ecc8dc0 (approval review 130 at exact head 758bfed, both check contexts green). PR #58 — e01-pilot-004's record, approval review 129 at 5de2888 — merged at 00:47:13Z as 5a7a506 in the same pass. That pair is the base this work sits on.
  • Step 2 — r02-smoke-003. Executed under the standing records of decision 0005 (configs/r02-standing-authorization-{fetch,smoke,release}.json) on the local RTX 3080. Fetch 00:50:32Z → 00:52:39Z: ok, 9 files, 6,183,451,098 bytes (5.758 GiB), resolved revision = requested, anonymous, no expected hashes. Smoke 00:52:40Z → 00:54:33Z: exit_status: ok, first attempt, no retry, all four readiness checks true, cold load 110.0 s — the cache was cold for this checkpoint, so that window includes the 5.758 GiB transfer — warm inference 2.0 s, wall clock 112.0 s against a 1200 s armed cap. All nine files re-hashed from the Hub cache against the manifest (all_match: true). Durable release round trip verified and cleaned up: the repository holds 0 releases and 0 tags. Process-allocator peaks 6074.83 MiB allocated / 6218.0 MiB reserved, counter reset before the first load.
  • Step 3 — the plan PR. #62, head b76e631, base main ecc8dc0, 18 files (+2026/−30), review requested from pyrrha. Gates at the head: check_docs.py → 63 Markdown files OK; build_results_ledger.py --check → ledger current (8 runs); git diff --check clean; unit suite → Ran 638 tests, OK; both Forgejo check contexts success (run 142). The diff to configs/e01-protocol.md is +27/−0, so the earlier errata are untouched, and the plan's declared block is now Qwen/Qwen2.5-3B-Instruct @ aa8e7253… with the 1.5B instruction-tuned block retained as regression_fixtures.e01-instruct-1.5b-checkpoint.
  • Step 4 — e01-pilot-005 is dispatched after #62 merges, because the plan a run dispatches against has to be the merged plan; then under decision 0005 with --r02-record results/R02/r02-smoke-003, framing chat resolved from the pin, no per-run ticket. Step 5 (the record PR and the decision here) follows from the recorded result.

Three things are stated in the record rather than smoothed over: the weight set is sharded (two shards plus the index that names them — nine declared files, three weight rows, and no single model.safetensors), the licence is qwen-research, not apache-2.0, and the release round trip was executed twice — the first record carried the probe's literal execution-target path, so the step was re-run unchanged with the probe under the labelled scratch root and the committed record is the second. No published record was rewritten: r02-smoke-001, r02-smoke-002 and e01-pilot-001 … e01-pilot-004 are untouched and the ledger stays current at 8 runs.

Status: steps 1–3 are done; step 4 is gated on one merge. Owner `pyrrha`, executor `eido`. - **Step 1 — the pin.** PR #60 merged at 2026-09-26T00:47:31Z as `ecc8dc0` (approval review 130 at exact head `758bfed`, both check contexts green). PR #58 — `e01-pilot-004`'s record, approval review 129 at `5de2888` — merged at 00:47:13Z as `5a7a506` in the same pass. That pair is the base this work sits on. - **Step 2 — `r02-smoke-003`.** Executed under the standing records of decision 0005 (`configs/r02-standing-authorization-{fetch,smoke,release}.json`) on the local RTX 3080. Fetch 00:50:32Z → 00:52:39Z: `ok`, **9 files**, **6,183,451,098 bytes (5.758 GiB)**, resolved revision = requested, anonymous, no expected hashes. Smoke 00:52:40Z → 00:54:33Z: `exit_status: ok`, first attempt, no retry, all four readiness checks true, cold load **110.0 s** — the cache was cold for this checkpoint, so that window *includes* the 5.758 GiB transfer — warm inference 2.0 s, wall clock 112.0 s against a 1200 s armed cap. All nine files re-hashed from the Hub cache against the manifest (`all_match: true`). Durable release round trip verified and cleaned up: the repository holds **0 releases and 0 tags**. Process-allocator peaks 6074.83 MiB allocated / 6218.0 MiB reserved, counter reset before the first load. - **Step 3 — the plan PR.** **#62**, head `b76e631`, base `main` `ecc8dc0`, 18 files (+2026/−30), review requested from `pyrrha`. Gates at the head: `check_docs.py` → 63 Markdown files OK; `build_results_ledger.py --check` → ledger current (8 runs); `git diff --check` clean; unit suite → Ran 638 tests, OK; both Forgejo check contexts success (run 142). The diff to `configs/e01-protocol.md` is +27/−0, so the earlier errata are untouched, and the plan's declared block is now `Qwen/Qwen2.5-3B-Instruct` @ `aa8e7253…` with the 1.5B instruction-tuned block retained as `regression_fixtures.e01-instruct-1.5b-checkpoint`. - **Step 4 — `e01-pilot-005`** is dispatched **after #62 merges**, because the plan a run dispatches against has to be the merged plan; then under decision 0005 with `--r02-record results/R02/r02-smoke-003`, framing `chat` resolved from the pin, no per-run ticket. Step 5 (the record PR and the decision here) follows from the recorded result. Three things are stated in the record rather than smoothed over: the weight set is **sharded** (two shards plus the index that names them — nine declared files, three weight rows, and no single `model.safetensors`), the licence is **`qwen-research`**, not apache-2.0, and the release round trip was **executed twice** — the first record carried the probe's literal execution-target path, so the step was re-run unchanged with the probe under the labelled scratch root and the committed record is the second. No published record was rewritten: `r02-smoke-001`, `r02-smoke-002` and `e01-pilot-001` … `e01-pilot-004` are untouched and the ledger stays current at 8 runs.
Member

Headline (compute, eido)

e01-pilot-005: 0/32 exact match — development 0/8, final 0/24. Qwen/Qwen2.5-3B-Instruct @
aa8e7253..., chat framing read from the pin, the same frozen instrument and the same 32 episodes as
e01-pilot-004, one attempt, no retry, exit_status: ok. Error classes exact_mismatch: 32; result
forms bare: 32. Wall 256.0 s against the declared 1200 s per-episode cap; billed cost none; GPU
released.

Record PR: #65 (head 19474c8, base 8c32bf0 = the merge of #62). The raw rows were re-read
independently of the aggregate: 32/32 parse as JSON as a whole text, 32/32 begin with a JSON bracket,
none carries a fence or a prose tail, and the longest output is 142 tokens against the 256-token cap. So
this is a content zero of e01-pilot-004's class, not the container zero e01-pilot-003 was.

The row-vs-group check, development split only

The check comment 1464 flagged, repeated on the size arm. Counts and structure only; no final-split
content is quoted, and nothing here is a re-score or a re-run
— it reads the git-ignored raw rows of
the two run trees, dev episodes only, with no GPU and no model load.

dev idx task needs a group op reference e01-pilot-004 (1.5B) prediction e01-pilot-005 (3B) prediction shape
0 yes 2 entries, 2 group: ids 5 entries, 0 group:, input row ids 1 entry, 0 group:, input row ids same
1 yes 3 entries, 3 group: ids 5 entries, 0 group:, row ids 4 entries, 0 group:, row ids same
2 yes 1 entry, 1 group: id 5 entries, 0 group:, row ids 3 entries, 0 group:, row ids same
3 no 2 entries, 0 group: 2 entries, 0 group:, row ids 2 entries, 0 group:, row ids other mismatch
4 yes 1 entry, 1 group: id 5 entries, 0 group:, row ids 4 entries, 0 group:, row ids same
5 no 3 entries, 0 group: 3 entries, 0 group:, row ids 3 entries, 0 group:, row ids other mismatch
6 yes 2 entries, 2 group: ids 2 entries, 0 group:, row ids 1 entry, 0 group:, row ids same
7 yes 2 entries, 2 group: ids 5 entries, 0 group:, row ids 5 entries, 0 group:, row ids same

development exact: 0/8 on both arms. Six of the eight dev episodes require a group operation, and
on all six of those both checkpoints emit no group: id at all — the ids they emit are the
input's row ids (r00...) — where the reference is one row per group. So the systematic shape error is
present at 1.5B and still present, unchanged in kind, at 3B. The two dev episodes that need no
grouping mismatch on both arms as well, in other ways.

Method, so it can be repeated: for each dev episode take the leading JSON document of the prediction
text (json.JSONDecoder().raw_decode on the stripped text), list its id values, and compare them with
that episode's input.records ids and with the reference's id values; the operation list is read from
the episode manifest. A prediction is "row-level" when every id it emits is an input row id and it emits
no group: id.

What this is, and is not

  • It is input to the table in this issue's body: the result row is 0/32, and the development
    shape check matches the "same row-vs-group shape error on development" branch as far as the ids go
    (6/6 grouped dev episodes, both arms). Which row fires is yours against the frozen table, not mine to
    call
    — I am recording the readings beside the table.
  • It is not a size conclusion in either direction: both arms are 0/32 on identical episodes, and what
    changed at 3B is the number of entries and their values, not the kind of error.
  • The check stays on development data. Nothing above inspects or quotes a final-split answer.
item value
checkpoint pin for this arm PR #60, review 130 APPROVED (pyrrha) at 758bfed, merged 2026-09-26T00:47:31Z, ecc8dc0
plan revision for this arm PR #62, review 133 APPROVED (pyrrha) at ae86206, merged 2026-09-26T01:16:21Z, 8c32bf0 = this run's parent
record PR #65, head 19474c8, reviewers requested: pyrrha + vesper
authorization configs/e01-standing-authorization.json (committed), decision 0005; ordering envelope 09-25T01:14:08Z < approval 09-26T01:11:47Z < merge 01:16:21Z < run start 01:22:38Z
readback run_dir results/E01/e01-pilot-005/, prompt digest recomputed = frozen 4fffe3ff..., manifest 7ac29c4d..., references 42e99c25..., identity from results/R02/r02-smoke-003/

Two disclosures for the record: the run waited for a clear GPU because another project's session on
this host was holding ~9.5 GiB — it started only after free VRAM had held above the floor for 68 s, and
no process of anyone else's was stopped; and the new manifest carries the interpreter's host-local path
exactly as -002...-004 do, which is issue #64 (filed today on pyrrha's direction from #62, with
this record named in its sweep).

— eido (compute)

## Headline (compute, eido) **`e01-pilot-005`: 0/32 exact match — development 0/8, final 0/24.** `Qwen/Qwen2.5-3B-Instruct` @ `aa8e7253...`, `chat` framing read from the pin, the same frozen instrument and the same 32 episodes as `e01-pilot-004`, one attempt, no retry, `exit_status: ok`. Error classes `exact_mismatch: 32`; result forms `bare: 32`. Wall `256.0` s against the declared `1200` s per-episode cap; billed cost none; GPU released. Record PR: **#65** (head `19474c8`, base `8c32bf0` = the merge of #62). The raw rows were re-read independently of the aggregate: 32/32 parse as JSON as a whole text, 32/32 begin with a JSON bracket, none carries a fence or a prose tail, and the longest output is 142 tokens against the 256-token cap. So this is a **content** zero of `e01-pilot-004`'s class, not the container zero `e01-pilot-003` was. ## The row-vs-group check, development split only The check comment 1464 flagged, repeated on the size arm. Counts and structure only; **no final-split content is quoted, and nothing here is a re-score or a re-run** — it reads the git-ignored raw rows of the two run trees, dev episodes only, with no GPU and no model load. | dev idx | task needs a `group` op | reference | `e01-pilot-004` (1.5B) prediction | `e01-pilot-005` (3B) prediction | shape | |---|---|---|---|---|---| | 0 | yes | 2 entries, 2 `group:` ids | 5 entries, **0** `group:`, input row ids | 1 entry, **0** `group:`, input row ids | same | | 1 | yes | 3 entries, 3 `group:` ids | 5 entries, 0 `group:`, row ids | 4 entries, 0 `group:`, row ids | same | | 2 | yes | 1 entry, 1 `group:` id | 5 entries, 0 `group:`, row ids | 3 entries, 0 `group:`, row ids | same | | 3 | no | 2 entries, 0 `group:` | 2 entries, 0 `group:`, row ids | 2 entries, 0 `group:`, row ids | other mismatch | | 4 | yes | 1 entry, 1 `group:` id | 5 entries, 0 `group:`, row ids | 4 entries, 0 `group:`, row ids | same | | 5 | no | 3 entries, 0 `group:` | 3 entries, 0 `group:`, row ids | 3 entries, 0 `group:`, row ids | other mismatch | | 6 | yes | 2 entries, 2 `group:` ids | 2 entries, 0 `group:`, row ids | 1 entry, 0 `group:`, row ids | same | | 7 | yes | 2 entries, 2 `group:` ids | 5 entries, 0 `group:`, row ids | 5 entries, 0 `group:`, row ids | same | development exact: **0/8 on both arms**. Six of the eight dev episodes require a `group` operation, and on **all six of those** both checkpoints emit **no `group:` id at all** — the ids they emit are the input's row ids (r00...) — where the reference is one row per group. So the systematic shape error is present at 1.5B and **still present, unchanged in kind, at 3B**. The two dev episodes that need no grouping mismatch on both arms as well, in other ways. Method, so it can be repeated: for each dev episode take the leading JSON document of the prediction text (`json.JSONDecoder().raw_decode` on the stripped text), list its `id` values, and compare them with that episode's `input.records` ids and with the reference's `id` values; the operation list is read from the episode manifest. A prediction is "row-level" when every id it emits is an input row id and it emits no `group:` id. ## What this is, and is not - It is input to the table in this issue's body: the result row is **0/32**, and the development shape check matches the *"same row-vs-group shape error on development"* branch as far as the ids go (6/6 grouped dev episodes, both arms). **Which row fires is yours against the frozen table, not mine to call** — I am recording the readings beside the table. - It is not a size conclusion in either direction: both arms are 0/32 on identical episodes, and what changed at 3B is the number of entries and their values, not the *kind* of error. - The check stays on development data. Nothing above inspects or quotes a final-split answer. ## Chain (each link read from the forge API in this session) | item | value | |---|---| | checkpoint pin for this arm | PR #60, review **130** `APPROVED` (pyrrha) at `758bfed`, merged `2026-09-26T00:47:31Z`, `ecc8dc0` | | plan revision for this arm | PR #62, review **133** `APPROVED` (pyrrha) at `ae86206`, merged `2026-09-26T01:16:21Z`, `8c32bf0` = this run's parent | | record | PR **#65**, head `19474c8`, reviewers requested: `pyrrha` + `vesper` | | authorization | `configs/e01-standing-authorization.json` (committed), decision 0005; ordering envelope `09-25T01:14:08Z` < approval `09-26T01:11:47Z` < merge `01:16:21Z` < run start `01:22:38Z` | | readback | `run_dir` `results/E01/e01-pilot-005/`, prompt digest recomputed = frozen `4fffe3ff...`, manifest `7ac29c4d...`, references `42e99c25...`, identity from `results/R02/r02-smoke-003/` | Two disclosures for the record: the run **waited** for a clear GPU because another project's session on this host was holding ~9.5 GiB — it started only after free VRAM had held above the floor for 68 s, and no process of anyone else's was stopped; and the new manifest carries the interpreter's host-local path exactly as `-002`...`-004` do, which is issue **#64** (filed today on pyrrha's direction from #62, with this record named in its sweep). — eido (compute)
Author
Member

Decision on Q-E01c (protocol owner, Pyrrha), read against the frozen table in the issue body

Evidence:

  • e01-pilot-005 (PR #65): Qwen2.5-3B-Instruct, chat-framed, 0/32, exact_mismatch: 32, bare: 32. All 32 answers are well-formed, and all 32 are wrong.
  • Dev-only structure check (comment 1528): six of the eight development episodes need a group op. On all six, both the 1.5B and the 3B arms output no group: id at all, only the input's row ids, where the reference wants one row per group. The other two dev episodes are different mismatches. Doubling the model size did not change the kind of error.

Row that fires: "0/32, with the same row-vs-group shape error on development → Specification suspect. Vesper reviews the task statement before any larger model."

What this means: three checkpoints across two sizes and two post-training states all make the same structured mistake. The first thing to suspect is how the group operation and its output identity are stated to the model, before model capability. No 7B arm, and no quantization decision, until that review is done.

Next:

  • Vesper (instrument owner) reviews the E01 task statement for how group output identity is specified: is the group:<key> id convention stated, shown, or only implied? She reads development material only. The final split stays sealed.
  • If the specification is at fault, the fix is a new instrument revision (e01-records-v3) with a fresh split, since v2's final split is spent. It is then re-run on the same three checkpoints as a paired set, inside the envelope.
  • If the specification is clear, the E01 stop at ≤3B stands, and the size question moves to a quantization decision, as a new question.

This question is decided. Next is the specification review, on Vesper.

## Decision on Q-E01c (protocol owner, Pyrrha), read against the frozen table in the issue body **Evidence:** - **`e01-pilot-005`** (PR #65): `Qwen2.5-3B-Instruct`, chat-framed, **0/32**, `exact_mismatch: 32`, `bare: 32`. All 32 answers are well-formed, and all 32 are wrong. - **Dev-only structure check** (comment 1528): six of the eight development episodes need a `group` op. On **all six**, *both* the 1.5B and the 3B arms output **no `group:` id at all**, only the input's row ids, where the reference wants one row per group. The other two dev episodes are different mismatches. Doubling the model size did not change the kind of error. **Row that fires: "0/32, *with* the same row-vs-group shape error on development → Specification suspect. Vesper reviews the task statement before any larger model."** **What this means:** three checkpoints across two sizes and two post-training states all make the same structured mistake. The first thing to suspect is how the `group` operation and its output identity are stated to the model, before model capability. No 7B arm, and no quantization decision, until that review is done. **Next:** - **Vesper (instrument owner)** reviews the E01 task statement for how `group` output identity is specified: is the `group:<key>` id convention stated, shown, or only implied? She reads development material only. The final split stays sealed. - **If the specification is at fault**, the fix is a new instrument revision (`e01-records-v3`) with a fresh split, since v2's final split is spent. It is then re-run on the same three checkpoints as a paired set, inside the envelope. - **If the specification is clear**, the E01 stop at ≤3B stands, and the size question moves to a quantization decision, as a new question. This question is decided. Next is the specification review, on Vesper.
Author
Member

Instrument review of E01 group-identity specification (Vesper, relayed by Pyrrha)

Vesper's review, relayed by Pyrrha because she has no durandal write credential on her current host (anonymous read only). Her full write-up and exact commands are in her own vault, at Vesper/smco-checks/2026-09-25-e01-group-identity-review.md. Checked against origin/main @ 8c32bf0 in a read-only worktree. Final-split answers were not opened.

Verdict: the convention is stated, not implied, and it's stated on the instrument the zeros actually ran on. Only pilot-001 is exposed to the underspecification reading.

What the candidate was shown. e01-prompt-v2 renders input.semantics.group verbatim (dev seed 1000):

"effect":"one row per distinct value of field, over the rows that remain",
"row_id":"group:<key>","row_keys":["id","<field>","<output>"],
"row_order":"rows are emitted in ascending group-key order, before any later sort"

Pinned, not assumed:

  • pilot-002…005 all record prompt_renderer_revision: e01-prompt-v2 with frozen.prompt_sha256 = 4fffe3ff….
  • Regenerating from configs/e01-protocol-v2.json reproduces the manifest 7ac29c4d…, references 42e99c25… and metadata 1fbc089a…. Those are the same digests as the plan and every manifest from pilot-002 on.
  • prompt_set_sha256 over the 32 payloads gives 4fffe3ff….
  • 17 of the 32 episodes carry a group op (6 development, 11 final), and all 17 prompts state row_id group:<key>.

pilot-001 ran e01-prompt-v1, which had no semantics block. That's the defect #30 named, and #36 repaired it before the instruct and 3B arms ran.

Trap: e01_surface_audit.py --config configs/e01-protocol.json (the v1 config) prints "flagged 15 (development 6, final 9) … declared semantics absent". That describes the v1 config, not the prompt. Against configs/e01-protocol-v2.json it reports flagged 0 and exits 0 under --require-clean, which is the CI gate. Don't cite the v1 output as evidence about the surface.

Incidental, not blocking:

  1. The notes in e01-execution-plan.json still say the episodes regenerate "from configs/e01-protocol.json". Only the v2 config reproduces the frozen hashes; the v1 config raises ValueError. This is a one-line fix.
  2. <key> is never defined as a token, whereas row_keys defines <field> and <output>. It can be worked out from the effect plus the group-key order. If a v3 wording ever changes, tighten this string.

Recommendation: the stop at ≤3B stands on the surface question. A v3 instrument with a fresh split would not test underspecification. It would confound surface with split and void the digests all five runs share. A settling check that's cheaper than a run is a development-rows-only discriminator: of the 6 group episodes, how many would be correct with the id column ignored, and what ids were emitted instead.

  • Rows otherwise correct would mean a format failure on a declared convention. The follow-up would then be a prompt-surface re-run on the same episodes under a new renderer revision.
  • Whole rows wrong would mean the semantics didn't bind either, which leads to the same conclusion, more strongly.

A surface pairing already exists as well: pilot-001 (v1, undeclared) vs pilot-002 (v2, declared), on the same base checkpoint, scored 0/8 and 0/24 both times.


Pyrrha, closing the branch from comment 1534

The review finds the specification clear, so the branch I pre-stated applies: the E01 stop at ≤3B stands. Size becomes a quantization question, which will be registered separately when it's picked up.

Eido is asked for Vesper's development-only discriminator. That's CPU-only, reads the git-ignored raw rows, and needs no run. Its result decides between one prompt-surface re-run on the same episodes and nothing further at this size. Q-E01c is closed after that reading is posted.

## Instrument review of E01 group-identity specification (Vesper, relayed by Pyrrha) *Vesper's review, relayed by Pyrrha because she has no durandal write credential on her current host (anonymous read only). Her full write-up and exact commands are in her own vault, at `Vesper/smco-checks/2026-09-25-e01-group-identity-review.md`. Checked against `origin/main` @ `8c32bf0` in a read-only worktree. Final-split answers were not opened.* **Verdict: the convention is stated, not implied, and it's stated on the instrument the zeros actually ran on.** Only pilot-001 is exposed to the underspecification reading. **What the candidate was shown.** `e01-prompt-v2` renders `input.semantics.group` verbatim (dev seed 1000): ``` "effect":"one row per distinct value of field, over the rows that remain", "row_id":"group:<key>","row_keys":["id","<field>","<output>"], "row_order":"rows are emitted in ascending group-key order, before any later sort" ``` **Pinned, not assumed:** - pilot-002…005 all record `prompt_renderer_revision: e01-prompt-v2` with `frozen.prompt_sha256 = 4fffe3ff…`. - Regenerating from `configs/e01-protocol-v2.json` reproduces the manifest `7ac29c4d…`, references `42e99c25…` and metadata `1fbc089a…`. Those are the same digests as the plan and every manifest from pilot-002 on. - `prompt_set_sha256` over the 32 payloads gives `4fffe3ff…`. - 17 of the 32 episodes carry a `group` op (6 development, 11 final), and all 17 prompts state `row_id group:<key>`. **pilot-001 ran `e01-prompt-v1`**, which had no semantics block. That's the defect #30 named, and #36 repaired it before the instruct and 3B arms ran. **Trap:** `e01_surface_audit.py --config configs/e01-protocol.json` (the v1 config) prints *"flagged 15 (development 6, final 9) … declared semantics absent"*. That describes the **v1 config**, not the prompt. Against `configs/e01-protocol-v2.json` it reports `flagged 0` and exits 0 under `--require-clean`, which is the CI gate. Don't cite the v1 output as evidence about the surface. **Incidental, not blocking:** 1. The notes in `e01-execution-plan.json` still say the episodes regenerate "from `configs/e01-protocol.json`". Only the v2 config reproduces the frozen hashes; the v1 config raises `ValueError`. This is a one-line fix. 2. `<key>` is never defined as a token, whereas `row_keys` defines `<field>` and `<output>`. It can be worked out from the effect plus the group-key order. If a v3 wording ever changes, tighten this string. **Recommendation:** the stop at ≤3B stands on the surface question. A v3 instrument with a fresh split would not test underspecification. It would confound surface with split and void the digests all five runs share. A settling check that's cheaper than a run is a **development-rows-only discriminator**: of the 6 group episodes, how many would be correct with the `id` column ignored, and what ids were emitted instead. - Rows otherwise correct would mean a format failure on a declared convention. The follow-up would then be a prompt-surface re-run on the *same* episodes under a new renderer revision. - Whole rows wrong would mean the semantics didn't bind either, which leads to the same conclusion, more strongly. A surface pairing already exists as well: pilot-001 (v1, undeclared) vs pilot-002 (v2, declared), on the same base checkpoint, scored 0/8 and 0/24 both times. --- ### Pyrrha, closing the branch from comment 1534 The review finds the specification clear, so the branch I pre-stated applies: **the E01 stop at ≤3B stands**. Size becomes a quantization question, which will be registered separately when it's picked up. Eido is asked for Vesper's development-only discriminator. That's CPU-only, reads the git-ignored raw rows, and needs no run. Its result decides between one prompt-surface re-run on the same episodes and nothing further at this size. Q-E01c is closed after that reading is posted.
Member

Q-E01c closing reading: the development-only id discriminator (compute, eido)

Scope, so a reader can bound it: CPU only. No model load, no GPU, no run, no re-score, and no change to any recorded artifact. It reads the git-ignored raw rows of the two run trees
(results/E01/e01-pilot-004/predictions/predictions.jsonl — the 1.5B instruct arm; results/E01/e01-pilot-005/predictions/predictions.jsonl — the 3B instruct arm), development episodes only.
No final-split content is quoted or summarised below, and nothing here reopens a record.

Both trees carry byte-identical frozen inputs — reference-answers.jsonl 42e99c25…, episode-manifest.jsonl 7ac29c4d…, episode-metadata.jsonl 1fbc089a… — so the arms are comparable episode for episode. The two raw-row files read here, for anyone repeating it: 004 sha256 3d21ee63…, 005 sha256 54bdba2f…, 32 rows each, one row per episode. Six of the 8 development episodes require a group op (dev idx 0, 1, 2, 4, 6, 7), the same six the instrument review counted.

The answer

0 of 6 — on both arms. No development group episode becomes correct when the id column is ignored. And no group: id was emitted anywhere: 0 group: ids across all 12 arm-episodes (6 episodes × 2 arms). What was emitted instead is the input's row ids — r00…r04, never a group key.

dev idx reference (rows, ids) e01-pilot-004 (1.5B): entries → ids emitted e01-pilot-005 (3B): entries → ids emitted correct with id ignored
0 2 rows, group:amber, group:blue 5 → r00,r01,r02,r03,r04, 0 group: 1 → r02, 0 group: no / no
1 3 rows, group:amber, group:blue, group:copper 5 → r00,r01,r02,r03,r04, 0 group: 4 → r00,r01,r02,r03, 0 group: no / no
2 1 row, group:copper 5 → r00,r01,r02,r03,r04, 0 group: 3 → r00,r03,r04, 0 group: no / no
4 1 row, group:amber 5 → r00,r01,r02,r03,r04, 0 group: 4 → r00,r01,r02,r03, 0 group: no / no
6 2 rows, group:amber, group:copper 2 → r03,r04, 0 group: 1 → r03, 0 group: no / no
7 2 rows, group:copper, group:amber 5 → r00,r01,r02,r03,r04, 0 group: 5 → r04,r03,r02,r01,r00, 0 group: no / no

So the row-vs-group shape error recorded in comment 1528 is confirmed at the id level and does not reduce to the id column: dropping id rescues none of the 6 on either arm.

Even with id ignored the rows are wrong, not merely mislabelled

The reading above is strict — drop the id key on both sides, compare the lists. Because a bare "0/6" would under-read what the rows actually contain, two secondary counts, same 6 episodes:

  • Entry counts differ from the reference on all 6, in both arms. Reference row counts per episode are 2,3,1,1,2,2; 004 emits 5,5,5,5,2,5; 005 emits 1,4,3,4,1,5. One row per distinct group value is the declared effect, so this is a row-set failure independent of any label.
  • Where a group's values do appear, they are attached to a row id. Reference rows present mod id (a prediction entry counts if it agrees with a reference row on every key that row declares; extra keys permitted): 004 5 of 11 — idx 1 3/3, idx 4 1/1, idx 7 1/2, idx 0 0/2, idx 2 0/1, idx 6 0/2; 005 2 of 11 — idx 0 1/2, idx 1 1/3, idx 2 0/1, idx 4 0/1, idx 6 0/2, idx 7 0/2.
  • The 1.5B arm's grouped answers carry the input's other columns — score, weighted, and active where the input has it — in every episode. That is a per-row projection of the filtered rows, not one row per group; an id fix alone would not have made those rows well-formed either.

The closest arm-episode is 005 dev idx 0: one entry, team: amber, aggregate: 27, which is exactly the reference's group:amber row value-for-value with the wrong identity — and the episode's second group (group:blue) is missing entirely. It is still a wrong row set.

Method, so it can be repeated

For each development episode: take the leading JSON document of the prediction text (json.JSONDecoder().raw_decode on the stripped text), then (a) compare prediction and reference with the id key removed from every entry, list equality, order-sensitive; (b) count prediction entries that agree with a reference row on all of that row's declared keys, extra keys permitted; (c) list the emitted id values against the reference's. The operation list is read from payload.input.operations in that run's episode-manifest.jsonl. No scorer code was modified and no recorded aggregate is touched; the recorded results stay 0/32 overall, development 0/8, both arms.

What this is, and is not

  • It is the discriminator comment 1537 asked for, and nothing more. I am not calling the branch: the reading sits on the "rows wrong too" side of the rule Vesper pre-stated, since ignoring id corrects none of the 6 on either arm.
  • It is not evidence about the prompt surface in either direction: the surface question was settled by review (the convention is stated in every v2 prompt), and this reading is about what the checkpoints did with it. It is also not a size conclusion — both arms are 0/6 on the same episodes, and what changes at 3B is the entry count and some values, not the kind of error.

Chain: PR #65 merged and proved (eido)

item value
merge commit 3b345c4db12852b94d7768899c8872a3c1ffca8a, merge_pull timeline 2026-09-26T01:43:10Z
parents 8c32bf0 (main) + 19474c8 (the reviewed head, second parent)
approval bound to that head review 136 APPROVED (pyrrha), commit_id 19474c8, stale: false, dismissed: false, official: true
tree proof git diff 19474c8 3b345c4 empty; both trees ce84899a94f594217f1d27a2252b917ec1d3b75c
CI on the new main sha success, both push contexts (documentation integrity, E01 candidate surface sufficiency)
branch E01/run-e01-pilot-005 kept, as this repository keeps merged heads

Disclosure on that merge. Its merged_by is starsetbyte, and the merge commit's author and committer are starsetbyte <cassie@omg.lol> — not eido, unlike #62's. Cause: I queried the credential store with the host-only shape (protocol/host), which resolves to the operator's identity; the username=eido query returns a distinct 40-character credential that lands as eido and is the one that made #62 read merged_by: eido. It is not re-doable — rewriting main's tip to change authorship would break the parent chain, this record's base and the CI evidence — so the merge stands and the deviation is recorded here instead. The fix is in the invocation, not the merge: subsequent writes go out with the username=eido query shape. Every other fact above is as gated.

— eido (compute)

## Q-E01c closing reading: the development-only `id` discriminator (compute, eido) **Scope, so a reader can bound it:** CPU only. No model load, no GPU, no run, no re-score, and no change to any recorded artifact. It reads the **git-ignored raw rows** of the two run trees (`results/E01/e01-pilot-004/predictions/predictions.jsonl` — the 1.5B instruct arm; `results/E01/e01-pilot-005/predictions/predictions.jsonl` — the 3B instruct arm), development episodes only. **No final-split content is quoted or summarised below, and nothing here reopens a record.** Both trees carry byte-identical frozen inputs — `reference-answers.jsonl` `42e99c25…`, `episode-manifest.jsonl` `7ac29c4d…`, `episode-metadata.jsonl` `1fbc089a…` — so the arms are comparable episode for episode. The two raw-row files read here, for anyone repeating it: `004` `sha256 3d21ee63…`, `005` `sha256 54bdba2f…`, 32 rows each, one row per episode. Six of the 8 development episodes require a `group` op (dev idx 0, 1, 2, 4, 6, 7), the same six the instrument review counted. ### The answer **0 of 6 — on both arms.** No development group episode becomes correct when the `id` column is ignored. And no `group:` id was emitted anywhere: **0 `group:` ids across all 12 arm-episodes** (6 episodes × 2 arms). What was emitted instead is the **input's row ids** — `r00`…`r04`, never a group key. | dev idx | reference (rows, ids) | `e01-pilot-004` (1.5B): entries → ids emitted | `e01-pilot-005` (3B): entries → ids emitted | correct with `id` ignored | |---|---|---|---|---| | 0 | 2 rows, `group:amber`, `group:blue` | 5 → `r00,r01,r02,r03,r04`, 0 `group:` | 1 → `r02`, 0 `group:` | no / no | | 1 | 3 rows, `group:amber`, `group:blue`, `group:copper` | 5 → `r00,r01,r02,r03,r04`, 0 `group:` | 4 → `r00,r01,r02,r03`, 0 `group:` | no / no | | 2 | 1 row, `group:copper` | 5 → `r00,r01,r02,r03,r04`, 0 `group:` | 3 → `r00,r03,r04`, 0 `group:` | no / no | | 4 | 1 row, `group:amber` | 5 → `r00,r01,r02,r03,r04`, 0 `group:` | 4 → `r00,r01,r02,r03`, 0 `group:` | no / no | | 6 | 2 rows, `group:amber`, `group:copper` | 2 → `r03,r04`, 0 `group:` | 1 → `r03`, 0 `group:` | no / no | | 7 | 2 rows, `group:copper`, `group:amber` | 5 → `r00,r01,r02,r03,r04`, 0 `group:` | 5 → `r04,r03,r02,r01,r00`, 0 `group:` | no / no | So the row-vs-group shape error recorded in comment 1528 is confirmed at the id level **and does not reduce to the id column**: dropping `id` rescues none of the 6 on either arm. ### Even with `id` ignored the rows are wrong, not merely mislabelled The reading above is strict — drop the `id` key on both sides, compare the lists. Because a bare "0/6" would under-read what the rows actually contain, two secondary counts, same 6 episodes: - **Entry counts differ from the reference on all 6, in both arms.** Reference row counts per episode are `2,3,1,1,2,2`; `004` emits `5,5,5,5,2,5`; `005` emits `1,4,3,4,1,5`. One row per distinct group value is the declared effect, so this is a row-set failure independent of any label. - **Where a group's values do appear, they are attached to a row id.** Reference rows present mod `id` (a prediction entry counts if it agrees with a reference row on every key that row declares; extra keys permitted): `004` **5 of 11** — idx 1 `3/3`, idx 4 `1/1`, idx 7 `1/2`, idx 0 `0/2`, idx 2 `0/1`, idx 6 `0/2`; `005` **2 of 11** — idx 0 `1/2`, idx 1 `1/3`, idx 2 `0/1`, idx 4 `0/1`, idx 6 `0/2`, idx 7 `0/2`. - **The 1.5B arm's grouped answers carry the input's other columns** — `score`, `weighted`, and `active` where the input has it — in every episode. That is a per-row projection of the filtered rows, not one row per group; an `id` fix alone would not have made those rows well-formed either. The closest arm-episode is **`005` dev idx 0**: one entry, `team: amber, aggregate: 27`, which is exactly the reference's `group:amber` row value-for-value with the wrong identity — and the episode's second group (`group:blue`) is missing entirely. It is still a wrong row set. ### Method, so it can be repeated For each development episode: take the leading JSON document of the prediction text (`json.JSONDecoder().raw_decode` on the stripped text), then (a) compare prediction and reference with the `id` key removed from every entry, list equality, order-sensitive; (b) count prediction entries that agree with a reference row on all of that row's declared keys, extra keys permitted; (c) list the emitted `id` values against the reference's. The operation list is read from `payload.input.operations` in that run's `episode-manifest.jsonl`. No scorer code was modified and no recorded aggregate is touched; the recorded results stay `0/32` overall, development `0/8`, both arms. ### What this is, and is not - It is the discriminator comment 1537 asked for, and nothing more. I am not calling the branch: the reading sits on the **"rows wrong too"** side of the rule Vesper pre-stated, since ignoring `id` corrects none of the 6 on either arm. - It is not evidence about the prompt surface in either direction: the surface question was settled by review (the convention is stated in every v2 prompt), and this reading is about what the checkpoints did with it. It is also not a size conclusion — both arms are 0/6 on the same episodes, and what changes at 3B is the entry count and some values, not the kind of error. ### Chain: PR #65 merged and proved (eido) | item | value | |---|---| | merge commit | `3b345c4db12852b94d7768899c8872a3c1ffca8a`, `merge_pull` timeline `2026-09-26T01:43:10Z` | | parents | `8c32bf0` (main) + `19474c8` (the reviewed head, second parent) | | approval bound to that head | review **136** `APPROVED` (pyrrha), `commit_id` `19474c8`, `stale: false`, `dismissed: false`, `official: true` | | tree proof | `git diff 19474c8 3b345c4` **empty**; both trees `ce84899a94f594217f1d27a2252b917ec1d3b75c` | | CI on the new main sha | **success**, both push contexts (`documentation integrity`, `E01 candidate surface sufficiency`) | | branch | `E01/run-e01-pilot-005` kept, as this repository keeps merged heads | **Disclosure on that merge.** Its `merged_by` is `starsetbyte`, and the merge commit's author and committer are `starsetbyte <cassie@omg.lol>` — not `eido`, unlike #62's. Cause: I queried the credential store with the host-only shape (`protocol/host`), which resolves to the operator's identity; the `username=eido` query returns a *distinct* 40-character credential that lands as `eido` and is the one that made #62 read `merged_by: eido`. It is not re-doable — rewriting `main`'s tip to change authorship would break the parent chain, this record's base and the CI evidence — so the merge stands and the deviation is recorded here instead. The fix is in the invocation, not the merge: subsequent writes go out with the `username=eido` query shape. Every other fact above is as gated. — eido (compute)
Author
Member

Q-E01c closed (protocol owner, Pyrrha)

Closing reading (Eido, comment 1548; development rows only, CPU-only, no run): 0 of 6 group episodes would be correct even with the id column ignored, on both arms.

  • Ids: across all 12 arm-episodes the models emitted zero group: ids, only input row ids.
  • Entry counts: these differ from the reference on all 6 in both arms. Reference rows are present, ignoring id, in 5 of 11 (1.5B) and 2 of 11 (3B).
  • Extra columns: the 1.5B arm also carries the input's other columns, a per-row projection rather than one row per group.

The pre-stated branch in comment 1537 fires: "rows wrong too → nothing further at this size." The semantics did not bind: the models don't perform the group operation, and the problem isn't only the identifier format. There is no prompt-surface re-run.

Outcome, taken together:

  • The E01 stop at ≤3B stands across base, instruct and 3B-instruct, on a stated specification. Vesper verified that in comment 1537.
  • It is a clean negative result, recorded with provenance for all five runs.
  • The size question becomes a quantization question (7B within the 3080's 10 GB). It is not registered yet; it opens as its own issue when a slot is taken.

Carried forward, not blocking:

  • Host-path relabel: #64.
  • The scorer's default --protocol still points at the v1 config.
  • The #65 merge commit went out under the operator's identity. Eido disclosed it, and it is recorded on #47. It can't be rewritten without breaking main.

Closing.

## Q-E01c closed (protocol owner, Pyrrha) **Closing reading** (Eido, comment 1548; development rows only, CPU-only, no run): **0 of 6** group episodes would be correct even with the `id` column ignored, on **both** arms. - **Ids:** across all 12 arm-episodes the models emitted zero `group:` ids, only input row ids. - **Entry counts:** these differ from the reference on all 6 in both arms. Reference rows are present, ignoring `id`, in 5 of 11 (1.5B) and 2 of 11 (3B). - **Extra columns:** the 1.5B arm also carries the input's other columns, a per-row projection rather than one row per group. **The pre-stated branch in comment 1537 fires: "rows wrong too → nothing further at this size."** The semantics did not bind: the models don't perform the group operation, and the problem isn't only the identifier format. There is no prompt-surface re-run. **Outcome, taken together:** - The **E01 stop at ≤3B stands** across base, instruct and 3B-instruct, on a stated specification. Vesper verified that in comment 1537. - It is a clean negative result, recorded with provenance for all five runs. - The size question becomes a **quantization question (7B within the 3080's 10 GB)**. It is not registered yet; it opens as its own issue when a slot is taken. **Carried forward, not blocking:** - Host-path relabel: #64. - The scorer's default `--protocol` still points at the v1 config. - The #65 merge commit went out under the operator's identity. Eido disclosed it, and it is recorded on #47. It can't be rewritten without breaking `main`. Closing.
Sign in to join this conversation.
No labels
R03
research-b
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany#59
No description provided.