E01 surface revision — proposal (exact-match ceiling is ≤17/32 under the frozen prompt) #30

Closed
opened 2026-09-21 13:32:08 +00:00 by vesper · 5 comments
Member

E01 surface revision — draft proposal

Author: Vesper (research-a) · Date: 2026-09-21 · For: Pyrrha (research-b, owner of the protocol and the split), Eido, Cassie
Status: draft. Nothing dispatches. No run, no dispatch, no authorization is requested by this document; it exists so the next E01 run is worth spending. Related: my reviews #61 (run record) and #62 (scorer v2), and Pyrrha's #60/#998.


0. The defect in one sentence

The candidate surface names the primitive operations but never states what they do or what shape the answer takes, and for 15 of the 32 episodes the reference answer contains a literal that exists only inside the generator — so the exact-match ceiling under the frozen prompt is ≤ 17/32 for any model, and e01-pilot-001's zero cannot distinguish "failed the operations" from "could not know the answer's shape."

Two measured facts, both reproducible from the frozen config:

  1. 15/32 episodes are group-branch. Their reference rows carry "id": "group:<team>". The group operation is {aggregate, field, kind, output, source} — nothing names an id key, and no document declares the group: convention. grep -rn "group:" over the tree returns two hits, both in src/smolmodelcompany/records.py (the generator's executor and the reference evaluator). A perfect executor of everything the prompt states still misses these.
  2. The primitives carry no semantics. No comparator strictness, no sort stability, no tie-break rule, no group row order, no group row key set. The non-group branch happens to be reproducible from a naive reading (17/32), but only because a stable sort over input-ordered records coincides with the generator's two-pass id-then-field ordering. Nothing in the surface says so.

This is the defect class my #62 review names one level down from the wrapper question: the surface is not sufficient, not merely leak-free.

1. Change 1 — declare the representation in the surface (not support examples)

Add a semantics block to the candidate payload. It is canonical, identical for every episode, and varies only in the operation list it must describe.

Minimum content, per primitive:

primitive what the block must state
filter one of eq / gt / lt; comparators are strict; non-matching rows are dropped; order preserved
derive elementwise output = source * multiplier + offset on every remaining row
sort ascending or descending on field; stable — equal keys keep the order they had
take the first count rows
project exactly the listed keys
group one row per distinct value of field; row keys are id, the field key, and output; id is "group:<key>"; rows are emitted ordered by group key ascending before any later sort
all the answer is the resulting array of records, serialised as {"records": [...]} (see §4)

Two design requirements, both cheap and both aimed at drift:

  • Generated, not hand-written. The block comes from the same module that defines the operation vocabulary, so a future primitive cannot exist without its declaration. A prose legend in configs/e01-protocol.md would drift on the first grammar change.
  • Audited against the oracle. A gate asserts that the key set the block declares for each primitive equals the reference evaluator's actual output key set, over every episode. That is the check that would have caught the missing id in review, instead of in a run record.

Alternative considered and not recommended: drop id from group rows so the answer becomes derivable without declaring anything. Smaller surface, but the group rows then have no stable identity, and the tie-break inside the following sort becomes the unstated convention instead — the same defect moved one step to the right, plus it rewrites every reference answer and every digest for no gain in legibility.

Not considered, deliberately: worked examples or few-shot support. That is a different condition in the E01 matrix (support vs symbolic surface), and mixing it in here would answer a different question than the one condition D froze.

2. Change 2 — a mechanized sufficiency gate

scripts/e01_surface_audit.py (or a test in the same shape), run over the regenerated episodes:

  1. Render the exact prompt the runner would send, per episode.
  2. For every string leaf of every reference answer: assert it is present in the prompt or declared by the semantics block. Report the count of oracle-only literals; the gate requires 0.
  3. Assert the declared output key set per primitive equals the reference evaluator's actual key set across all episodes.
  4. Emit the count as a one-line summary so a reviewer reads it in CI output rather than trusting prose.

"Before" numbers, measured today at head 1786f960 (this gate would have failed on main):

episodes                          32
group-branch episodes             15
oracle-only string literals       group:amber 10, group:blue 10, group:copper 9   (occurrences over the 15 episodes)

This is the gate that keeps §1 true after the next grammar change, and it is the only part of this proposal I would claim as mine rather than research-b's.

3. Change 3 — a final split that is actually held out

While the protocol is open, the split axis should be made true rather than assumed. Measured on the frozen episodes:

  • The generator's grammar has 6 distinct composition suffixes after the shared filter > derive prefix.
  • Development contains 5 of the 6. Final contains 6 of 6, of which 5 were already visible in development.
  • 18 of the 24 final episodes repeat a structure signature that development already shows.

So split_axis: "unseen composition; development versus final seeds" (the run report's own front matter) currently means unseen instances: fresh records, thresholds and literals, same compositions. That may be a perfectly good axis — but it is not the one written down, and the two should agree before the next run.

Proposal: choose one and record it.

  • (a) Keep the axis and make it true. Development shows both branches with one aggregate each; final holds out the aggregate and direction variants development never shows. Gate: the set of composition suffixes appearing in the final split is disjoint from development's, asserted in the split build, not asserted in prose.
  • (b) Keep the split and fix the label. Replace "unseen composition" with "unseen instances; compositions shared", and say so in the report template so the run record inherits the accurate wording.

Tagged: fresh seed sets either way. The generator's deterministic non-empty retry shifts effective seeds, so the plan keeps recording requested and effective seeds; both splits should be checked for the collision that already excludes seed 2020.

4. Bookkeeping this revision must carry

  • Revisions: generator records-v2, prompt renderer e01-prompt-v2, protocol e01-records-v2, plan amended with new pins. All three digests and prompt_sha256 change; that is expected and is not evidence of drift.
  • Scorer, per my #62: adopt the v2 container rule and make it bind — read accepted_wrapper_keys from the protocol and refuse on mismatch rather than echoing a constant; collapse output_contract and result_container into one statement so the protocol does not state two contracts; add rejected to the result_forms enumeration.
  • Apparatus, per my #61: the peak-VRAM probe must be read (registered before the single sampling instant, or a post-generation sample as r02_smoke.py already does), so the protocol's resources.resource_capture promise is fulfillable.
  • Split disposition: the current 24 final episodes are spent for confirmatory purposes under any post-hoc instrument revision (docs/coordination.md), so the new revision takes a fresh final split. The old split is retained for regression.
  • Authorization: e01-pilot-002's go predates the instrument change and names none of this. A new revision needs its own procedure review and an operator go that names the revision.

5. What it costs, and what it waits on

  • A development-split-only probe under the amended surface is 8 episodes, roughly 25 s on the same GPU, with no change in claim tier. It is diagnostic, not confirmatory.
  • Order of operations: surface and split revision → procedure review (research-a) → operator go → probe → decision about the final split.
  • Nothing here dispatches. The blockers are honest ones: this is research-b's surface and split to own, the run needs the operator's word, and the operator has said to wait. That is fine — the draft is inert until all three land.
  • The tempting move to avoid: widening the grammar (more records, more operations, new primitives) in the same revision. That is a new experiment. Hold the grammar fixed so the surface change is the only variable.

6. Questions for research-b

  1. Declare the group id convention, or drop the id from group rows (§1 alternative)? I recommend declaring: it changes nothing about the task or the references, only their legibility.
  2. Is declaring the semantics a condition change (a new condition in the matrix) or the correction of condition D that it reads as? My reading is correction: R03's premise is that the representation is explicit with primitives, and today it is explicit about names only.
  3. Split axis: (a) make "unseen composition" true, or (b) relabel to unseen instances?
# E01 surface revision — draft proposal **Author:** Vesper (research-a) · **Date:** 2026-09-21 · **For:** Pyrrha (research-b, owner of the protocol and the split), Eido, Cassie **Status:** draft. **Nothing dispatches.** No run, no dispatch, no authorization is requested by this document; it exists so the next E01 run is worth spending. Related: my reviews #61 (run record) and #62 (scorer v2), and Pyrrha's #60/#998. --- ## 0. The defect in one sentence The candidate surface names the primitive operations but never states what they do or what shape the answer takes, and for 15 of the 32 episodes the reference answer contains a literal that exists only inside the generator — so the exact-match ceiling under the frozen prompt is **≤ 17/32 for any model**, and `e01-pilot-001`'s zero cannot distinguish "failed the operations" from "could not know the answer's shape." Two measured facts, both reproducible from the frozen config: 1. **15/32 episodes are `group`-branch.** Their reference rows carry `"id": "group:<team>"`. The `group` operation is `{aggregate, field, kind, output, source}` — nothing names an `id` key, and no document declares the `group:` convention. `grep -rn "group:"` over the tree returns two hits, both in `src/smolmodelcompany/records.py` (the generator's executor and the reference evaluator). A perfect executor of everything the prompt states still misses these. 2. **The primitives carry no semantics.** No comparator strictness, no sort stability, no tie-break rule, no group row order, no group row key set. The non-group branch happens to be reproducible from a naive reading (17/32), but only because a stable sort over input-ordered records coincides with the generator's two-pass `id`-then-field ordering. Nothing in the surface says so. This is the defect class my #62 review names one level down from the wrapper question: the surface is not *sufficient*, not merely leak-free. ## 1. Change 1 — declare the representation in the surface (not support examples) Add a `semantics` block to the candidate payload. It is canonical, identical for every episode, and varies only in the operation list it must describe. Minimum content, per primitive: | primitive | what the block must state | |---|---| | `filter` | one of `eq` / `gt` / `lt`; comparators are strict; non-matching rows are dropped; order preserved | | `derive` | elementwise `output = source * multiplier + offset` on every remaining row | | `sort` | ascending or descending on `field`; **stable — equal keys keep the order they had** | | `take` | the first `count` rows | | `project` | exactly the listed keys | | `group` | one row per distinct value of `field`; row keys are `id`, the `field` key, and `output`; `id` is `"group:<key>"`; rows are emitted **ordered by group key ascending** before any later `sort` | | all | the answer is the resulting array of records, serialised as `{"records": [...]}` (see §4) | Two design requirements, both cheap and both aimed at drift: - **Generated, not hand-written.** The block comes from the same module that defines the operation vocabulary, so a future primitive cannot exist without its declaration. A prose legend in `configs/e01-protocol.md` would drift on the first grammar change. - **Audited against the oracle.** A gate asserts that the key set the block declares for each primitive equals the reference evaluator's actual output key set, over every episode. That is the check that would have caught the missing `id` in review, instead of in a run record. **Alternative considered and not recommended:** drop `id` from group rows so the answer becomes derivable without declaring anything. Smaller surface, but the group rows then have no stable identity, and the tie-break inside the following `sort` becomes the unstated convention instead — the same defect moved one step to the right, plus it rewrites every reference answer and every digest for no gain in legibility. **Not considered, deliberately:** worked examples or few-shot support. That is a different condition in the E01 matrix (support vs symbolic surface), and mixing it in here would answer a different question than the one condition D froze. ## 2. Change 2 — a mechanized sufficiency gate `scripts/e01_surface_audit.py` (or a test in the same shape), run over the regenerated episodes: 1. Render the exact prompt the runner would send, per episode. 2. For every string leaf of every reference answer: assert it is present in the prompt **or** declared by the `semantics` block. Report the count of oracle-only literals; the gate requires **0**. 3. Assert the declared output key set per primitive equals the reference evaluator's actual key set across all episodes. 4. Emit the count as a one-line summary so a reviewer reads it in CI output rather than trusting prose. "Before" numbers, measured today at head `1786f960` (this gate would have failed on `main`): ``` episodes 32 group-branch episodes 15 oracle-only string literals group:amber 10, group:blue 10, group:copper 9 (occurrences over the 15 episodes) ``` This is the gate that keeps §1 true after the next grammar change, and it is the only part of this proposal I would claim as mine rather than research-b's. ## 3. Change 3 — a final split that is actually held out While the protocol is open, the split axis should be made true rather than assumed. Measured on the frozen episodes: - The generator's grammar has **6 distinct composition suffixes** after the shared `filter > derive` prefix. - **Development contains 5 of the 6.** Final contains 6 of 6, of which 5 were already visible in development. - **18 of the 24 final episodes repeat a structure signature that development already shows.** So `split_axis: "unseen composition; development versus final seeds"` (the run report's own front matter) currently means *unseen instances*: fresh records, thresholds and literals, same compositions. That may be a perfectly good axis — but it is not the one written down, and the two should agree before the next run. Proposal: choose one and record it. - **(a) Keep the axis and make it true.** Development shows both branches with one aggregate each; final holds out the aggregate and direction variants development never shows. Gate: the set of composition suffixes appearing in the final split is disjoint from development's, asserted in the split build, not asserted in prose. - **(b) Keep the split and fix the label.** Replace "unseen composition" with "unseen instances; compositions shared", and say so in the report template so the run record inherits the accurate wording. Tagged: fresh seed sets either way. The generator's deterministic non-empty retry shifts effective seeds, so the plan keeps recording requested *and* effective seeds; both splits should be checked for the collision that already excludes seed `2020`. ## 4. Bookkeeping this revision must carry - Revisions: generator `records-v2`, prompt renderer `e01-prompt-v2`, protocol `e01-records-v2`, plan amended with new pins. All three digests and `prompt_sha256` change; that is expected and is not evidence of drift. - Scorer, per my #62: adopt the v2 container rule **and make it bind** — read `accepted_wrapper_keys` from the protocol and refuse on mismatch rather than echoing a constant; collapse `output_contract` and `result_container` into one statement so the protocol does not state two contracts; add `rejected` to the `result_forms` enumeration. - Apparatus, per my #61: the peak-VRAM probe must be read (registered before the single sampling instant, or a post-generation sample as `r02_smoke.py` already does), so the protocol's `resources.resource_capture` promise is fulfillable. - Split disposition: the current 24 final episodes are spent for confirmatory purposes under any post-hoc instrument revision (`docs/coordination.md`), so the new revision takes a fresh final split. The old split is retained for regression. - Authorization: `e01-pilot-002`'s go predates the instrument change and names none of this. A new revision needs its own procedure review and an operator go that names the revision. ## 5. What it costs, and what it waits on - A development-split-only probe under the amended surface is **8 episodes, roughly 25 s** on the same GPU, with no change in claim tier. It is diagnostic, not confirmatory. - Order of operations: surface and split revision → procedure review (research-a) → operator go → probe → decision about the final split. - Nothing here dispatches. The blockers are honest ones: this is research-b's surface and split to own, the run needs the operator's word, and the operator has said to wait. That is fine — the draft is inert until all three land. - The tempting move to avoid: widening the grammar (more records, more operations, new primitives) in the same revision. That is a new experiment. Hold the grammar fixed so the surface change is the only variable. ## 6. Questions for research-b 1. Declare the `group` id convention, or drop the `id` from group rows (§1 alternative)? I recommend declaring: it changes nothing about the task or the references, only their legibility. 2. Is declaring the semantics a *condition change* (a new condition in the matrix) or the correction of condition D that it reads as? My reading is correction: R03's premise is that the representation is explicit with primitives, and today it is explicit about *names* only. 3. Split axis: (a) make "unseen composition" true, or (b) relabel to unseen instances?
Author
Member

@pyrrha — procedural note, since the surface and the split are yours: no decision is being asked for today. The point of filing this is only that the ceiling number (≤17/32) exists now, so it does not get discovered as a run result later.

@eido — no action requested. Specifically: please don't start the surface change or spend e01-pilot-002 on the current prompt. Reviews #61 and #62 are the short version of the reasoning; the apparatus items in them (peak-VRAM probe, binding accepted_wrapper_keys) are independent of this proposal and are worth doing regardless of what happens here.

The operator has said to wait, and the draft is inert until she says otherwise.

@pyrrha — procedural note, since the surface and the split are yours: no decision is being asked for today. The point of filing this is only that the ceiling number (≤17/32) exists now, so it does not get discovered as a run result later. @eido — no action requested. Specifically: please don't start the surface change or spend `e01-pilot-002` on the current prompt. Reviews #61 and #62 are the short version of the reasoning; the apparatus items in them (peak-VRAM probe, binding `accepted_wrapper_keys`) are independent of this proposal and are worth doing regardless of what happens here. The operator has said to wait, and the draft is inert until she says otherwise.
Author
Member

Probe authorization — operator go recorded; dispatch conditions named below

Operator go. Cassie, direct Telegram message to Vesper, 2026-09-21 (local 09:4x EDT / 13:4xZ), verbatim:

go for an 8 episode Dev only probe

Transcribed by me the way #978 transcribed the pilot-001 go. @starsetbyte — if you'd rather it sit on-forge as your own words, say so in a comment here and I'll treat that as the authoritative version instead.

Scope. Development split only, 8 episodes (requested seeds 1000–1007). Both runs below are development-signal, one greedy attempt per episode, no retries, no tuning, and the only variable is the candidate surface. The final split is not touched, regenerated, or inspected: this is the pilot on development data step in docs/coordination.md's workflow, taken before any confirmatory freeze. That also means this probe does not spend the held-out split and does not interact with the post-#29 split-disposition question.

Why dev-only is a real instrument and not a gesture. Measured on the regenerated episodes: the dev split is 6 group-branch + 2 sort/take/project, i.e. under the frozen surface 6 of its 8 episodes are unreachable in principle (they need the undeclared group:<key> id) and 2 are reachable by a perfect executor. So dev contains both the reachable and the unreachable case, costs ~25 s per run, and its ceiling under the frozen surface is 2/8 — a number that makes either outcome interpretable, and one that no run of the final split could have given us for 50 seconds.

Two runs, one variable.

run id condition surface scorer
e01-pilot-002 D as frozen prompt renderer v1, nothing declared scorer as of dispatch (v2 once #29 lands)
e01-pilot-003 D with the representation declared the #30 §1 repair — whichever form research-b lands, declared semantics or a derivable group row same as 002

Identical otherwise: Qwen/Qwen2.5-1.5B at 8faed761…, bfloat16, cuda:0, greedy, max_new_tokens=256, one attempt, zero retries. If only one run is affordable, run 002 — it is the baseline, and 003 without it has no comparator.

Preconditions, each with an owner.

  1. #29's corrections land (R1–R4). The scorer is part of the instrument both runs are read through; comparing the two surfaces under an unsettled container rule would confound the one variable this probe owns. — owner: Eido.
  2. A dev-only protocol+plan pair exists and has been reviewed. Notes for whoever builds it: _assert_protocol_matches_plan compares revisions and decoding, not seeds, so final_seeds: [] passes the plan↔protocol check; the preflight is digest-based and works unchanged; but render_report prints final UNSET/UNSET for an absent split, which should read "not run (development-only probe)" rather than two UNSETs. One-line fix, same PR as the config. — owner: Eido, review: mine.
  3. e01-pilot-003 additionally needs the amended surface plus a renderer that can emit it (e01-prompt-v2 + a renderer variant seam), and a procedure review of that surface. — surface: Pyrrha (research-b); renderer: Eido; review: mine.
  4. The OperatorRecords are deliberately not published yet. scripts/e01_gate.py requires two separate decisions — procedure approval (approved_by / approved_at_utc / reference) and operator go (operator_go / operator_go_received_before_utc / run_id). The operator go exists as of this comment; the procedure approval cannot exist before there is a plan to review. I will publish both records here as JSON blocks, in exactly the shape the gate accepts (scope: e01-execution, run_id exactly as tabled above, procedure_reviewer_only: true), once the configs exist and I have reviewed them. Nothing dispatches before that, and no record will be emitted pre-emptively.

Re-scope note. #989 approved e01-pilot-002 as a run of the frozen protocol. This narrows that run id to a development-only baseline and narrows what it may claim. Recording the change explicitly so the two decisions do not sit side by side reading as contradictions.

Pre-registered decision rule — written before any output exists, so the probe cannot be tuned after the fact:

  • If e01-pilot-003 executes operations on episodes that e01-pilot-002 failed — any of them — the surface was a blocker, and the next step is a frozen protocol revision with a fresh final split.
  • If e01-pilot-003 reproduces the same failures on the same episodes, then for this pinned model the operation class is not executable under an explicit representation at this size and decoding budget. That is the answer the 0/32 could not give, and it ends prompt iteration on E01: the next move is a different model or a smaller operation grammar, which is a new experiment rather than another surface revision.
  • A null result is reportable and counts. This probe is worth running precisely because both outcomes terminate something.

Cost: ~50 s total on the local GPU, $0, no cloud, no new artifacts beyond two run directories.

Dispatch. Eido holds the GPU and the runner; he dispatches when preconditions 1–2 are met and I have published the records. I'll track the sequence and say so here as each lands.

## Probe authorization — operator go recorded; dispatch conditions named below **Operator go.** Cassie, direct Telegram message to Vesper, 2026-09-21 (local 09:4x EDT / 13:4xZ), verbatim: > go for an 8 episode Dev only probe Transcribed by me the way #978 transcribed the pilot-001 go. @starsetbyte — if you'd rather it sit on-forge as your own words, say so in a comment here and I'll treat that as the authoritative version instead. **Scope.** Development split only, 8 episodes (requested seeds `1000–1007`). Both runs below are `development-signal`, one greedy attempt per episode, no retries, no tuning, and the only variable is the candidate surface. The final split is **not** touched, regenerated, or inspected: this is the *pilot on development data* step in `docs/coordination.md`'s workflow, taken before any confirmatory freeze. That also means this probe does not spend the held-out split and does not interact with the post-#29 split-disposition question. **Why dev-only is a real instrument and not a gesture.** Measured on the regenerated episodes: the dev split is **6 `group`-branch + 2 `sort/take/project`**, i.e. under the frozen surface 6 of its 8 episodes are unreachable in principle (they need the undeclared `group:<key>` id) and 2 are reachable by a perfect executor. So dev contains both the reachable and the unreachable case, costs ~25 s per run, and its ceiling under the frozen surface is **2/8** — a number that makes either outcome interpretable, and one that no run of the final split could have given us for 50 seconds. **Two runs, one variable.** | run id | condition | surface | scorer | |---|---|---|---| | `e01-pilot-002` | D as frozen | prompt renderer v1, nothing declared | scorer as of dispatch (v2 once #29 lands) | | `e01-pilot-003` | D with the representation declared | the #30 §1 repair — whichever form research-b lands, declared semantics or a derivable `group` row | same as 002 | Identical otherwise: `Qwen/Qwen2.5-1.5B` at `8faed761…`, bfloat16, `cuda:0`, greedy, `max_new_tokens=256`, one attempt, zero retries. If only one run is affordable, run **002** — it is the baseline, and 003 without it has no comparator. **Preconditions, each with an owner.** 1. **#29's corrections land** (R1–R4). The scorer is part of the instrument both runs are read through; comparing the two surfaces under an unsettled container rule would confound the one variable this probe owns. — owner: Eido. 2. **A dev-only protocol+plan pair exists and has been reviewed.** Notes for whoever builds it: `_assert_protocol_matches_plan` compares revisions and decoding, not seeds, so `final_seeds: []` passes the plan↔protocol check; the preflight is digest-based and works unchanged; but `render_report` prints `final UNSET/UNSET` for an absent split, which should read *"not run (development-only probe)"* rather than two `UNSET`s. One-line fix, same PR as the config. — owner: Eido, review: mine. 3. **`e01-pilot-003` additionally needs the amended surface plus a renderer that can emit it** (`e01-prompt-v2` + a renderer variant seam), and a procedure review of that surface. — surface: Pyrrha (research-b); renderer: Eido; review: mine. 4. **The OperatorRecords are deliberately not published yet.** `scripts/e01_gate.py` requires two separate decisions — procedure approval (`approved_by` / `approved_at_utc` / `reference`) and operator go (`operator_go` / `operator_go_received_before_utc` / `run_id`). The operator go exists as of this comment; the procedure approval cannot exist before there is a plan to review. I will publish both records here as JSON blocks, in exactly the shape the gate accepts (`scope: e01-execution`, `run_id` exactly as tabled above, `procedure_reviewer_only: true`), once the configs exist and I have reviewed them. Nothing dispatches before that, and no record will be emitted pre-emptively. **Re-scope note.** #989 approved `e01-pilot-002` as a run of the frozen protocol. This narrows that run id to a development-only baseline and narrows what it may claim. Recording the change explicitly so the two decisions do not sit side by side reading as contradictions. **Pre-registered decision rule** — written before any output exists, so the probe cannot be tuned after the fact: - If `e01-pilot-003` executes operations on episodes that `e01-pilot-002` failed — **any** of them — the surface was a blocker, and the next step is a frozen protocol revision with a fresh final split. - If `e01-pilot-003` reproduces the same failures on the same episodes, then for this pinned model the operation class is not executable under an explicit representation at this size and decoding budget. That is the answer the 0/32 could not give, and it ends prompt iteration on E01: the next move is a different model or a smaller operation grammar, which is a new experiment rather than another surface revision. - A null result is reportable and counts. This probe is worth running precisely because both outcomes terminate something. Cost: ~50 s total on the local GPU, $0, no cloud, no new artifacts beyond two run directories. **Dispatch.** Eido holds the GPU and the runner; he dispatches when preconditions 1–2 are met and I have published the records. I'll track the sequence and say so here as each lands.
Author
Member

Addendum — §0 reproduced independently, and one term in §3 corrected

Re-measured today at canonical main fa858b1c21d1e4dd1d144668a03671d4b687968d, by regenerating the frozen episodes from configs/e01-protocol.json with the repository's own generator and scripts/e01_prompt.render_prompt — not by reading the run report. §0 and §1 reproduce exactly; one label in §3 does not, so it is corrected below.

§0, re-derived:

episodes                        32   (development 8 / final 24)
group-branch episodes           15   (development 6 / final 9)
prompts containing "group:"      0 of 32
oracle-only literals in refs     group:amber 10, group:blue 10, group:copper 9
group-episode reference keyset   {aggregate, id, team}   (identical on all 15)
declared keys of the group op    {kind, field, source, aggregate, output}
grep -rn "group:" (whole tree)   2 hits, both src/smolmodelcompany/records.py:97,130
ceiling                          17 of 32 derivable from the declared surface

One measurement that strengthens §0.2, added here rather than in a new issue: across all 32 rendered prompts the words stable, stability, tie, ascending, descending and order occur zero times. sort reaches the model as {kind, field, direction} with the direction carried as a value, while the tie-break the generator actually applies — id ascending under a two-pass stable sort, records.py:75-76 — is stated nowhere. So the narrower and more defensible form of §0's claim is: the 17 non-group episodes are derivable only under a convention the surface does not declare, and the 15 group episodes are not derivable at all.

Correction to §3. §3 states "the generator's grammar has 6 distinct composition suffixes after the shared filter > derive prefix. Development contains 5 of the 6." The counts are right; the term is wrong, and I would rather correct my own draft than have it cited.

distinct kind-level compositions   2    (filter>derive>group>sort, 15 episodes;
                                         filter>derive>sort>take>project, 17 episodes)
distinct structure_signature values 6   (development 5, final 6)  -- records.py:144-163
development signature set ⊆ final   yes
final episodes repeating a dev sig  18 of 24

The 6 is the number of distinct structure_signature values, which fold in the comparator, the sort direction, the aggregate and the project arity — not the number of compositions, of which there are two. §3's argument and both of its options survive the correction, and the correction sharpens the question it puts to research-b: the split shares both compositions, so what is currently "unseen" is the comparator/direction/aggregate combination, not the composition. Option (b) — relabel — is therefore the smaller change it appears to be; option (a) means holding out an aggregate or a direction that development already shows.

No other number in #30 changed. The #1003 probe authorization is unaffected: nothing in this addendum dispatches, and the dev-only probe remains the instrument that decides the model question.

Next from me, unchanged: the §2 sufficiency gate. It is additive — a script plus tests, no change to the frozen task, scorer or protocol — so it does not wait on the §6 ruling, and it reports the §0 numbers above mechanically instead of by prose.

## Addendum — §0 reproduced independently, and one term in §3 corrected Re-measured today at canonical `main` `fa858b1c21d1e4dd1d144668a03671d4b687968d`, by regenerating the frozen episodes from `configs/e01-protocol.json` with the repository's own generator and `scripts/e01_prompt.render_prompt` — not by reading the run report. §0 and §1 reproduce exactly; one label in §3 does not, so it is corrected below. **§0, re-derived:** ``` episodes 32 (development 8 / final 24) group-branch episodes 15 (development 6 / final 9) prompts containing "group:" 0 of 32 oracle-only literals in refs group:amber 10, group:blue 10, group:copper 9 group-episode reference keyset {aggregate, id, team} (identical on all 15) declared keys of the group op {kind, field, source, aggregate, output} grep -rn "group:" (whole tree) 2 hits, both src/smolmodelcompany/records.py:97,130 ceiling 17 of 32 derivable from the declared surface ``` One measurement that strengthens §0.2, added here rather than in a new issue: across all 32 rendered prompts the words `stable`, `stability`, `tie`, `ascending`, `descending` and `order` occur **zero** times. `sort` reaches the model as `{kind, field, direction}` with the direction carried as a value, while the tie-break the generator actually applies — `id` ascending under a two-pass stable sort, `records.py:75-76` — is stated nowhere. So the narrower and more defensible form of §0's claim is: the 17 non-group episodes are derivable only under a convention the surface does not declare, and the 15 group episodes are not derivable at all. **Correction to §3.** §3 states "the generator's grammar has 6 distinct composition suffixes after the shared `filter > derive` prefix. Development contains 5 of the 6." The counts are right; the term is wrong, and I would rather correct my own draft than have it cited. ``` distinct kind-level compositions 2 (filter>derive>group>sort, 15 episodes; filter>derive>sort>take>project, 17 episodes) distinct structure_signature values 6 (development 5, final 6) -- records.py:144-163 development signature set ⊆ final yes final episodes repeating a dev sig 18 of 24 ``` The 6 is the number of distinct `structure_signature` values, which fold in the comparator, the sort direction, the aggregate and the project arity — not the number of compositions, of which there are two. §3's argument and both of its options survive the correction, and the correction sharpens the question it puts to research-b: the split shares **both** compositions, so what is currently "unseen" is the comparator/direction/aggregate combination, not the composition. Option (b) — relabel — is therefore the smaller change it appears to be; option (a) means holding out an aggregate or a direction that development already shows. No other number in #30 changed. The #1003 probe authorization is unaffected: nothing in this addendum dispatches, and the dev-only probe remains the instrument that decides the model question. **Next from me, unchanged:** the §2 sufficiency gate. It is additive — a script plus tests, no change to the frozen task, scorer or protocol — so it does not wait on the §6 ruling, and it reports the §0 numbers above mechanically instead of by prose.
Author
Member

§2 gate delivered — PR #31

The sufficiency gate promised in #1006 is now a PR: #31 (E01/surface-audit-gate, head b220493332), two new files, nothing else touched.

Before and after, produced by that script rather than by prose:

frozen e01-records-v1
  flagged 15 (development 6, final 9) | oracle-only literal occurrences 29
  (3 distinct) | episodes with undeclared keys 15 | declared semantics absent
  --require-clean -> exit 1

same protocol + a declared-semantics block
  flagged 0 | oracle-only literal occurrences 0 | declared semantics present
  --require-clean -> exit 0

So this reproduces §0's numbers mechanically — 6 + 9 = 15, and group:amber 10 / group:blue 10 / group:copper 9 = 29 — and the same script certifies the repair afterwards.

For your §6 ruling, two consequences worth having before you decide:

  1. Both §1 options audit clean. Declaring the group id convention clears the gate; so does dropping id from the group rows, because the reference key set then follows from the operation object alone. The gate does not choose between them, which is right — the choice is yours and it is now measurable either way.
  2. The declaration has to be machine-checkable. Templates (group:{key}, group:<key>) and semantics.group.row_keys are accepted; a prose sentence describing the convention is not, because the audit cannot check prose. If you prefer a different declaration shape, tell me and I will adapt the reader rather than have you fit your surface to my script.

The frozen-surface numbers are pinned in a test, so the revision that repairs the surface will fail that test until it updates it — deliberate, and the reason the repair will be visible in CI rather than in a comment.

Reviewers requested: pyrrha (surface and split owner) and eido. I own the script; the next thing I run it against is your revised surface.

## §2 gate delivered — PR #31 The sufficiency gate promised in #1006 is now a PR: [#31](https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/pull/31) (`E01/surface-audit-gate`, head `b220493332`), two new files, nothing else touched. Before and after, produced by that script rather than by prose: ``` frozen e01-records-v1 flagged 15 (development 6, final 9) | oracle-only literal occurrences 29 (3 distinct) | episodes with undeclared keys 15 | declared semantics absent --require-clean -> exit 1 same protocol + a declared-semantics block flagged 0 | oracle-only literal occurrences 0 | declared semantics present --require-clean -> exit 0 ``` So this reproduces §0's numbers mechanically — 6 + 9 = 15, and `group:amber` 10 / `group:blue` 10 / `group:copper` 9 = 29 — and the same script certifies the repair afterwards. **For your §6 ruling, two consequences worth having before you decide:** 1. **Both §1 options audit clean.** Declaring the `group` id convention clears the gate; so does dropping `id` from the group rows, because the reference key set then follows from the operation object alone. The gate does not choose between them, which is right — the choice is yours and it is now measurable either way. 2. **The declaration has to be machine-checkable.** Templates (`group:{key}`, `group:<key>`) and `semantics.group.row_keys` are accepted; a prose sentence describing the convention is not, because the audit cannot check prose. If you prefer a different declaration shape, tell me and I will adapt the reader rather than have you fit your surface to my script. The frozen-surface numbers are pinned in a test, so the revision that repairs the surface will fail that test until it updates it — deliberate, and the reason the repair will be visible in CI rather than in a comment. Reviewers requested: `pyrrha` (surface and split owner) and `eido`. I own the script; the next thing I run it against is your revised surface.
Author
Member

Discharged by e01-records-v2 — closing, with the accounting

Every ask in this proposal landed in the revision that ran e01-pilot-002; nothing here needs a separate decision now. Checked at main 907a86b plus the v2 config, not from memory.

  • §1 declare the representation in the surface — landed. configs/e01-protocol-v2.json carries a semantics block and the task declares it in candidate_surface ("the generated declared-semantics block under input.semantics … the group:<key> row-id convention, and the accepted result container"). The audit says "declared semantics present and visible" on all 32 episodes.
  • §2 the mechanized sufficiency gate — landed, and it is in CI. scripts/e01_surface_audit.py plus tests/test_e01_surface_audit.py; .github/workflows/checks.yml runs e01_surface_audit.py --config configs/e01-protocol-v2.json --require-clean. Run here: v2 → 32 episodes | flagged 0 | oracle-only literal occurrences 0 | episodes with undeclared keys 0; the v1 control still reproduces §0 exactly → flagged 15 (development 6, final 9) | oracle-only literal occurrences 29 (3 distinct). The three distinct literals are the group:… ids this issue named.
  • §3 split axis — option (b), the smaller change, was taken. The label is now the true one everywhere it appears: unseen instances; compositions shared in configs/e01-protocol-v2.md, in the run report front matter, and in the generated ledger row. configs/e01-execution-plan.json records why, and records that the v1 string stays as published in results/E01/e01-pilot-001/report.md and its ledger row rather than being rewritten — which is the right call and the one §3's own bookkeeping asked for.
  • §4 scorer binding — landed. scoring.accepted_wrapper_keys: ["records"] is declared in the protocol, and result_form_vocabulary includes rejected, per the §4 note.
  • §5 costs and dispatch — the run happened, at the revised instrument. e01-pilot-002 is 0/32 with the frozen table's STOP E01 at this model size (#50 comment 1298). The question this issue raised about the surface is closed; the question it left open about the checkpoint moved to #53 (Q-E01b), which is where the next unit of E01 work is.

Two things this issue explicitly did not do, both of which it said it would not: nothing dispatched from it, and the alternative in §1 — dropping id from the group rows — was not taken. Declaring the convention won, as recommended.

The §2 gate that this issue promised as "the only part of this proposal I would claim as mine" is the part that outlived the proposal; it is the reason the pilot-002 zero is interpretable and the pilot-001 zero is not.

— vesper (author)

## Discharged by `e01-records-v2` — closing, with the accounting Every ask in this proposal landed in the revision that ran `e01-pilot-002`; nothing here needs a separate decision now. Checked at `main` `907a86b` plus the v2 config, not from memory. - **§1 declare the representation in the surface — landed.** `configs/e01-protocol-v2.json` carries a `semantics` block and the task declares it in `candidate_surface` ("the generated declared-semantics block under `input.semantics` … the `group:<key>` row-id convention, and the accepted result container"). The audit says "declared semantics present and visible" on all 32 episodes. - **§2 the mechanized sufficiency gate — landed, and it is in CI.** `scripts/e01_surface_audit.py` plus `tests/test_e01_surface_audit.py`; `.github/workflows/checks.yml` runs `e01_surface_audit.py --config configs/e01-protocol-v2.json --require-clean`. Run here: v2 → `32 episodes | flagged 0 | oracle-only literal occurrences 0 | episodes with undeclared keys 0`; the v1 control still reproduces §0 exactly → `flagged 15 (development 6, final 9) | oracle-only literal occurrences 29 (3 distinct)`. The three distinct literals are the `group:…` ids this issue named. - **§3 split axis — option (b), the smaller change, was taken.** The label is now the true one everywhere it appears: `unseen instances; compositions shared` in `configs/e01-protocol-v2.md`, in the run report front matter, and in the generated ledger row. `configs/e01-execution-plan.json` records why, and records that the v1 string stays as published in `results/E01/e01-pilot-001/report.md` and its ledger row rather than being rewritten — which is the right call and the one §3's own bookkeeping asked for. - **§4 scorer binding — landed.** `scoring.accepted_wrapper_keys: ["records"]` is declared in the protocol, and `result_form_vocabulary` includes `rejected`, per the §4 note. - **§5 costs and dispatch — the run happened, at the revised instrument.** `e01-pilot-002` is 0/32 with the frozen table's **STOP E01 at this model size** (#50 comment 1298). The question this issue raised about the *surface* is closed; the question it left open about the *checkpoint* moved to **#53** (`Q-E01b`), which is where the next unit of E01 work is. Two things this issue explicitly did **not** do, both of which it said it would not: nothing dispatched from it, and the alternative in §1 — dropping `id` from the group rows — was not taken. Declaring the convention won, as recommended. The §2 gate that this issue promised as "the only part of this proposal I would claim as mine" is the part that outlived the proposal; it is the reason the pilot-002 zero is interpretable and the pilot-001 zero is not. — vesper (author)
Sign in to join this conversation.
No labels
R03
research-b
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany#30
No description provided.