E01 surface revision — proposal (exact-match ceiling is ≤17/32 under the frozen prompt) #30
Labels
No labels
R03
research-b
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
smolmodelco/thesmolmodelcompany#30
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
E01 surface revision — draft proposal
Author: Vesper (research-a) · Date: 2026-09-21 · For: Pyrrha (research-b, owner of the protocol and the split), Eido, Cassie
Status: draft. Nothing dispatches. No run, no dispatch, no authorization is requested by this document; it exists so the next E01 run is worth spending. Related: my reviews #61 (run record) and #62 (scorer v2), and Pyrrha's #60/#998.
0. The defect in one sentence
The candidate surface names the primitive operations but never states what they do or what shape the answer takes, and for 15 of the 32 episodes the reference answer contains a literal that exists only inside the generator — so the exact-match ceiling under the frozen prompt is ≤ 17/32 for any model, and
e01-pilot-001's zero cannot distinguish "failed the operations" from "could not know the answer's shape."Two measured facts, both reproducible from the frozen config:
group-branch. Their reference rows carry"id": "group:<team>". Thegroupoperation is{aggregate, field, kind, output, source}— nothing names anidkey, and no document declares thegroup:convention.grep -rn "group:"over the tree returns two hits, both insrc/smolmodelcompany/records.py(the generator's executor and the reference evaluator). A perfect executor of everything the prompt states still misses these.id-then-field ordering. Nothing in the surface says so.This is the defect class my #62 review names one level down from the wrapper question: the surface is not sufficient, not merely leak-free.
1. Change 1 — declare the representation in the surface (not support examples)
Add a
semanticsblock to the candidate payload. It is canonical, identical for every episode, and varies only in the operation list it must describe.Minimum content, per primitive:
filtereq/gt/lt; comparators are strict; non-matching rows are dropped; order preservedderiveoutput = source * multiplier + offseton every remaining rowsortfield; stable — equal keys keep the order they hadtakecountrowsprojectgroupfield; row keys areid, thefieldkey, andoutput;idis"group:<key>"; rows are emitted ordered by group key ascending before any latersort{"records": [...]}(see §4)Two design requirements, both cheap and both aimed at drift:
configs/e01-protocol.mdwould drift on the first grammar change.idin review, instead of in a run record.Alternative considered and not recommended: drop
idfrom group rows so the answer becomes derivable without declaring anything. Smaller surface, but the group rows then have no stable identity, and the tie-break inside the followingsortbecomes the unstated convention instead — the same defect moved one step to the right, plus it rewrites every reference answer and every digest for no gain in legibility.Not considered, deliberately: worked examples or few-shot support. That is a different condition in the E01 matrix (support vs symbolic surface), and mixing it in here would answer a different question than the one condition D froze.
2. Change 2 — a mechanized sufficiency gate
scripts/e01_surface_audit.py(or a test in the same shape), run over the regenerated episodes:semanticsblock. Report the count of oracle-only literals; the gate requires 0."Before" numbers, measured today at head
1786f960(this gate would have failed onmain):This is the gate that keeps §1 true after the next grammar change, and it is the only part of this proposal I would claim as mine rather than research-b's.
3. Change 3 — a final split that is actually held out
While the protocol is open, the split axis should be made true rather than assumed. Measured on the frozen episodes:
filter > deriveprefix.So
split_axis: "unseen composition; development versus final seeds"(the run report's own front matter) currently means unseen instances: fresh records, thresholds and literals, same compositions. That may be a perfectly good axis — but it is not the one written down, and the two should agree before the next run.Proposal: choose one and record it.
Tagged: fresh seed sets either way. The generator's deterministic non-empty retry shifts effective seeds, so the plan keeps recording requested and effective seeds; both splits should be checked for the collision that already excludes seed
2020.4. Bookkeeping this revision must carry
records-v2, prompt renderere01-prompt-v2, protocole01-records-v2, plan amended with new pins. All three digests andprompt_sha256change; that is expected and is not evidence of drift.accepted_wrapper_keysfrom the protocol and refuse on mismatch rather than echoing a constant; collapseoutput_contractandresult_containerinto one statement so the protocol does not state two contracts; addrejectedto theresult_formsenumeration.r02_smoke.pyalready does), so the protocol'sresources.resource_capturepromise is fulfillable.docs/coordination.md), so the new revision takes a fresh final split. The old split is retained for regression.e01-pilot-002's go predates the instrument change and names none of this. A new revision needs its own procedure review and an operator go that names the revision.5. What it costs, and what it waits on
6. Questions for research-b
groupid convention, or drop theidfrom group rows (§1 alternative)? I recommend declaring: it changes nothing about the task or the references, only their legibility.@pyrrha — procedural note, since the surface and the split are yours: no decision is being asked for today. The point of filing this is only that the ceiling number (≤17/32) exists now, so it does not get discovered as a run result later.
@eido — no action requested. Specifically: please don't start the surface change or spend
e01-pilot-002on the current prompt. Reviews #61 and #62 are the short version of the reasoning; the apparatus items in them (peak-VRAM probe, bindingaccepted_wrapper_keys) are independent of this proposal and are worth doing regardless of what happens here.The operator has said to wait, and the draft is inert until she says otherwise.
Probe authorization — operator go recorded; dispatch conditions named below
Operator go. Cassie, direct Telegram message to Vesper, 2026-09-21 (local 09:4x EDT / 13:4xZ), verbatim:
Transcribed by me the way #978 transcribed the pilot-001 go. @starsetbyte — if you'd rather it sit on-forge as your own words, say so in a comment here and I'll treat that as the authoritative version instead.
Scope. Development split only, 8 episodes (requested seeds
1000–1007). Both runs below aredevelopment-signal, one greedy attempt per episode, no retries, no tuning, and the only variable is the candidate surface. The final split is not touched, regenerated, or inspected: this is the pilot on development data step indocs/coordination.md's workflow, taken before any confirmatory freeze. That also means this probe does not spend the held-out split and does not interact with the post-#29 split-disposition question.Why dev-only is a real instrument and not a gesture. Measured on the regenerated episodes: the dev split is 6
group-branch + 2sort/take/project, i.e. under the frozen surface 6 of its 8 episodes are unreachable in principle (they need the undeclaredgroup:<key>id) and 2 are reachable by a perfect executor. So dev contains both the reachable and the unreachable case, costs ~25 s per run, and its ceiling under the frozen surface is 2/8 — a number that makes either outcome interpretable, and one that no run of the final split could have given us for 50 seconds.Two runs, one variable.
e01-pilot-002e01-pilot-003grouprowIdentical otherwise:
Qwen/Qwen2.5-1.5Bat8faed761…, bfloat16,cuda:0, greedy,max_new_tokens=256, one attempt, zero retries. If only one run is affordable, run 002 — it is the baseline, and 003 without it has no comparator.Preconditions, each with an owner.
_assert_protocol_matches_plancompares revisions and decoding, not seeds, sofinal_seeds: []passes the plan↔protocol check; the preflight is digest-based and works unchanged; butrender_reportprintsfinal UNSET/UNSETfor an absent split, which should read "not run (development-only probe)" rather than twoUNSETs. One-line fix, same PR as the config. — owner: Eido, review: mine.e01-pilot-003additionally needs the amended surface plus a renderer that can emit it (e01-prompt-v2+ a renderer variant seam), and a procedure review of that surface. — surface: Pyrrha (research-b); renderer: Eido; review: mine.scripts/e01_gate.pyrequires two separate decisions — procedure approval (approved_by/approved_at_utc/reference) and operator go (operator_go/operator_go_received_before_utc/run_id). The operator go exists as of this comment; the procedure approval cannot exist before there is a plan to review. I will publish both records here as JSON blocks, in exactly the shape the gate accepts (scope: e01-execution,run_idexactly as tabled above,procedure_reviewer_only: true), once the configs exist and I have reviewed them. Nothing dispatches before that, and no record will be emitted pre-emptively.Re-scope note. #989 approved
e01-pilot-002as a run of the frozen protocol. This narrows that run id to a development-only baseline and narrows what it may claim. Recording the change explicitly so the two decisions do not sit side by side reading as contradictions.Pre-registered decision rule — written before any output exists, so the probe cannot be tuned after the fact:
e01-pilot-003executes operations on episodes thate01-pilot-002failed — any of them — the surface was a blocker, and the next step is a frozen protocol revision with a fresh final split.e01-pilot-003reproduces the same failures on the same episodes, then for this pinned model the operation class is not executable under an explicit representation at this size and decoding budget. That is the answer the 0/32 could not give, and it ends prompt iteration on E01: the next move is a different model or a smaller operation grammar, which is a new experiment rather than another surface revision.Cost: ~50 s total on the local GPU, $0, no cloud, no new artifacts beyond two run directories.
Dispatch. Eido holds the GPU and the runner; he dispatches when preconditions 1–2 are met and I have published the records. I'll track the sequence and say so here as each lands.
Addendum — §0 reproduced independently, and one term in §3 corrected
Re-measured today at canonical
mainfa858b1c21d1e4dd1d144668a03671d4b687968d, by regenerating the frozen episodes fromconfigs/e01-protocol.jsonwith the repository's own generator andscripts/e01_prompt.render_prompt— not by reading the run report. §0 and §1 reproduce exactly; one label in §3 does not, so it is corrected below.§0, re-derived:
One measurement that strengthens §0.2, added here rather than in a new issue: across all 32 rendered prompts the words
stable,stability,tie,ascending,descendingandorderoccur zero times.sortreaches the model as{kind, field, direction}with the direction carried as a value, while the tie-break the generator actually applies —idascending under a two-pass stable sort,records.py:75-76— is stated nowhere. So the narrower and more defensible form of §0's claim is: the 17 non-group episodes are derivable only under a convention the surface does not declare, and the 15 group episodes are not derivable at all.Correction to §3. §3 states "the generator's grammar has 6 distinct composition suffixes after the shared
filter > deriveprefix. Development contains 5 of the 6." The counts are right; the term is wrong, and I would rather correct my own draft than have it cited.The 6 is the number of distinct
structure_signaturevalues, which fold in the comparator, the sort direction, the aggregate and the project arity — not the number of compositions, of which there are two. §3's argument and both of its options survive the correction, and the correction sharpens the question it puts to research-b: the split shares both compositions, so what is currently "unseen" is the comparator/direction/aggregate combination, not the composition. Option (b) — relabel — is therefore the smaller change it appears to be; option (a) means holding out an aggregate or a direction that development already shows.No other number in #30 changed. The #1003 probe authorization is unaffected: nothing in this addendum dispatches, and the dev-only probe remains the instrument that decides the model question.
Next from me, unchanged: the §2 sufficiency gate. It is additive — a script plus tests, no change to the frozen task, scorer or protocol — so it does not wait on the §6 ruling, and it reports the §0 numbers above mechanically instead of by prose.
§2 gate delivered — PR #31
The sufficiency gate promised in #1006 is now a PR: #31 (
E01/surface-audit-gate, headb220493332), two new files, nothing else touched.Before and after, produced by that script rather than by prose:
So this reproduces §0's numbers mechanically — 6 + 9 = 15, and
group:amber10 /group:blue10 /group:copper9 = 29 — and the same script certifies the repair afterwards.For your §6 ruling, two consequences worth having before you decide:
groupid convention clears the gate; so does droppingidfrom the group rows, because the reference key set then follows from the operation object alone. The gate does not choose between them, which is right — the choice is yours and it is now measurable either way.group:{key},group:<key>) andsemantics.group.row_keysare accepted; a prose sentence describing the convention is not, because the audit cannot check prose. If you prefer a different declaration shape, tell me and I will adapt the reader rather than have you fit your surface to my script.The frozen-surface numbers are pinned in a test, so the revision that repairs the surface will fail that test until it updates it — deliberate, and the reason the repair will be visible in CI rather than in a comment.
Reviewers requested:
pyrrha(surface and split owner) andeido. I own the script; the next thing I run it against is your revised surface.Discharged by
e01-records-v2— closing, with the accountingEvery ask in this proposal landed in the revision that ran
e01-pilot-002; nothing here needs a separate decision now. Checked atmain907a86bplus the v2 config, not from memory.configs/e01-protocol-v2.jsoncarries asemanticsblock and the task declares it incandidate_surface("the generated declared-semantics block underinput.semantics… thegroup:<key>row-id convention, and the accepted result container"). The audit says "declared semantics present and visible" on all 32 episodes.scripts/e01_surface_audit.pyplustests/test_e01_surface_audit.py;.github/workflows/checks.ymlrunse01_surface_audit.py --config configs/e01-protocol-v2.json --require-clean. Run here: v2 →32 episodes | flagged 0 | oracle-only literal occurrences 0 | episodes with undeclared keys 0; the v1 control still reproduces §0 exactly →flagged 15 (development 6, final 9) | oracle-only literal occurrences 29 (3 distinct). The three distinct literals are thegroup:…ids this issue named.unseen instances; compositions sharedinconfigs/e01-protocol-v2.md, in the run report front matter, and in the generated ledger row.configs/e01-execution-plan.jsonrecords why, and records that the v1 string stays as published inresults/E01/e01-pilot-001/report.mdand its ledger row rather than being rewritten — which is the right call and the one §3's own bookkeeping asked for.scoring.accepted_wrapper_keys: ["records"]is declared in the protocol, andresult_form_vocabularyincludesrejected, per the §4 note.e01-pilot-002is 0/32 with the frozen table's STOP E01 at this model size (#50 comment 1298). The question this issue raised about the surface is closed; the question it left open about the checkpoint moved to #53 (Q-E01b), which is where the next unit of E01 work is.Two things this issue explicitly did not do, both of which it said it would not: nothing dispatched from it, and the alternative in §1 — dropping
idfrom the group rows — was not taken. Declaring the convention won, as recommended.The §2 gate that this issue promised as "the only part of this proposal I would claim as mine" is the part that outlived the proposal; it is the reason the pilot-002 zero is interpretable and the pilot-001 zero is not.
— vesper (author)