feat(E01): derive the declared-semantics block from the executor (stacks on #31) #32

Closed
vesper wants to merge 1 commit from E01/surface-semantics into main
Member

Task / experiment ID

Issue #30 §1 — E01 surface revision. Additive tooling only: no protocol, scorer, task, prompt or split change, and nothing here dispatches anything.

Stacked on #31 (E01/surface-audit-gate, 6321e57). Three commits, four files: the first two files are #31's audit, still under review; this PR adds two. The diff collapses to scripts/e01_semantics.py + tests/test_e01_semantics.py once #31 lands, and this one rebases onto main.

The gap this closes

§1 asks for a declaration that is generated, not hand-written, so a future primitive cannot exist without its declaration, and it asks the gate to audit the declaration against the oracle. #31 delivered the reader — it accepts a declaration, checks the declared key set against the reference's actual key set, and excuses a literal only through an anchored, instantiable template. It did not deliver the generator, and the declaration in #31's own body was mine, typed by hand.

scripts/e01_semantics.py derives it instead. For every operation the frozen grammar uses, it executes the episode's own program prefix up to and including that operation, reads the rows the reference evaluator produces, and declares only what the operation object alone does not account for.

Derived, for the frozen protocol

$ python3 scripts/e01_semantics.py --config configs/e01-protocol.json
E01 semantics: 32 episodes | kinds needing a declaration ['group'] |
declared 3 row keys | with the block applied: flagged 0, oracle-only 0, undeclared keys 0

{
  "group": {
    "row_keys": ["id", "<field>", "<output>"],
    "templates": {"id": "group:{key}"}
  }
}

That is the same block §1 proposes, except that nothing in it is a claim I made:

  • row_keys are the oracle's own row keys at the group step. <field> and <output> resolve per episode, which is what keeps the block identical for every episode while the field and output names are chosen by the generator.
  • templates.id is read off the rows: every group row's id is group: + that row's own field value. The generator emits a template only when that holds for every observed row and carries fixed text of its own — a key that merely copies its sibling declares nothing and is refused.
  • group is the only kind that needs declaring. For filter, derive, sort, take and project the operation object already accounts for every key the oracle produces, so the block states nothing about them.

What it refuses to do

Each of these raises instead of shipping a surface, and each has a test:

  • a primitive with no declaration reader — a kind that produces an undeclared key and has no reader in e01_surface_audit.py fails the build. §1's "a future primitive cannot exist without its declaration", enforced rather than promised;
  • a key set that varies by episode — a block identical for every episode cannot describe it;
  • a value convention that does not hold, or that holds ambiguously against two siblings;
  • a convention with no fixed text — a bare placeholder.

Verified

  • Applied to the frozen protocol, the audit reports 0 flagged, 0 oracle-only literal occurrences, 0 undeclared keys — through the anchored reader, so not by a catch-all, and the same script still flags a decoy id (group:zanzibar) the model could not instantiate.
  • With the block removed the frozen surface is back to 15 flagged, 29 occurrences, 3 distinct; the block earns the clean audit rather than hiding the defect.
  • python3 -m unittest discover -s tests -t . → Ran 333 tests, OK (310 at the base, +23 new). CI runs only the documentation integrity job, so this is my reproduction and a reviewer should re-run it.
  • check_docs.py 33 files OK; build_results_ledger.py --check ledger current; git diff --check clean.

Read-only by construction: no model library, no socket, no file written unless --out is given. The frozen protocol is not touched by this PR or by running the script — the block is data for the revision research-b owns.

Not in scope, deliberately

No surface change, no configs/e01-protocol-v2.json, no re-freeze, no split work, no run record, no dispatch, no compute. e01-pilot-002/003 remain unauthorized pending the preconditions in #1003 and the operator gate.

Next owner

Research-b: the emitted block is the §1 declaration whenever you want it — paste it into whatever shape you prefer; if you prefer a different declaration shape, this is one emitter to adapt, and the check stays mechanical either way. The §6 rulings (declare the group id, or drop it; split option (a) or (b)) still gate the revision itself.

Me: once the revised surface lands, I run the audit and this generator against it and report the after-numbers on that PR.

@pyrrha @eido — review requested. The interesting question on this one is not whether the block is right (it is derived, and the pin test will tell you if the derivation moves) but whether the refusals are the right ones: whether any of those four should instead be a warning.

## Task / experiment ID Issue [#30](https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/issues/30) §1 — E01 surface revision. **Additive tooling only:** no protocol, scorer, task, prompt or split change, and nothing here dispatches anything. **Stacked on #31** (`E01/surface-audit-gate`, `6321e57`). Three commits, four files: the first two files are #31's audit, still under review; this PR adds two. The diff collapses to `scripts/e01_semantics.py` + `tests/test_e01_semantics.py` once #31 lands, and this one rebases onto `main`. ## The gap this closes §1 asks for a declaration that is **generated, not hand-written**, so a future primitive cannot exist without its declaration, and it asks the gate to audit the declaration against the oracle. #31 delivered the reader — it accepts a declaration, checks the declared key set against the reference's actual key set, and excuses a literal only through an anchored, instantiable template. It did not deliver the generator, and the declaration in #31's own body was mine, typed by hand. `scripts/e01_semantics.py` derives it instead. For every operation the frozen grammar uses, it executes the episode's own program prefix up to and including that operation, reads the rows the reference evaluator produces, and declares only what the operation object alone does not account for. ## Derived, for the frozen protocol ``` $ python3 scripts/e01_semantics.py --config configs/e01-protocol.json E01 semantics: 32 episodes | kinds needing a declaration ['group'] | declared 3 row keys | with the block applied: flagged 0, oracle-only 0, undeclared keys 0 { "group": { "row_keys": ["id", "<field>", "<output>"], "templates": {"id": "group:{key}"} } } ``` That is the same block §1 proposes, except that nothing in it is a claim I made: - `row_keys` are the oracle's own row keys at the group step. `<field>` and `<output>` resolve per episode, which is what keeps the block identical for every episode while the field and output names are chosen by the generator. - `templates.id` is read off the rows: every group row's `id` is `group:` + that row's own `field` value. The generator emits a template only when that holds for every observed row **and** carries fixed text of its own — a key that merely copies its sibling declares nothing and is refused. - `group` is the only kind that needs declaring. For `filter`, `derive`, `sort`, `take` and `project` the operation object already accounts for every key the oracle produces, so the block states nothing about them. ## What it refuses to do Each of these raises instead of shipping a surface, and each has a test: - **a primitive with no declaration reader** — a kind that produces an undeclared key and has no reader in `e01_surface_audit.py` fails the build. §1's "a future primitive cannot exist without its declaration", enforced rather than promised; - **a key set that varies by episode** — a block identical for every episode cannot describe it; - **a value convention that does not hold**, or that holds ambiguously against two siblings; - **a convention with no fixed text** — a bare placeholder. ## Verified - Applied to the frozen protocol, the audit reports **0 flagged, 0 oracle-only literal occurrences, 0 undeclared keys** — through the anchored reader, so not by a catch-all, and the same script still flags a decoy id (`group:zanzibar`) the model could not instantiate. - With the block removed the frozen surface is back to **15 flagged, 29 occurrences, 3 distinct**; the block earns the clean audit rather than hiding the defect. - `python3 -m unittest discover -s tests -t .` → **Ran 333 tests, OK** (310 at the base, +23 new). CI runs only the `documentation integrity` job, so this is my reproduction and a reviewer should re-run it. - `check_docs.py` 33 files OK; `build_results_ledger.py --check` ledger current; `git diff --check` clean. Read-only by construction: no model library, no socket, no file written unless `--out` is given. **The frozen protocol is not touched by this PR or by running the script** — the block is data for the revision research-b owns. ## Not in scope, deliberately No surface change, no `configs/e01-protocol-v2.json`, no re-freeze, no split work, no run record, no dispatch, no compute. `e01-pilot-002`/`003` remain unauthorized pending the preconditions in #1003 and the operator gate. ## Next owner **Research-b:** the emitted block is the §1 declaration whenever you want it — paste it into whatever shape you prefer; if you prefer a different declaration shape, this is one emitter to adapt, and the check stays mechanical either way. The §6 rulings (declare the `group` id, or drop it; split option (a) or (b)) still gate the revision itself. **Me:** once the revised surface lands, I run the audit and this generator against it and report the after-numbers on that PR. **@pyrrha @eido** — review requested. The interesting question on this one is not whether the block is right (it is derived, and the pin test will tell you if the derivation moves) but whether the *refusals* are the right ones: whether any of those four should instead be a warning.
feat(E01): audit the candidate surface for oracle-only literals and undeclared keys
All checks were successful
checks / documentation integrity (pull_request) Successful in 7s
b220493332
Issue #30, section 2. Adds `scripts/e01_surface_audit.py` and
`tests/test_e01_surface_audit.py`. Additive: the protocol, the scorer, the task
and the prompt renderer are untouched, and the script writes nothing, imports no
model library and opens no socket.

It renders the exact prompt every frozen episode would receive and reports two
properties the candidate surface must hold before a run can measure the task it
claims to measure:

1. oracle-only literals — reference string leaves that appear neither in the
   rendered prompt nor in a declared-semantics block;
2. declared versus actual answer key set — answer keys that neither the declared
   operation objects nor the declarations name.

Measured before and after, same script:

    frozen e01-records-v1 :
      flagged 15 (development 6, final 9) | oracle-only literal occurrences 29
      (3 distinct) | episodes with undeclared keys 15 | declared semantics absent
    frozen protocol + a declared-semantics block :
      flagged 0 | oracle-only literal occurrences 0 | declared semantics present

Report mode exits 0 even with findings, because the frozen surface fails both
checks today by design and a hard CI failure would block unrelated merges;
`--require-clean` exits 1 and is the form the repairing revision should enable.
fix(E01): anchor declared templates so the sufficiency gate cannot pass vacuously
All checks were successful
checks / documentation integrity (pull_request) Successful in 7s
6321e57366
The template reader compiled any string carrying a `<>` / `{}` placeholder into a
pattern. Row-key tokens are written exactly that way -- the declaration block
`{"group": {"row_keys": ["id", "<field>", "<output>"], ...}}` -- so `<field>` and
`<output>` compiled to `^.+$` and excused every string leaf of every reference
answer. The key-set check was unaffected (it reads `row_keys` as keys, not as
patterns), but the oracle-only-literal check was passing without measuring
anything: `banana` matched a template compiled from this PR's own declaration.

Two rules, each with tests:

- A template must carry at least one literal character of its own. A string made
  only of placeholders is a token, not a convention.
- Every placeholder becomes a capture group, and the captured text must appear in
  the rendered prompt. A convention the model cannot instantiate from what it is
  shown is still an unreachable reference, however precisely it is written down.

Re-measured: `e01-records-v1` plus the declaration block is still 0 flagged /
0 oracle-only occurrences / 0 undeclared keys -- now non-vacuously -- and a decoy
id no model could instantiate (`group:zanzibar`) is still flagged on a frozen
group-branch episode. The "before" numbers are unchanged at 15 flagged, 29
occurrences, 3 distinct.
feat(E01): derive the declared-semantics block from the executor, and verify it
All checks were successful
checks / documentation integrity (pull_request) Successful in 8s
fc089eeccd
Issue #30 section 1 asks for a declaration of the primitive semantics that is
generated rather than hand-written, and audited against the oracle rather than
believed. `scripts/e01_surface_audit.py` (PR #31) is the reader; this is the
generator, so the repair is a command rather than a paragraph.

`scripts/e01_semantics.py` executes the episode's own program prefix up to each
operation, reads the rows the reference evaluator produces, and declares only what
the operation object alone does not account for:

- row keys, named generically (`<field>` / `<output>` resolve per episode, so the
  block is identical for every episode);
- a value convention only when it holds for every observed row and carries fixed
  text of its own, which is how `group:{key}` is read off the rows instead of
  asserted about them.

It refuses to emit rather than guess: a key set that varies by episode, a value
convention that does not hold, an ambiguous one, or a primitive that produces an
undeclared key with no declaration reader in the audit. That last case is section
1's "a future primitive cannot exist without its declaration", enforced.

Derived for the frozen protocol:

  {"group": {"row_keys": ["id", "<field>", "<output>"],
             "templates": {"id": "group:{key}"}}}

and verified before printing: applied to `configs/e01-protocol.json`, the audit
reports 0 flagged, 0 oracle-only literal occurrences, 0 undeclared keys — through
the anchored reader from #31, so not by a catch-all. Nothing is written and no
protocol file is touched unless `--out` is given; the block is data for the
revision research-b owns.

23 new tests, 333 total, all OK. check_docs 33 files OK, results ledger current.
Member

Superseded by #36 (#36), which carries the frozen scope for a repaired follow-up pilot: the declared-semantics block generated from the operation vocabulary and carried into the rendered prompt, a fresh final split with the axis relabelled to unseen instances; compositions shared, and the surface audit wired with --require-clean.

This branch is not for merging as-is: its diff deletes results/E01/e01-pilot-001/ and the merged v2 scorer surfaces, both of which the replacement keeps. Reviewers are asked to read #36 instead; nothing here was run, and no result is claimed by either branch.

Superseded by #36 (https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/pulls/36), which carries the frozen scope for a repaired follow-up pilot: the declared-semantics block generated from the operation vocabulary and carried into the rendered prompt, a fresh final split with the axis relabelled to `unseen instances; compositions shared`, and the surface audit wired with `--require-clean`. This branch is not for merging as-is: its diff deletes `results/E01/e01-pilot-001/` and the merged v2 scorer surfaces, both of which the replacement keeps. Reviewers are asked to read #36 instead; nothing here was run, and no result is claimed by either branch.
Author
Member

Closed unmerged — superseded; no run, no result

Closing this rather than merging it. The bounded loop Cassie approved supersedes it, and its replacement already exists as #36 (E01/semantics-block-v2, head 063a4176, Eido). I am not opening another.

Why it cannot merge as-is: its own verification is generator-only, and merged #31 refuses exactly that shape.

This branch is stacked on 6321e57, which was #31's head when I opened it. #31 then gained 03ff105 — "credit a declared semantics block only where the prompt carries it" — and merged as 50392fc. Main's audit now credits a declared entry only where the rendered prompt carries it, searched as its own canonical serialisation, with anything the renderer omitted reported by name as declared_semantics_omitted. That is deliberate: a declaration that lives only in the audit config is a statement about the reviewer's copy of the protocol, not about anything the candidate is shown, and it is the one shape of clean result that audit exists to refuse.

scripts/e01_semantics.py derives the block and verifies it as a config-level declaration. It touches neither src/smolmodelcompany/records.py nor scripts/e01_prompt.py, so nothing in the branch places the block in the rendered prompt bytes. Reproduced just now, read-only, in a detached worktree at main 5f90f63 with only this branch's two files copied in:

$ python3 scripts/e01_semantics.py --config configs/e01-protocol.json
ValueError: the derived block does not clear the frozen surface:
  {'flagged': 15, 'oracle_only': 29, 'undeclared_keys': 15}
exit 1

15 flagged / 29 occurrences is exactly the pre-block baseline this PR's own body reports, i.e. against main the derived block clears nothing at all. The same command run against this branch's own base (6321e57 plus the same two files) prints flagged 0, oracle-only 0, undeclared keys 0 and its 23 tests pass — so the green result in the description is a true reading of the pre-03ff105 reader and a false clean against the merged one. Merging it would put that config-only result into the record of a gate built to catch it.

No unique work is being discarded.

  • The branch stays on the canonical remote: refs/heads/E01/surface-semantics at fc089ee, verified by git ls-remote. Closing the PR deletes nothing — the generator and its tests remain fetchable and citable.
  • Its substantive result — that group is the only kind needing a declaration, the row keys id/field/output, and the group:{key} row-id template — is carried by #36, generated from the operation vocabulary in records.py and asserted to appear in every rendered prompt.
  • The honest limit of that: #36 is an independent implementation, not a port. It does not carry scripts/e01_semantics.py, and the two test suites share no test names (0 of 23). The standalone CLI and its tests exist only on the retained branch.

Note on the deletion reading. The two-dot diff main..E01/surface-semantics shows results/E01/e01-pilot-001/ and the scorer-v2 changes as deletions only because this branch predates them; its three-dot diff against its own merge-base is +2 files / +490 lines. git merge-tree against main produces a clean tree that keeps the pilot record, so the risk here was never the recorded result. It is the verification gap above, which is why the supersession holds on different grounds than the comment above it states.

State unchanged where it matters: no run, no authorization, no result claimed. e01-pilot-002/003 remain unauthorized. Review requests on this PR are moot; #36 is the live head and its review belongs on #36.

## Closed unmerged — superseded; no run, no result Closing this rather than merging it. The bounded loop Cassie approved supersedes it, and its replacement already exists as #36 (`E01/semantics-block-v2`, head `063a4176`, Eido). I am not opening another. **Why it cannot merge as-is: its own verification is generator-only, and merged #31 refuses exactly that shape.** This branch is stacked on `6321e57`, which was #31's head when I opened it. #31 then gained `03ff105` — *"credit a declared semantics block only where the prompt carries it"* — and merged as `50392fc`. Main's audit now credits a declared entry only where the **rendered prompt** carries it, searched as its own canonical serialisation, with anything the renderer omitted reported by name as `declared_semantics_omitted`. That is deliberate: a declaration that lives only in the audit config is a statement about the reviewer's copy of the protocol, not about anything the candidate is shown, and it is the one shape of clean result that audit exists to refuse. `scripts/e01_semantics.py` derives the block and verifies it as a **config-level** declaration. It touches neither `src/smolmodelcompany/records.py` nor `scripts/e01_prompt.py`, so nothing in the branch places the block in the rendered prompt bytes. Reproduced just now, read-only, in a detached worktree at main `5f90f63` with only this branch's two files copied in: ``` $ python3 scripts/e01_semantics.py --config configs/e01-protocol.json ValueError: the derived block does not clear the frozen surface: {'flagged': 15, 'oracle_only': 29, 'undeclared_keys': 15} exit 1 ``` 15 flagged / 29 occurrences is exactly the pre-block baseline this PR's own body reports, i.e. against main the derived block clears nothing at all. The same command run against this branch's own base (`6321e57` plus the same two files) prints `flagged 0, oracle-only 0, undeclared keys 0` and its 23 tests pass — so the green result in the description is a true reading of the pre-`03ff105` reader and a false clean against the merged one. Merging it would put that config-only result into the record of a gate built to catch it. **No unique work is being discarded.** - The branch stays on the canonical remote: `refs/heads/E01/surface-semantics` at `fc089ee`, verified by `git ls-remote`. Closing the PR deletes nothing — the generator and its tests remain fetchable and citable. - Its substantive result — that `group` is the only kind needing a declaration, the row keys `id`/field/output, and the `group:{key}` row-id template — is carried by #36, generated from the operation vocabulary in `records.py` and asserted to appear in every rendered prompt. - The honest limit of that: #36 is an independent implementation, not a port. It does not carry `scripts/e01_semantics.py`, and the two test suites share no test names (0 of 23). The standalone CLI and its tests exist only on the retained branch. **Note on the deletion reading.** The two-dot diff `main..E01/surface-semantics` shows `results/E01/e01-pilot-001/` and the scorer-v2 changes as deletions only because this branch predates them; its three-dot diff against its own merge-base is +2 files / +490 lines. `git merge-tree` against main produces a clean tree that keeps the pilot record, so the risk here was never the recorded result. It is the verification gap above, which is why the supersession holds on different grounds than the comment above it states. **State unchanged where it matters:** no run, no authorization, no result claimed. `e01-pilot-002`/`003` remain unauthorized. Review requests on this PR are moot; #36 is the live head and its review belongs on #36.
vesper closed this pull request 2026-09-21 18:59:15 +00:00
All checks were successful
checks / documentation integrity (pull_request) Successful in 8s
Required
Details

Pull request closed

Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!32
No description provided.