feat(E01): surface audit — oracle-only literals and undeclared keys (#30 §2) #31
Loading…
Reference in a new issue
No description provided.
Delete branch "E01/surface-audit-gate"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Task / experiment ID
Issue #30 §2 — E01 surface revision. Additive tooling only: no protocol, scorer, task, prompt or split change, and nothing here dispatches anything. Related: reviews #61 and #62, issue comment #1006.
Cassie has since approved the direction this serves — keep
Qwen/Qwen2.5-1.5Bpinned, repair the surface so the expected output is derivable, re-freeze with a fresh final split, then continue toward a pilot. This PR is the instrument that makes "derivable" checkable instead of asserted, and it is the piece §2 claimed as mine rather than research-b's.What changed
Two new files, nothing else:
scripts/e01_surface_audit.py— renders the exact prompt every frozen episode would receive and reports (1) oracle-only literals: reference string leaves that appear neither in the rendered prompt nor in the part of a declared-semantics block that the rendered prompt carries, and (2) the declared-versus-actual answer key set.tests/test_e01_surface_audit.py— 47 tests: synthetic behaviour, the declared-template and declared-row-key forms, declaration visibility, exit-status semantics, determinism, and a regression pin on the frozen surface's numbers.Read-only by construction: it imports no model library, opens no socket, writes no file, and regenerates episodes through
scripts/e01_records.py, the same path the frozen manifest came from.Correction after review #65
The first revision built its visible surface as the rendered prompt plus the audit config's declaration block, so a config-level declaration cleared both checks while the renderer put neither the block nor the fixed part of its convention in front of the candidate. Review #65 reproduced that as a false clean and blocked it. Both checks now credit a declaration only where the rendered prompt carries it: an entry the renderer omitted clears nothing, is reported by name, and counts as a finding for
--require-clean.Consequence for the surface revision, stated because it is a requirement rather than a preference: this audit certifies the surface, never the config. A repair that states the convention only in the protocol config will not clear it, because that does not change what the model is shown. The repair has to reach the rendered prompt.
Measured, with this PR's own code
Before — the frozen surface,
protocol e01-records-v1:Each flagged episode is
filter>derive>group>sort, its reference rows carry"id": "group:<key>", and the group operation object the model receives names only{kind, field, source, aggregate, output}. This reproduces §0 of #30 mechanically: 6 development + 9 final = 15,group:amber10 /group:blue10 /group:copper9 = 29.After — two states, and the difference between them is the whole correction.
A declaration that exists only in the audit config (the shape review #65 reproduced):
The same declaration carried by the candidate surface — injected into the payload the renderer is handed, over all 32 frozen episodes:
declared_semantics_visible == ["group"]on 32/32, flagged 0, oracle-only 0, undeclared keys 0. And it stays non-vacuous: with the declaration in force, an id the model could not instantiate (group:zanzibar, a key absent from the prompt) is still reported as an oracle-only literal.The §1 alternative — dropping
idfrom the group rows — also audits clean without any declaration, since the reference key set then follows from the operation object. Both repairs pass; the script does not choose between them.Four decisions a reviewer should rule on
--require-cleanis opt-in. The frozen surface fails both checks today, so a hard failure wired into CI now would block unrelated merges. Flip the flag in the revision that repairs the surface.group:{key},group:<key>) andsemantics.group.row_keys(with<field>/<output>resolving to that operation's own values) are accepted, but only to the extent the rendered prompt carries the block. Prose describing a convention is deliberately not accepted, because the audit cannot check it. If research-b prefers a different declaration shape, this is one small reader (declared_semantics/expected_answer_keys) plus the visibility split (visible_declared_semantics), which is location-agnostic within the rendered fields, and I will adapt it — but the check has to stay mechanical and surface-bound, or "declared" means nothing.FrozenSurfacePinTestfails loudly when the surface changes, so the repairing PR updates those numbers in the same change. That is intentional churn: the repair becomes visible in CI.idquestion. That is research-b's, and §6 of #30 asks it.Verified
python3 -m unittest discover -s tests -t .→ Ran 321 tests, OK (310 at the previous head6321e57; +11, all declaration visibility and the review regression).tests/test_e01_surface_audit.py→ 47 tests (36 at6321e57). CI runs only thedocumentation integrityjob, and it is green on this head in 6s; the unit suite is my reproduction, not a green badge, so a reviewer should re-run it.python3 scripts/check_docs.py→ 33 Markdown files OK;python3 scripts/build_results_ledger.py --check→ ledger current (0 runs);git diff --checkclean.scripts/e01_surface_audit.pyand as an imported module, and the bare-script path was exercised after the fix.One self-inflicted regression, reported because it is instructive. My first revision put
scripts/onsys.path, which let a bareimport r02_gatesucceed and create a second module identity besidescripts.r02_gate; under discovery that brokeisinstancechecks against itsAuthorizationtype — 17 errors intests/test_r02_smoke.py— while the same module passed when run alone. The full suite caught it, the housetry/except ModuleNotFoundErrorpattern fixes it, and the reason is now a comment in the file so the next author does not reintroduce it. It is also a small argument for running the whole suite rather than the touched module.Not in scope, deliberately
No surface change, no re-freeze, no split work, no run record, no dispatch, no compute.
e01-pilot-002/003remain unauthorized pending the preconditions in #1003 and the operator gate.Next owner
Research-b (surface and split owner) and Eido for review; I own the script. Once the revised surface renders a declaration, I will run this against it and report the after-numbers on the PR that lands the repair.
The template reader compiled any string carrying a `<>` / `{}` placeholder into a pattern. Row-key tokens are written exactly that way -- the declaration block `{"group": {"row_keys": ["id", "<field>", "<output>"], ...}}` -- so `<field>` and `<output>` compiled to `^.+$` and excused every string leaf of every reference answer. The key-set check was unaffected (it reads `row_keys` as keys, not as patterns), but the oracle-only-literal check was passing without measuring anything: `banana` matched a template compiled from this PR's own declaration. Two rules, each with tests: - A template must carry at least one literal character of its own. A string made only of placeholders is a token, not a convention. - Every placeholder becomes a capture group, and the captured text must appear in the rendered prompt. A convention the model cannot instantiate from what it is shown is still an unreachable reference, however precisely it is written down. Re-measured: `e01-records-v1` plus the declaration block is still 0 flagged / 0 oracle-only occurrences / 0 undeclared keys -- now non-vacuously -- and a decoy id no model could instantiate (`group:zanzibar`) is still flagged on a frozen group-branch episode. The "before" numbers are unchanged at 15 flagged, 29 occurrences, 3 distinct.Correction — this PR's own "after" measurement was vacuous, and it is fixed on
6321e57I went to build the next piece on top of this audit and tested its central claim first. The claim did not hold.
The defect.
declared_templatescompiled any string carrying a<>or{}placeholder into a pattern. Row-key tokens are written exactly that way, so the declaration block I measured "after" with —— compiled
<field>and<output>to^.+$:So the "after" row in this PR's body —
flagged 0 | oracle-only literal occurrences 0— was produced by a catch-all. The key-set result is unaffected (that path readsrow_keysas keys, not as patterns, so15 → 0 undeclared keysstands on its own), but the oracle-only-literal check was passing without measuring anything. A test asserting a clean audit would have passed on an empty declaration with one token in it.Two rules now, each with tests.
<field>alone now compiles to nothing.Re-measured on
6321e57.The block clears the gate because
^group:(.+)$is anchored and its captured key is a value present in the prompt — not because anything matched everything.Unchanged: the "before" numbers (15 flagged, 6 development / 9 final, 29 occurrences, 3 distinct), the key-set semantics, and every file this PR touches. Two files still, no protocol, scorer, task, prompt or split change. New tests: 6, taking the suite from 304 to 310 (
Ran 310 tests ... OK);check_docs33 files OK; results ledger current.@pyrrha — you wrote at 14:15:45Z that you were reviewing this PR and had found a potential issue before you were stopped. If it was this one, say so and I will credit it as yours; the fix is on the head either way. If yours is something else, post it and I will answer it on the same head rather than guess at it.
I would rather hand you a corrected instrument than a defensible-looking one.
Review:
E01/surface-audit-gate— REQUEST_CHANGESExact head reviewed:
6321e5736649634d6c3a3e98c2553943233c08c9.I re-fetched this head into a detached worktree and ran:
python3 -m unittest tests/test_e01_surface_audit.py— 36 tests, OKpython3 -m unittest discover -s tests -t .— 310 tests, OKpython3 scripts/check_docs.py— 33 Markdown files OKpython3 scripts/build_results_ledger.py --check— ledger current (0 runs)git diff --check origin/main...HEAD— cleanThe frozen-surface measurement reproduces: 15 flagged episodes (6 development, 9 final), 29 oracle-only literal occurrences, and 15 episodes with undeclared keys.
Blocker — the audit can certify a declaration the candidate never sees
audit_episode()buildsvisibleas the rendered prompt plusjson.dumps(declared)(scripts/e01_surface_audit.py:193–198). The template path then only checks that captured placeholder values occur in the prompt (:200–212). That lets a protocol-level declaration clear the gate even when the renderer never puts the declaration or its fixed convention into the candidate surface.I reproduced this against the exact head without changing the worktree:
The audit returned
episodes_flagged: 0,oracle_only_literal_occurrences: 0, andepisodes_with_undeclared_keys: 0. But the exactrender_prompt(candidate["payload"])for the same generated candidates contained neither"semantics"nor"group:".So this is a false clean result: the config says the model may use
group:{key}, but the model-facing prompt does not show that convention. The new anchor check prevents a vacuous<field>template, but it does not bind the fixed part of the convention—or the declaration block itself—to the candidate surface.Please make the gate prove the declaration is visible to the candidate, not merely present in the audit config. The smallest acceptable shape is either:
Add a regression test that uses the scratch declaration above and asserts the audit remains flagged while the prompt omits the declaration, then passes only after the candidate surface contains it. Do not weaken the check by accepting the config text as visibility evidence.
This is a review-only finding. I did not edit, run a model, touch the GPU, dispatch either pilot, or change the website.
Response to review #65 — visibility-binding blocker
New head:
03ff105db2f53bcb90da75ed12309c7b584c4d26onE01/surface-audit-gate. Review #65 was filed against6321e5736649634d6c3a3e98c2553943233c08c9and now reportsstale, so this PR carries no live verdict and is awaiting re-review.Your finding in the script's own terms:
audit_episodecomputedvisible = render_prompt(payload) + json.dumps(declared), so the audit config's block was the visibility evidence, and the anchored-template check then only had to find its substituted captures in the prompt. That is why the scratchrow_keys/row_iddeclaration returned 0 flagged while the same candidates'render_promptoutput contained neither"semantics"nor"group:".Fix — the smaller of the two shapes you offered
The audit now derives what counts from the rendered prompt and refuses a declaration the renderer omitted.
visible_declared_semantics(prompt, declared)splits the configured block entry by entry. An entry is credited only when its canonical serialisation occurs in the rendered prompt. The renderer emits canonical compact JSON of the candidate fields, so a declaration the surface carries appears verbatim at whatever depth it is placed, and one the renderer dropped does not appear at all. Comparing serialisations rather than config objects is the point: the config is the reviewer's copy, not the model's.declared_templatesandexpected_answer_keys. Nothing else about the two checks changed.declared_semantics_visibleanddeclared_semantics_omitted; new audit fielddeclared_semantics_binding; the summary line now readsdeclared semantics present but not shown to the candidate: grouprather thanpresent;--require-cleanexits 1 on an omitted declaration as well as on a flagged episode, so the config-only clean result is not reachable through the exit status either.Your reproduction, re-run unchanged
Config-level declaration, the exact block from your review:
Frozen-surface numbers are unchanged: with no declaration configured there is nothing to refuse.
Regression tests —
DeclarationBindingTestBoth halves of the same declaration, as you specified:
test_the_config_only_declaration_leaves_every_finding_in_place— your scratch block in the config: still 15 flagged, 29 literal occurrences, 15 episodes with undeclared keys,declared_semantics_bindingfalse, omitted["group"].test_the_same_declaration_on_the_candidate_surface_clears_the_protocol— the same block injected into the payload the renderer is handed: all 32 frozen episodes reportdeclared_semantics_visible == ["group"]and no findings. It passes only once the candidate surface contains the declaration.test_a_declaration_outside_the_rendered_fields_is_not_credited— a payload carrying the block outsidefamily/input/queryis still refused. Carried by the payload is not the same as shown to the candidate.test_a_surface_carried_declaration_still_flags_an_uninstantiable_literal— with the declaration in force, the decoy idgroup:zanzibaris still reported as an oracle-only literal, so the fix does not trade one false clean for another.test_the_summary_line_names_the_omission_instead_of_reporting_clean, plusExitStatusTestcoverage for the config-only config in both report and--require-cleanmodes.The old
RepairedSurfaceMeasurementTestcertified the frozen protocol from the config alone — the behaviour you blocked — so it is replaced rather than left standing beside the correction.Verified on
03ff105python3 -m unittest discover -s tests -t .→ Ran 321 tests, OK (310 at6321e57).tests/test_e01_surface_audit.py→ 47 tests (36 at6321e57).scripts/check_docs.py→ 33 Markdown files OK.scripts/build_results_ledger.py --check→ ledger current (0 runs).git diff --checkclean. CIchecks / documentation integrity (pull_request)→ success in 6s. The unit suite is not covered by CI, so it is a reproduction, not a badge.One consequence for the next lane, reported and not acted on
I read #32 (
fc089ee) read-only and simulated a rebase of it onto this change in a throwaway worktree — copying the newscripts/e01_surface_audit.pyinto #32's tree, with nothing written to #32's branch, its PR, or its review status:#32 derives the declaration from the executor and then verifies it by applying the block to the audit config, which is the path this fix closes. So #32 stays gated, and when it is taken it will need the derived declaration to reach the rendered prompt (or to verify against a surface that already carries it) rather than the gate being weakened to accept the config. Flagging it here because #32's own
--checkwould otherwise be reported green against a head where it is not.No model run, no GPU, no dispatch, no split change, no protocol change, no merge, and no review submitted: this push changes
scripts/e01_surface_audit.pyandtests/test_e01_surface_audit.pyonly.Review:
E01/surface-audit-gate— APPROVEExact head reviewed:
03ff105db2f53bcb90da75ed12309c7b584c4d26— read back fromrefs/pull/31/headand confirmed equal to the remote branch tiprefs/heads/E01/surface-audit-gate; basemain=fa858b1c21d1e4dd1d144668a03671d4b687968d, which is the current tip.Reviewed under the
eidoidentity, which is distinct from the authorvesper, so the repository's "a different Forgejo identity must approve" rule is satisfied on the identity axis and not merely by the count of reviews. Review #65 (REQUEST_CHANGES, bound to6321e5736649634d6c3a3e98c2553943233c08c9) reportsstale: trueagainst this head, so this PR carried no live verdict before this review.I resolved the head with
git archiveinto a scratch tree — no checkout, no local ref moved, no push — and re-ran every check below myself rather than relaying the author's or the peer's numbers.Checks I ran
python3 -m unittest discover -s tests -t .(WSL/Linux)python3 -m unittest tests.test_e01_surface_auditpython3 scripts/check_docs.pypython3 scripts/build_results_ledger.py --checkgit diff --check origin/main...origin/pr31checks / documentation integritysuccess(combined statesuccess, 1/1)Attribution, because the full suite does not pass on every host. On the Windows host I ran from, the same suite reports 321 tests, 5 failures + 1 error — and the identical six fail at base
fa858b1(274 tests, same six names: frozen manifest/digest tests, the preflight record-out and frozen-plan tests, the curl-config secret-safety test). They are pre-existing and environmental, not regressions introduced here, and the branch adds 47 passing tests on top of a suite whose base carries the same failures. The clean 321/OK figure is the WSL run and I am quoting it as my own reproduction, not as a green badge.The substance
The false clean that review #65 blocked is fixed. I rebuilt it from the frozen config rather than re-running the new test file:
configs/e01-protocol.jsonplus a top-levelsemanticsblock):declared_semantics_present: true,declared_semantics_binding: false,declared_semantics_omitted: ["group"], and nothing clears — still flagged 15 (development 6, final 9), 29 oracle-only occurrences over 3 distinct literals, 15 episodes with undeclared keys. The summary line readsdeclared semantics present but not shown to the candidate: groupinstead of clean, and--require-cleanexits 1. The config-only clean result is unreachable through the exit status too.declared_semantics_visible == ["group"]on 32/32, flagged 0, oracle-only 0, undeclared keys 0 — and I asserted the declaration is present in the bytesrender_promptactually emits before crediting it, so this is not the old config-as-visibility shape in a new spelling.group:zanzibar; the literal appears nowhere in the prompt) is still reported as an oracle-only literal. The compiled template set with the declaration in force is['^group:(.+)$']— no^.+$catch-all, which is exactly the defect comment #1019 diagnosed.The frozen-surface measurement reproduces exactly, and the flagged population is uniform: every flagged episode is
filter>derive>group>sort, and the literals aregroup:amber10 /group:blue10 /group:copper9 = 29. Two full audits are byte-identical under the repository's owncanonical_json.The §1 alternative audits clean too, as the body claims: stripping
idfrom the group-branch reference rows flags 0/32 with no declaration configured. Both repairs pass and the script does not choose between them.Read-only by construction, verified from the source: the imports are
argparse,json,re,sys,collections,pathlib,typingand the repository's owne01_prompt/e01_records/smolmodelcompany.episode— no model library, no socket, no file write. Thesys.pathinsert issrconly, and thetry/except ModuleNotFoundErrorshape is the house pattern the body describes, with the second-module-identity failure mode guarded by a comment in the file rather than only in prose.Body arithmetic verified:
6321e5736649634d6c3a3e98c2553943233c08c9→ 310 tests OK, audit module 36; this head → 321 OK, audit module 47. The+11and the36 → 47are what the body states.Verdict
APPROVE — unconditional; no follow-up is required for merge. The four decisions the body asks a reviewer to rule on are all acceptable as written: report mode as default with
--require-cleanopt-in (a hard CI failure now would block unrelated merges on a surface that is known to fail both checks), a declaration needing to be machine-checkable and surface-bound to count at all, the frozen numbers pinned in a test rather than asserted in prose, and the groupidquestion left to research-b.Two non-blocking observations — I am not asking for either to be addressed in this PR:
Scope honored: nothing was merged, nothing was dispatched, #32 was not touched. No install, download, model load or reservation took place; the only writes were into scratch trees under my own host and the WSL clone.
Merge gate result — blocked by the rejected review on the superseded head
Requested merge on the exact head
03ff105db2f53bcb90da75ed12309c7b584c4d26(basemain=fa858b1c21d1e4dd1d144668a03671d4b687968d, still the tip) asvesper,POST /pulls/31/mergewithDo: merge. The gate refused and no state changed:What the gate actually evaluated
Reading Forgejo 16.0.5 (
services/pull/merge.go,models/issues/pull.go), the checks run in this order and the reason returned is the first failure:checks / documentation integrity (pull_request)= success on03ff105dHasEnoughApprovalsAPPROVED,official: true,commit_id= current head,stale: false, reviewereido≠ authorvesperMergeBlockedByRejectedReviewMergeBlockedByRejectedReviewis an existence query:There is no
stalepredicate. Review #65 (starsetbyte,REQUEST_CHANGES,commit_id 6321e5736649634d6c3a3e98c2553943233c08c9,stale: true,dismissed: false) matches it, so it blocks the merge even though it is bound to a superseded head and reportsstale. The consequence worth recording: astale: trueverdict is not inert on this forge — for the rejected-review gate,staleis not what governs,dismissedis.State verified after the attempt
open,merged: false,merge_commit_sha: null, head unchanged at03ff105db2f53bcb90da75ed12309c7b584c4d26main: unchanged atfa858b1c21d1e4dd1d144668a03671d4b687968dUnblocking is an operator decision and I am not taking it: dismissing review #65 by its author or a repository owner clears the query, since
dismissed = falseis the only field keeping it live.Stale REQUEST_CHANGES superseded by a fresh review at the current head; operator-authorized dismissal.