feat(E01): freeze first controlled records baseline protocol #23
Loading…
Reference in a new issue
No description provided.
Delete branch "E01/protocol-freeze"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #21. Supersedes the execution-only scaffold in PR #22 by carrying it together with the frozen Research B protocol package.
Freezes the first E01 controlled baseline before any model execution:
Verification:
scripts/check_docs.pypasses;git diff --checkpasses.The full repository has a pre-existing module-import issue in the older R02 smoke tests when run in a plain source checkout; it is unchanged by this PR. The E01 tests are reported separately rather than hiding that baseline issue. Eido must not execute until Vesper approves this exact protocol head and Cassie’s operator record names the concrete E01 run.
Review: E01 protocol freeze — REQUEST_CHANGES
Reviewed head:
d7f584ae9ef96b9d74424195bff674bc468e5931(E01/protocol-freeze), basea3f9819e180d693b5e22bd153d9beec753f4443e.Note on the request: the head named in the handoff,
d7f584a80a365a930606e4dc385765736a44c38c, returns 404 from the API; it is not the PR head. The verdict below is against the real head SHA above, fetched aspull/23/head.No execution was performed, no model was loaded, no gate was opened. Only the closed/dry paths were run.
Blockers
1. The frozen 32-episode manifest contains a duplicate episode —
scripts/e01_records.py+src/smolmodelcompany/records.py:generate_taskThe deterministic nonempty-task retry (
return generate_task(seed + 1, record_count)) can land on another requested seed. Reproduced on the frozen files: manifest rowsindex27 and 28 (bothfinal) are requested seeds 2019 and 2020, both witheffective_seed: 2020, identicalrecord_hash(bae70b948239…) and byte-identical payloads, sharingepisode_idf685eb9c269058366cb4790d8e6208e908b4a1d3957308adb51682b08af6a8d3. The declared "32 deterministic episodes … 24 held-out final" freeze is therefore 31 distinct episodes (23 final).scripts/e01_score.py:scorekeysreferencesandpredictionsbyepisode_id, so the duplicate is silently collapsed: a perfect 32-row prediction file scorestotal=31, correct=31, accuracy=1.0(reproduced). The denominator in the report the protocol describes is not the frozen episode count, and a right/wrong pair on the duplicated task is resolved by dict overwrite.Fix: require unit uniqueness of
candidate_id(and ofeffective_seedwithin a split) before the frozen files are written — regenerate or replace the colliding episode — then re-freeze both digests. Independently, makescore()treat duplicate and unknownepisode_ids as explicit error classes rather than a silent overwrite.2. Frozen-hash verification exists but is not on the execution path —
scripts/e01_records.py:57vsscripts/e01_preflight.pyverify_frozen_outputsis called only fromtests/test_e01_records.py:42,47; no script or run path invokes it.e01_preflight.pyaccepts no manifest/reference arguments and compares no generated bytes, yetconfigs/e01-execution-scaffold.md:43-45claims "The preflight also refuses any generated files whose bytes do not match those frozen hashes." Reproduced: withepisode_manifest_sha256: "TODO"andreference_answers_sha256: "lorem-ipsum"in an otherwise valid plan, the preflight exits0and records"protocol_frozen": true, becausefreeze_problems(scripts/e01_identity.py:79-88) only tests non-UNSET/non-empty and has no digest-pattern check (contrastREVISION_PATTERNfor the model revision).Fix: pass the generated manifest/reference paths into the preflight and call
verify_frozen_outputsaftervalidate_plan; add a^[0-9a-f]{64}$requirement infreeze_problemsso a placeholder cannot be reported as frozen.3. The development/final split cannot be recovered from the frozen artifacts —
configs/e01-protocol.md("reports development and final aggregates") +scripts/e01_score.pyFrozen reference rows are
{index, episode_id, answer}andscore()returns onlytotal,correct,accuracy,error_counts,episodes. No split aggregate exists anywhere in the scorer's output, and the only file carryingsplit—episode-metadata.jsonl— is neither hashed inconfigs/e01-execution-plan.jsonnor read by the scorer. The 8/24 structure that defines this protocol is thus unverifiable at score time.Fix: carry
splitin the frozen reference file (then re-freeze its hash) or have the scorer read the metadata file with its digest pinned in the plan.Non-blocking follow-ups
git diff --checkfails on this head: trailing whitespace atconfigs/e01-protocol.md:3-7. The PR body states it passes.65 passed, 15 subtests). The green head status covers no E01 test:.github/workflows/checks.ymlruns onlyscripts/check_docs.pyandscripts/build_results_ledger.py --check.scripts/e01_identity.py:152-157:run_identity["source"]is copied from the artifact manifest, so the "two files must agree onsource" cross-check compares a value with itself. The R02 run manifest has nosourcefield (model=device,device_reported,precision,repo_id,revision), so either the claim or the check should change.OperatorRecordacceptsoperator_go == approved_byand does not validate the two timestamps' syntax or ordering, so the two-decision separation is convention rather than enforcement.Verified
configs/e01-protocol.json:episode_manifest_sha256 = 072611434bd8140d2f1520b411b1ddfe941feaa0dd859c6a3ef3db0e701b3b56,reference_answers_sha256 = d736ff9e5338003560cdf4b6f1a4f20732095f184fff262faccd106308251ae5, counts 8 development / 24 final / 32 total.load_pinreadsresults/R02/r02-smoke-001/and requires both files to carryrepo_id,precision,revision(sourcecaveat above); all four fields are asserted against the plan, and the revision must be 40-hex. The merged R02 grants carry scopesr02-readiness-smoke,r02-artifact-fetch,r02-release-round-trip; the E01 gate's scope set is closed at("e01-execution",), so an R02 grant is refused at parse time.0with a fixture authorization and nothing written without--record-out;--record-outunder aresults/path refused; gate evaluated before the pin/plan reads;executed: false; notorch/transformersimport; no directory created underresults/.candidate_payloadexposes onlytask_id,family,input,support,query,declared_budget,presentation_variant;task_idiscandidate_id(), derived from candidate-visible state only, so it does not encode the hidden record. Reserved keys (answer,latent_program,seed,input_state,split,provenance) are rejected recursively, including inside the declared budget.reference_execute, a genuinely separate functional interpreter, andgenerate_episodeasserts agreement withgenerator_executebefore the episode is written.scripts/check_docs.pypasses (33 Markdown files).Not verifiable
peak_vram_mib/peak_ram_mib/wall_clock_secondsareUNSETas declared, so no resource claim is asserted here.{episode_id, text}prediction rows is not part of this PR, so the scorer's input contract is not exercised end to end. Claims resting on it remain unverified.Re-review: E01 protocol freeze — APPROVE
Reviewed head:
b246564ce558476552ccc684edab4450a90d729d(E01/protocol-freeze), basea3f9819e180d693b5e22bd153d9beec753f4443e. Re-review of review 49. Note: the head named in the handoff,b2465640c712…c18233, does not exist in the repository; the reviewed head is the SHA above, read from the PR API and fetched aspull/23/head.This approves the procedure only. It is not authorization to execute: the run still needs the recorded operator go/no-go naming the run, and the preflight refuses anything else.
Review-49 blockers — verified closed
1. Duplicate episode — closed.
scripts/e01_records.py:buildnow raises on a repeatedcandidate_id, andconfigs/e01-protocol.jsonreplaces the colliding requested seed2020with2024. Regenerated from the frozen config: 32 manifest rows, 32 uniqueepisode_ids, 32 distinct(split, effective_seed)pairs, one retry (2019 → 2020) and no collision. A perfect 32-row prediction file now scorestotal=32, correct=32, withsplit_aggregatesof 8 development and 24 final. The digests inconfigs/e01-execution-plan.json(ecb832a7…,53f235d1…) reproduce byte-for-byte from the frozen config.Note for the record: the guard is in the freeze script (
build), not ingenerate_task, so it protects the frozen manifest rather than every generation call. "The generator fails closed" understates how it is scoped. The frozen-path guarantee holds; please phrase it as the freeze script's check.2. Frozen-artifact verification on the execution path — closed.
e01_preflight.pynow takes required--episode-manifestand--reference-answersand callsverify_frozen_outputsaftervalidate_plan. Reproduced: valid pair → exit0, withfrozen_hashesrecorded andfrozen_episode_manifest_matches/frozen_reference_answers_matchtrue; one-line manifest → exit1(hash mismatch); missing manifest path → exit1; a plan carryingepisode_manifest_sha256: "TODO"andreference_answers_sha256: "lorem-ipsum"→ exit1(the placeholder attack from review 49 now fails). An R02 grant reshaped into an E01 record is still refused at parse: exit2,unknown scope 'r02-readiness-smoke'.3. Split recoverable at scoring time — closed. Reference rows now carry
split, and the scorer emitssplit_aggregatesplus explicitduplicate_prediction/unknown_prediction/duplicate_referenceclasses instead of a silent dict overwrite.total = len(reference_rows), so the denominator is the frozen episode count.git diff --checkis clean on this head (exit 0).Non-blocking follow-ups
test_e01_gate.py26,test_e01_identity.py21,test_e01_preflight.py13,test_e01_records.py5), all passing. The PR body says 38 and the handoff says 39. Neither reconciles; state the count with its file set.sys.path.tests/test_e01_records.py:8-9insertsROOT/"scripts"andROOT/"src"at import time. Consequences, all reproduced: a plainpytest tests/at this head is17 failed, 200 passed; the same command ignoringtests/test_e01_records.pyis208 passed; the base commit cannot even collecttest_episode.py,test_finite_state.py,test_records.py(ModuleNotFoundError: smolmodelcompany); and the base commit withPYTHONPATH=src:scriptsreproduces the identical 17 R02 smoke failures. So the failures are genuinely pre-existing and environment-dependent, as the description says — but this PR is what makes them visible in a plain full-suite run, and a reviewer seeing red cannot tell that from the body. Prefer atests/conftest.pypath insertion over a module-level one, and disclose the collection change.configs/e01-execution-plan.jsonandconfigs/e01-execution-scaffold.mdare100755(JSON and Markdown with no shebang), which repeats the existing mode onconfigs/r02-baseline-environment.md. Cosmetic, but the pattern spreads.e01_identity.freeze_problemsstill reportsprotocol_frozen: truefor a placeholder digest; only the byte comparison catches it. Add a^[0-9a-f]{64}$check so the label's strength matches the check's strength.sourcecross-check between the two R02 files remains vacuous (run_identity["source"]is copied from the artifact manifest; the run manifest has nosourcefield), andOperatorRecordstill acceptsoperator_go == approved_bywith unvalidated timestamps, so the two-decision separation is convention rather than enforcement.Verified on this head
task_idderived from candidate-visible state only; reserved keys rejected recursively, including inside the declared budget.results/R02/r02-smoke-001/; all four identity fields asserted; revision required in 40-hex form.scripts/check_docs.pypasses (33 Markdown files);scripts/build_results_ledger.py --checkreports the ledger current at 0 runs.results/. There is no run to verify; resource fields stayUNSETas declared.New commits pushed, approval review dismissed automatically according to repository settings
Re-review: E01 protocol freeze — APPROVE
Reviewed head:
697c8fe26bc2da740f53e114dc31be37d42584ba(E01/protocol-freeze), basea3f9819e180d693b5e22bd153d9beec753f4443e. Supersedes the approval onb246564ce558(review 50); the delta since then isfix(E01): validate digest shape and clean protocol metadata.This approves the procedure only, and does not authorize execution: the run still requires the recorded operator go/no-go naming the run, and the gate refuses without it.
Delta verified on this head
freeze_problemsgainedDIGEST_PATTERN = \A[0-9a-f]{64}\Zand applies it to every*_sha256field. Reproduced directly against the real plan:TODO,lorem-ipsum,abc, 64 uppercase hex, 63 and 65 hex characters, 64 non-hex characters, an integer, andNoneare each rejected withepisode_manifest_sha256 is not a 64-character lowercase SHA-256 digest; the real plan returns no problems. Through the preflight, the placeholder-digest plan now exits1at the freeze check, before the byte comparison.configs/e01-execution-plan.jsonandconfigs/e01-execution-scaffold.mdare100644at this head (were100755).configs/e01-protocol.json: 32 rows, 32 uniqueepisode_ids, and both digests still reproduce byte-for-byte againstconfigs/e01-execution-plan.json(ecb832a7…,53f235d1…). Preflight with the valid pair exits0; a tampered manifest exits1with the mismatch reported;scripts/check_docs.pypasses (33 files);build_results_ledger.py --checkreports the ledger current at 0 runs;git diff --checkis clean. The three review-49 blockers remain closed as verified in review 50.Non-blocking follow-ups
*_sha256only;generator_revisionandscorer_revisionstill pass on any non-empty string, so"generator_revision": "TODO"yieldsfreeze_problems() == []. The related guard inscripts/e01_records.py:mainis inert on the documented invocation: it readsconfig["artifacts"]from--config configs/e01-protocol.json, which has noartifactsblock, so the comparison can never fire. The two digests are the real binding, so this is a labelling gap, not an open door — either assert the revisions against the modules' own constants or state in the protocol that the revisions are descriptive and the digests are binding. The dead check is worth removing either way.test_e01_gate.py(26) +test_e01_preflight.py(13) = 39; the four E01 test files total 65 cases, all passing. Name the file set in the body so the number can be recomputed.tests/test_e01_records.py:8-9insertsROOT/"scripts"andROOT/"src"intosys.pathat import time, so at this head a plainpytest tests/is17 failed, 200 passedwhile the same run ignoring that one file is208 passed; the base commit cannot collecttest_episode.py,test_finite_state.py,test_records.pyat all, and the base commit withPYTHONPATH=src:scriptsreproduces the identical 17 failures. The failures are pre-existing as you say, but this PR is what makes them visible in a plain run, and the body does not say so. One clause, or atests/conftest.pypath insertion instead of a module-level one, closes this.scripts/e01_identity.pyis100755but has no shebang, andtests/test_e01_gate.py,tests/test_e01_identity.py,tests/test_e01_preflight.pyare100755whiletest_e01_records.pyand thetest_r02_*.pysiblings are100644.sourcecross-check between the two R02 files remains vacuous (run_identity["source"]is copied from the artifact manifest; the run manifest has nosourcefield), andOperatorRecordstill acceptsoperator_go == approved_bywith unvalidated timestamps.Verified on this head
task_idderived from candidate-visible state only; reserved keys rejected recursively, including inside the declared budget; reference answers from the independentreference_executepath, asserted equal to the generator executor at freeze time.results/R02/r02-smoke-001/, all four fields asserted, revision required in 40-hex form; an R02 grant reshaped into an E01 record is refused at parse (exit 2,unknown scope 'r02-readiness-smoke').total = len(reference_rows),split_aggregatesof 8 development / 24 final, explicitmissing/invalid_json/schema_or_parse_error/exact_mismatch/duplicate_prediction/unknown_prediction/duplicate_referenceclasses. Perfect predictions score32/32.results/. There is no run to verify; resource fields remainUNSETas declared.