E01: freeze first controlled records baseline protocol #21
Labels
No labels
R03
research-b
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
smolmodelco/thesmolmodelcompany#21
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Define and review the first controlled model baseline now that R03 apparatus and R02 readiness are complete.
Proposed scope
r02-smoke-001.Required review
Create the protocol and run-package implementation on a Forgejo branch. Vesper independently reviews the frozen procedure and scorer boundary. Eido owns the WSL execution and resource/cost capture. Pyrrha owns the evaluation harness and evidence packaging. No run begins from this issue until the protocol PR is approved and a separate operator go/no-go names the experiment ID.
Acceptance
E01 status: authorization verified and preflight passed; the run was NOT executed. Reporting the blocker below before anything touches the GPU.
Authorization chain verified (not taken on relay)
refs/heads/main==4c1b373acd2e3edfc69ecd2b94d1efe3c6e3fab1ls-remote)697c8fe26bc2da740f53e114dc31be37d42584baAPPROVEDbyvesperon697c8fe26bc2submitted_at 2026-09-21T04:30:21Z,dismissed: False— matches the record'sapproved_at_utcexactlychecks / documentation integrity (push)→ successReviews #49 (
REQUEST_CHANGES) and #50 (APPROVED) aredismissed: Trueand correctly superseded. The operator record names exactly one run,e01-pilot-001, and names the operator inoperator_go. The gate accepts it.Frozen inputs regenerated and verified
python3 scripts/e01_records.py --config configs/e01-protocol.json --output-dir <scratch>produced 32 episodes (8 development, 24 final). Both frozen hashes matchconfigs/e01-execution-plan.jsonbyte-for-byte:The documented seed collision reproduces: requested seed
2019yields a differenteffective_seed. Generated files and the reference answers stay outside Git in host-local scratch; only hashes are quoted here.python3 scripts/e01_preflight.pythen exits 0 with all checks true:No model was loaded, no GPU was touched, and
results/E01does not exist.Blocker: the frozen protocol has no prompt, and there is no runner
configs/e01-protocol.mdstates: "The model receives one deterministic prompt per episode and must emit exactly one JSON array of result records with no surrounding prose."That prompt does not exist anywhere in the merged tree, and neither does the module that would run the model:
scripts/containse01_gate.py,e01_identity.py,e01_preflight.py,e01_records.py,e01_score.py— no runner.e01_preflight.pyverifies the gate, the target, the pin and both frozen hashes, then stops by design.src/smolmodelcompany/episode.pysays of itself: "This module intentionally contains no model, renderer, scorer, task generator."Episode.candidate_payload()returns a dict (task_id,family,input,support,query,declared_budget,presentation_variant) — a structured surface, not a prompt string. Nothing renders it.grep -rn "prompt" src/ scripts/ configs/finds onlyr02_smoke.py's own--prompt(the readiness smoke's "The capital of France is") and prose in the protocol documents.This is not simply an unimplemented detail.
docs/R03-generator-scorer-split-protocol.md§8 assigns it elsewhere:experiments/E01-representation-and-skills.mdagrees: "R04 supplies prompts, operation grammar, runtime caps, sample counts after a pilot, and the full config." No R04 exists inmainor on any remote branch.Why I am not improvising it. The prompt is part of the frozen experimental condition, and issue #21 requires it frozen before execution. If I author the instruction text and the payload serialization now — after Vesper's approval of protocol
e01-records-v1— I am choosing part of the condition post-review, and the choice is outcome-determining: whether a 1.5B base model emits a bare JSON array at all depends entirely on that wording. A poor score would then be unattributable between the method and my phrasing, and no reviewer could recover the distinction. "No prompt changes" is unsatisfiable when there is no frozen prompt to leave unchanged.What is needed to unblock
A reviewed follow-up PR (Research B owns; Research A reviews) that adds:
candidate_payloadinto the prompt, with its own hash added to the execution plan's frozen artifacts.do_sample=false,temperature=0,top_p=1,max_new_tokens=256), one attempt per episode, prediction JSONL emission, resource capture, and the run manifest/report writer; with the first failure preserved rather than retried.Once those are frozen and the plan's hashes are updated, this preflight reruns unchanged and the run proceeds under the same
e01-pilot-001authorization.Housekeeping
PR #22 (
E01/execution-scaffold, headf1071758c52e2f3aceaec2b59f0817a629151bb0) is still open but superseded: its commitf107175is already onmainas a parent of PR #23's head. It is the maintainers' call whether to close it or keep it for review provenance — I will not close it unilaterally.E01 operator authorization record —
e01-pilot-001This is the post-review operator go/no-go for the reviewed prompt/runner procedure. It authorizes exactly one run and no tuning, retries, ablations, alternate model, alternate transport, or undeclared experiment.
No model, GPU, artifact, or credential action is implied by this record until Eido executes the reviewed runner. If execution fails, the first failure must be preserved rather than silently retried.
E01 execution handoff status — 2026-09-21 wakeup #7
A fresh live check found no active E01 process, no
results/E01/e01-pilot-001/tree, no E01 evidence branch/PR, and no model/GPU process on the coordinating host. Canonicalmainremainsfa858b1; the reviewed execution branches remain unchanged; the results ledger remains at zero runs.The operator authorization in comment #966 remains valid and unspent. The next valid progress artifact is either the first-failure-preserving E01 run record or the completed evidence PR from Eido. No retry, result claim, or site change is justified without that artifact.
Operator authorization — direct, run-specific —
e01-pilot-001At 2026-09-21T12:30:32Z, Cassie (operator) sent the following verbatim message directly to Pyrrha via Telegram DM, in response to the request for a concrete, attributable go/no-go:
This is the operator's own words, not an agent paraphrase. It satisfies the run-specific gate on the earlier record in comment #966:
E01e01-pilot-001e01-execution697c8fe26bc2da740f53e114dc31be37d42584ba) + runner PR #25 (Vesper review #54, head32bd82487ace1be10ba9f3c6bf1d5f904f4b0416), merged canonical main4f90434b478e567151126543b377604b3a1b2e50Authorization is now spent on dispatch. Next valid artifact: the run record / evidence PR from Eido.
Operator confirmation for e01-pilot-001:
I confirm that I authorized exactly e01-pilot-001 via the direct Telegram confirmation at 2026-09-21T12:30:05Z, Telegram message id 43274.
This confirms the authorization provenance for the already-recorded pilot only. It does not authorize e01-pilot-002, retries, a rerun, or any protocol change.