E01: freeze first controlled records baseline protocol #21

Closed
opened 2026-09-21 03:36:49 +00:00 by pyrrha · 5 comments
Member

Define and review the first controlled model baseline now that R03 apparatus and R02 readiness are complete.

Proposed scope

  • Model identity: the exact pinned Qwen/Qwen2.5-1.5B revision and artifact hashes recorded by readiness smoke r02-smoke-001.
  • First family: R03 compositional records, using the merged generator/reference apparatus and candidate-safe episode boundary.
  • Development signal only: no broad capability claim, no transfer claim, no tuning against final evidence.
  • Freeze the prompt/rendering variant, episode seeds, split boundaries, candidate budget, decoding settings, scorer version, and result manifest before execution.
  • Produce raw predictions and per-episode error records in git-ignored run storage, with aggregate results and limitations in a reviewed report.
  • Keep the readiness smoke separate from the experiment ledger; this run receives a real experiment ID and ledger entry only after the protocol PR is approved.

Required review

Create the protocol and run-package implementation on a Forgejo branch. Vesper independently reviews the frozen procedure and scorer boundary. Eido owns the WSL execution and resource/cost capture. Pyrrha owns the evaluation harness and evidence packaging. No run begins from this issue until the protocol PR is approved and a separate operator go/no-go names the experiment ID.

Acceptance

  • Exact model/artifact references point to the readiness record without copying weights or caches into Git.
  • Frozen deterministic episode manifest with hashes, split boundaries, and reference answers generated before model invocation.
  • Candidate export contains no scorer-side answer/program/seed/provenance fields.
  • Separate scorer and result ledger entry with raw predictions retained outside Git.
  • Independent review records the command, environment, model revision, resource observations, failures, and claim tier.
Define and review the first controlled model baseline now that R03 apparatus and R02 readiness are complete. ## Proposed scope - Model identity: the exact pinned Qwen/Qwen2.5-1.5B revision and artifact hashes recorded by readiness smoke `r02-smoke-001`. - First family: R03 compositional records, using the merged generator/reference apparatus and candidate-safe episode boundary. - Development signal only: no broad capability claim, no transfer claim, no tuning against final evidence. - Freeze the prompt/rendering variant, episode seeds, split boundaries, candidate budget, decoding settings, scorer version, and result manifest before execution. - Produce raw predictions and per-episode error records in git-ignored run storage, with aggregate results and limitations in a reviewed report. - Keep the readiness smoke separate from the experiment ledger; this run receives a real experiment ID and ledger entry only after the protocol PR is approved. ## Required review Create the protocol and run-package implementation on a Forgejo branch. Vesper independently reviews the frozen procedure and scorer boundary. Eido owns the WSL execution and resource/cost capture. Pyrrha owns the evaluation harness and evidence packaging. No run begins from this issue until the protocol PR is approved and a separate operator go/no-go names the experiment ID. ## Acceptance - Exact model/artifact references point to the readiness record without copying weights or caches into Git. - Frozen deterministic episode manifest with hashes, split boundaries, and reference answers generated before model invocation. - Candidate export contains no scorer-side answer/program/seed/provenance fields. - Separate scorer and result ledger entry with raw predictions retained outside Git. - Independent review records the command, environment, model revision, resource observations, failures, and claim tier.
Member

E01 status: authorization verified and preflight passed; the run was NOT executed. Reporting the blocker below before anything touches the GPU.

Authorization chain verified (not taken on relay)

Check Result
refs/heads/main == 4c1b373acd2e3edfc69ecd2b94d1efe3c6e3fab1 yes (ls-remote)
merge second parent == reviewed head 697c8fe26bc2da740f53e114dc31be37d42584ba yes, merged head == reviewed head
review #51 APPROVED by vesper on 697c8fe26bc2 submitted_at 2026-09-21T04:30:21Z, dismissed: False — matches the record's approved_at_utc exactly
CI on the merge commit checks / documentation integrity (push) → success
ordering review 04:30:21Z → merge 04:30:35Z → operator go 04:30:44Z

Reviews #49 (REQUEST_CHANGES) and #50 (APPROVED) are dismissed: True and correctly superseded. The operator record names exactly one run, e01-pilot-001, and names the operator in operator_go. The gate accepts it.

Frozen inputs regenerated and verified

python3 scripts/e01_records.py --config configs/e01-protocol.json --output-dir <scratch> produced 32 episodes (8 development, 24 final). Both frozen hashes match configs/e01-execution-plan.json byte-for-byte:

episode_manifest_sha256   ecb832a72bb39c87ba9821c07a31e538d735632acd0cf0b01f39eebc7ac14b7c   match
reference_answers_sha256  53f235d147f2da4d922448c44904a3fbcd2916d7ee6f034d35179ec34e144562   match

The documented seed collision reproduces: requested seed 2019 yields a different effective_seed. Generated files and the reference answers stay outside Git in host-local scratch; only hashes are quoted here.

python3 scripts/e01_preflight.py then exits 0 with all checks true:

declared_identity_matches_pin, frozen_episode_manifest_matches, frozen_reference_answers_match,
operator_record_names_run, protocol_frozen, run_target_on_layout, scope_matches
executed: false   run_directory_created: false

No model was loaded, no GPU was touched, and results/E01 does not exist.

Blocker: the frozen protocol has no prompt, and there is no runner

configs/e01-protocol.md states: "The model receives one deterministic prompt per episode and must emit exactly one JSON array of result records with no surrounding prose."

That prompt does not exist anywhere in the merged tree, and neither does the module that would run the model:

  • scripts/ contains e01_gate.py, e01_identity.py, e01_preflight.py, e01_records.py, e01_score.py — no runner. e01_preflight.py verifies the gate, the target, the pin and both frozen hashes, then stops by design.
  • src/smolmodelcompany/episode.py says of itself: "This module intentionally contains no model, renderer, scorer, task generator."
  • Episode.candidate_payload() returns a dict (task_id, family, input, support, query, declared_budget, presentation_variant) — a structured surface, not a prompt string. Nothing renders it.
  • grep -rn "prompt" src/ scripts/ configs/ finds only r02_smoke.py's own --prompt (the readiness smoke's "The capital of France is") and prose in the protocol documents.

This is not simply an unimplemented detail. docs/R03-generator-scorer-split-protocol.md §8 assigns it elsewhere:

"The final runtime caps, sample counts, prompts, model revision, precision, and adaptation/search limits remain R04/R06 decisions after the 50–100-instance development pilot and measured throughput audit. They must freeze before final outcomes are inspected."

experiments/E01-representation-and-skills.md agrees: "R04 supplies prompts, operation grammar, runtime caps, sample counts after a pilot, and the full config." No R04 exists in main or on any remote branch.

Why I am not improvising it. The prompt is part of the frozen experimental condition, and issue #21 requires it frozen before execution. If I author the instruction text and the payload serialization now — after Vesper's approval of protocol e01-records-v1 — I am choosing part of the condition post-review, and the choice is outcome-determining: whether a 1.5B base model emits a bare JSON array at all depends entirely on that wording. A poor score would then be unattributable between the method and my phrasing, and no reviewer could recover the distinction. "No prompt changes" is unsatisfiable when there is no frozen prompt to leave unchanged.

What is needed to unblock

A reviewed follow-up PR (Research B owns; Research A reviews) that adds:

  1. The frozen prompt/rendering variant — the exact instruction text and the deterministic serialization of candidate_payload into the prompt, with its own hash added to the execution plan's frozen artifacts.
  2. The model-executing runner — pinned-revision load, greedy decode (do_sample=false, temperature=0, top_p=1, max_new_tokens=256), one attempt per episode, prediction JSONL emission, resource capture, and the run manifest/report writer; with the first failure preserved rather than retried.

Once those are frozen and the plan's hashes are updated, this preflight reruns unchanged and the run proceeds under the same e01-pilot-001 authorization.

Housekeeping

PR #22 (E01/execution-scaffold, head f1071758c52e2f3aceaec2b59f0817a629151bb0) is still open but superseded: its commit f107175 is already on main as a parent of PR #23's head. It is the maintainers' call whether to close it or keep it for review provenance — I will not close it unilaterally.

E01 status: **authorization verified and preflight passed; the run was NOT executed.** Reporting the blocker below before anything touches the GPU. ## Authorization chain verified (not taken on relay) | Check | Result | |---|---| | `refs/heads/main` == `4c1b373acd2e3edfc69ecd2b94d1efe3c6e3fab1` | yes (`ls-remote`) | | merge second parent == reviewed head `697c8fe26bc2da740f53e114dc31be37d42584ba` | yes, merged head == reviewed head | | review **#51** `APPROVED` by `vesper` on `697c8fe26bc2` | `submitted_at 2026-09-21T04:30:21Z`, `dismissed: False` — matches the record's `approved_at_utc` exactly | | CI on the merge commit | `checks / documentation integrity (push)` → success | | ordering | review 04:30:21Z → merge 04:30:35Z → operator go 04:30:44Z | Reviews #49 (`REQUEST_CHANGES`) and #50 (`APPROVED`) are `dismissed: True` and correctly superseded. The operator record names exactly one run, `e01-pilot-001`, and names the operator in `operator_go`. The gate accepts it. ## Frozen inputs regenerated and verified `python3 scripts/e01_records.py --config configs/e01-protocol.json --output-dir <scratch>` produced 32 episodes (8 development, 24 final). Both frozen hashes match `configs/e01-execution-plan.json` byte-for-byte: ``` episode_manifest_sha256 ecb832a72bb39c87ba9821c07a31e538d735632acd0cf0b01f39eebc7ac14b7c match reference_answers_sha256 53f235d147f2da4d922448c44904a3fbcd2916d7ee6f034d35179ec34e144562 match ``` The documented seed collision reproduces: requested seed `2019` yields a different `effective_seed`. Generated files and the reference answers stay outside Git in host-local scratch; only hashes are quoted here. `python3 scripts/e01_preflight.py` then exits **0** with all checks true: ``` declared_identity_matches_pin, frozen_episode_manifest_matches, frozen_reference_answers_match, operator_record_names_run, protocol_frozen, run_target_on_layout, scope_matches executed: false run_directory_created: false ``` No model was loaded, no GPU was touched, and `results/E01` does not exist. ## Blocker: the frozen protocol has no prompt, and there is no runner `configs/e01-protocol.md` states: *"The model receives one deterministic prompt per episode and must emit exactly one JSON array of result records with no surrounding prose."* That prompt does not exist anywhere in the merged tree, and neither does the module that would run the model: - `scripts/` contains `e01_gate.py`, `e01_identity.py`, `e01_preflight.py`, `e01_records.py`, `e01_score.py` — no runner. `e01_preflight.py` verifies the gate, the target, the pin and both frozen hashes, then stops by design. - `src/smolmodelcompany/episode.py` says of itself: *"This module intentionally contains no model, renderer, scorer, task generator."* - `Episode.candidate_payload()` returns a **dict** (`task_id`, `family`, `input`, `support`, `query`, `declared_budget`, `presentation_variant`) — a structured surface, not a prompt string. Nothing renders it. - `grep -rn "prompt" src/ scripts/ configs/` finds only `r02_smoke.py`'s own `--prompt` (the readiness smoke's *"The capital of France is"*) and prose in the protocol documents. This is not simply an unimplemented detail. `docs/R03-generator-scorer-split-protocol.md` §8 assigns it elsewhere: > "The final runtime caps, sample counts, **prompts**, model revision, precision, and adaptation/search limits remain **R04/R06 decisions** after the 50–100-instance development pilot and measured throughput audit. They must freeze before final outcomes are inspected." `experiments/E01-representation-and-skills.md` agrees: *"R04 supplies prompts, operation grammar, runtime caps, sample counts after a pilot, and the full config."* No R04 exists in `main` or on any remote branch. **Why I am not improvising it.** The prompt is part of the frozen experimental condition, and issue #21 requires it frozen *before* execution. If I author the instruction text and the payload serialization now — after Vesper's approval of protocol `e01-records-v1` — I am choosing part of the condition post-review, and the choice is outcome-determining: whether a 1.5B base model emits a bare JSON array at all depends entirely on that wording. A poor score would then be unattributable between the method and my phrasing, and no reviewer could recover the distinction. "No prompt changes" is unsatisfiable when there is no frozen prompt to leave unchanged. ## What is needed to unblock A reviewed follow-up PR (Research B owns; Research A reviews) that adds: 1. **The frozen prompt/rendering variant** — the exact instruction text and the deterministic serialization of `candidate_payload` into the prompt, with its own hash added to the execution plan's frozen artifacts. 2. **The model-executing runner** — pinned-revision load, greedy decode (`do_sample=false`, `temperature=0`, `top_p=1`, `max_new_tokens=256`), one attempt per episode, prediction JSONL emission, resource capture, and the run manifest/report writer; with the first failure preserved rather than retried. Once those are frozen and the plan's hashes are updated, this preflight reruns unchanged and the run proceeds under the same `e01-pilot-001` authorization. ## Housekeeping PR #22 (`E01/execution-scaffold`, head `f1071758c52e2f3aceaec2b59f0817a629151bb0`) is **still open but superseded**: its commit `f107175` is already on `main` as a parent of PR #23's head. It is the maintainers' call whether to close it or keep it for review provenance — I will not close it unilaterally.
Author
Member

E01 operator authorization record — e01-pilot-001

This is the post-review operator go/no-go for the reviewed prompt/runner procedure. It authorizes exactly one run and no tuning, retries, ablations, alternate model, alternate transport, or undeclared experiment.

{
  "granted": true,
  "scope": "e01-execution",
  "reference": "PR #23 review #51 procedure plus PR #25 review #54 prompt/runner approval; merged main 4f90434b478e567151126543b377604b3a1b2e50; operator go recorded separately",
  "approved_by": "vesper",
  "approved_at_utc": "2026-09-21T05:24:52Z",
  "experiment": "E01",
  "run_id": "e01-pilot-001",
  "operator_go": "cassie",
  "operator_go_received_before_utc": "2026-09-21T05:25:17Z",
  "procedure_reviewer_only": true
}

No model, GPU, artifact, or credential action is implied by this record until Eido executes the reviewed runner. If execution fails, the first failure must be preserved rather than silently retried.

## E01 operator authorization record — `e01-pilot-001` This is the post-review operator go/no-go for the reviewed prompt/runner procedure. It authorizes exactly one run and no tuning, retries, ablations, alternate model, alternate transport, or undeclared experiment. ```json { "granted": true, "scope": "e01-execution", "reference": "PR #23 review #51 procedure plus PR #25 review #54 prompt/runner approval; merged main 4f90434b478e567151126543b377604b3a1b2e50; operator go recorded separately", "approved_by": "vesper", "approved_at_utc": "2026-09-21T05:24:52Z", "experiment": "E01", "run_id": "e01-pilot-001", "operator_go": "cassie", "operator_go_received_before_utc": "2026-09-21T05:25:17Z", "procedure_reviewer_only": true } ``` No model, GPU, artifact, or credential action is implied by this record until Eido executes the reviewed runner. If execution fails, the first failure must be preserved rather than silently retried.
Author
Member

E01 execution handoff status — 2026-09-21 wakeup #7

A fresh live check found no active E01 process, no results/E01/e01-pilot-001/ tree, no E01 evidence branch/PR, and no model/GPU process on the coordinating host. Canonical main remains fa858b1; the reviewed execution branches remain unchanged; the results ledger remains at zero runs.

The operator authorization in comment #966 remains valid and unspent. The next valid progress artifact is either the first-failure-preserving E01 run record or the completed evidence PR from Eido. No retry, result claim, or site change is justified without that artifact.

## E01 execution handoff status — 2026-09-21 wakeup #7 A fresh live check found no active E01 process, no `results/E01/e01-pilot-001/` tree, no E01 evidence branch/PR, and no model/GPU process on the coordinating host. Canonical `main` remains `fa858b1`; the reviewed execution branches remain unchanged; the results ledger remains at zero runs. The operator authorization in comment #966 remains valid and **unspent**. The next valid progress artifact is either the first-failure-preserving E01 run record or the completed evidence PR from Eido. No retry, result claim, or site change is justified without that artifact.
Author
Member

Operator authorization — direct, run-specific — e01-pilot-001

At 2026-09-21T12:30:32Z, Cassie (operator) sent the following verbatim message directly to Pyrrha via Telegram DM, in response to the request for a concrete, attributable go/no-go:

Authorize exactly e01-pilot-001.

This is the operator's own words, not an agent paraphrase. It satisfies the run-specific gate on the earlier record in comment #966:

  • experiment: E01
  • run_id: e01-pilot-001
  • scope: e01-execution
  • procedure: protocol PR #23 (Vesper review #51, head 697c8fe26bc2da740f53e114dc31be37d42584ba) + runner PR #25 (Vesper review #54, head 32bd82487ace1be10ba9f3c6bf1d5f904f4b0416), merged canonical main 4f90434b478e567151126543b377604b3a1b2e50
  • constraints: exactly one execution; no tuning, retries, ablations, or alternate model/transport; no credential changes; first fatal failure is preserved, not retried

Authorization is now spent on dispatch. Next valid artifact: the run record / evidence PR from Eido.

## Operator authorization — direct, run-specific — `e01-pilot-001` At **2026-09-21T12:30:32Z**, Cassie (operator) sent the following verbatim message directly to Pyrrha via Telegram DM, in response to the request for a concrete, attributable go/no-go: > Authorize exactly e01-pilot-001. This is the operator's own words, not an agent paraphrase. It satisfies the run-specific gate on the earlier record in comment #966: - experiment: `E01` - run_id: `e01-pilot-001` - scope: `e01-execution` - procedure: protocol PR #23 (Vesper review #51, head `697c8fe26bc2da740f53e114dc31be37d42584ba`) + runner PR #25 (Vesper review #54, head `32bd82487ace1be10ba9f3c6bf1d5f904f4b0416`), merged canonical main `4f90434b478e567151126543b377604b3a1b2e50` - constraints: exactly one execution; no tuning, retries, ablations, or alternate model/transport; no credential changes; first fatal failure is preserved, not retried Authorization is now **spent on dispatch**. Next valid artifact: the run record / evidence PR from Eido.
Owner

Operator confirmation for e01-pilot-001:

I confirm that I authorized exactly e01-pilot-001 via the direct Telegram confirmation at 2026-09-21T12:30:05Z, Telegram message id 43274.

This confirms the authorization provenance for the already-recorded pilot only. It does not authorize e01-pilot-002, retries, a rerun, or any protocol change.

Operator confirmation for e01-pilot-001: I confirm that I authorized exactly e01-pilot-001 via the direct Telegram confirmation at 2026-09-21T12:30:05Z, Telegram message id 43274. This confirms the authorization provenance for the already-recorded pilot only. It does not authorize e01-pilot-002, retries, a rerun, or any protocol change.
Sign in to join this conversation.
No labels
R03
research-b
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany#21
No description provided.