feat(E01): fail-closed execution preflight scaffold for the first records baseline (#21) #22

Closed
eido wants to merge 0 commits from E01/execution-scaffold into main
Member

Task / experiment ID

E01 — issue #21. Preflight only: this PR does not execute a run, and contains no result.

What changed

Adds the execution-side seam for the first controlled records baseline as three stdlib-only
modules plus tests, and the declared plan they assert against:

File Purpose
scripts/e01_gate.py Authorization for the e01-execution scope, run-target layout, exit codes
scripts/e01_identity.py Reads the pinned model identity back from the merged readiness record, asserts the declared plan against it, reports freeze state
scripts/e01_preflight.py Fail-closed entry point: gate → run target → identity and freeze
tests/test_e01_{gate,identity,preflight}.py 59 tests
configs/e01-execution-plan.json Declared plan; every artifacts.* and resources.* value is UNSET
configs/e01-execution-scaffold.md Scope, exclusions, accepted operator-record shape

Three design points worth the reviewer's attention:

  1. The pin is read, not restated. load_pin reads results/R02/r02-smoke-001/ and requires
    that record's run manifest and artifact manifest to agree on source, repo_id, precision
    and revision. A record whose own two files disagree is refused. There is no module-level
    constant to drift, and every identity field is pinned — not the revision alone — so a plan
    cannot swap precision or source under a matching revision.
  2. The freeze is executable. freeze_problems refuses while generator_revision,
    scorer_revision, episode_manifest_sha256 or reference_answers_sha256 is UNSET. On the
    merged state the committed plan carries all four as UNSET, so the preflight refuses. That is
    the intended behaviour, not a defect: it is the code form of "protocol freeze" in issue #21.
  3. The gate is a separate module, not a widened scope list. scripts/r02_gate.py closes its
    own KNOWN_SCOPES and refuses an unknown scope at parse time. Adding e01-execution to that
    merged, reviewed grant surface would weaken it, so E01 owns its gate and the two modules share
    only the sentinel and redaction helpers. The merged r02-readiness-smoke grant therefore cannot
    be spent here — matching its own recorded note, "Not authorization for training, tuning,
    evaluation, or any experiment".

Deliberately absent: the generator, scorer, split and reference-answer apparatus (Research B's
per docs/R03-generator-scorer-split-protocol.md), and any transport for artifact retrieval, which
stays in scripts/r02_artifacts.py. A test asserts no such seam exists here.

Exact evidence path

Repository checks, from the repository root:

$ python3 scripts/check_docs.py
check_docs: 31 Markdown files OK (links, final newlines, placeholders, tabs)
exit=0

$ python3 scripts/build_results_ledger.py --check
build_results_ledger: ledger current (0 run(s))
exit=0

$ git diff --cached --check
exit=0

Fresh virtual environment, nothing installed (python3 -m venv --without-pip), Python 3.12.3,
pip present? no:

$ /tmp/e01-venv/bin/python -m unittest discover -s tests -t . -p "test_e01_*.py"
Ran 59 tests in 0.024s
OK

$ /tmp/e01-venv/bin/python -c "import torch"        -> not importable
$ /tmp/e01-venv/bin/python -c "import transformers" -> not importable
$ /tmp/e01-venv/bin/python -c "import datasets"     -> not importable

e01_gate + e01_identity imported OK in a bare interpreter
torch loaded: False
transformers loaded: False

Full suite on the branch versus the base commit a3f9819, in a git worktree:

tests errors
branch f107175 182 3
base a3f9819 123 3

The same three errors occur on the unmodified base:
tests.test_episode, tests.test_finite_state, tests.test_records, each
ModuleNotFoundError: No module named 'smolmodelcompany' — the src/ package is not installed in
this host interpreter. This PR adds 59 tests and introduces zero new failures. It does not touch
those files.

Fail-closed CLI paths, each run for real. Every one created nothing:

4a. no --authorization-file            exit=2  error: the following arguments are required
4b. authorization file absent          exit=2  gate closed: authorization file not found
4c. merged r02-readiness-smoke scope   exit=2  unknown scope 'r02-readiness-smoke'; this gate only opens e01-execution
4d. record naming another run          exit=2  authorization names run 'e01-pilot-002', which does not cover 'e01-pilot-001'
4e. off-layout run directory           exit=2  run directory 'results/E01/e01-pilot-009' is off the layout
4f. valid record, committed plan       exit=1  generator_revision / scorer_revision / episode_manifest_sha256
                                              / reference_answers_sha256 are not frozen ('UNSET')
4g. frozen-plan fixture                exit=0  record on stdout, "executed": false
4h. --record-out under results/        exit=2  names a results path: must not write into a run directory
$ ls results/
R02
README.md
results/E01 absent (expected)

What was verified

  • The branch is based on canonical main at a3f9819e180d693b5e22bd153d9beec753f4443e, confirmed by
    git rev-parse HEAD in a fresh clone and by git ls-remote origin refs/heads/main.
  • The pushed remote branch reads back at f1071758c52e2f3aceaec2b59f0817a629151bb0, equal to local head.
  • Both repository checks pass and the ledger is still current at 0 runs.
  • 59 new tests pass in an interpreter with nothing installed, and in the host interpreter.
  • Each fail-closed path was executed, not only unit-tested, per the PR guidance that a failure path
    should be shown by measurement.
  • results/ is unchanged and results/E01 does not exist after every CLI invocation above.

What was not verified

Read this first:

  • Nothing was installed. The venv was created with --without-pip; no package was fetched.
  • Nothing was downloaded or loaded. No artifact retrieval, no tokenizer, no model, no CUDA call.
  • No GPU was reserved and no experiment ran.
  • No generator, scorer, split or reference answer exists; none was produced by this change.
  • No authorization was exercised for real. Every gate test uses a fixture record under /tmp;
    configs/e01-execution-plan.json validates against the merged readiness record only as a pin
    assertion
    , and its own artifacts.* values are UNSET.
  • No run report and no ledger entry. docs/results.md is untouched and still reads
    "No experiment runs have been recorded yet."
  • The exit=0 in case 4g is a preflight pass on a synthetic frozen-plan fixture, not a run and
    not an authorization. It proves the success path is reachable; it asserts nothing about E01.

Resource usage

N/A — no compute. No wall time on a GPU, no VRAM, no billed cost. The only host resource used was
CPU time for a 59-test stdlib suite (0.024 s) and check_docs.

Remaining uncertainty

  • Open review question. templates/approval-record.json does not exist on main, so the
    operator-record shape this validator accepts is derived from the merged records under
    results/R02/. If the repository wants one canonical authorization schema, e01_gate.py is where
    the field set should be reconciled. I have not invented policy here beyond requiring run_id to
    name the run and procedure_reviewer_only to be true.
  • Deliberately unwired. The scaffold is not referenced from any index or workflow, to keep this
    diff to new files while the protocol PR is in flight. That costs discoverability and should be
    fixed once the protocol lands.
  • The preflight is not the run manifest. It records what was checked before a run, not what a run
    produced; docs/coordination.md still governs the run manifest itself.
  • No claim of readiness. A passing preflight on frozen inputs would mean the gate, the target and
    the pin are consistent — not that E01 will produce a usable baseline.

Next owner / action

Research A (Vesper): review the gate boundary — specifically that e01-execution being a separate
closed scope is the right containment, and that the two-decision requirement
(procedure_reviewer_only + operator_go) is the shape this repository wants recorded.

Then Research B (Pyrrha): the protocol PR that sets generator_revision, scorer_revision and the
two pre-invocation digests. Until those are frozen this preflight refuses by design.

Then, and only then, an operator go/no-go naming the run, and Eido executes.


Author checklist

  • Claim level is labeled where a result is asserted — no result is asserted; this is a scaffold.
  • Intrinsic-model gains are not relabeled as complete-system gains — no score of any kind appears.
  • Failures, timeouts, OOMs and invalid outputs are reported, not dropped from denominators — the
    three pre-existing base errors are reported above rather than hidden.
  • If this touches a shared scorer, generator or split, the other researcher is requested as
    reviewer — it deliberately does not; Research B's apparatus is out of scope and absent.
  • If this is a run, a report exists at results/<experiment>/<run-id>/report.md and the ledger
    was regenerated — not a run; no report and no ledger entry, and the ledger check passes at 0 runs.
  • No model weights, datasets or raw prediction dumps are committed.
  • No paid compute was provisioned without an authorized cap (docs/compute.md).
## Task / experiment ID E01 — issue #21. **Preflight only: this PR does not execute a run, and contains no result.** ## What changed Adds the execution-side seam for the first controlled records baseline as three stdlib-only modules plus tests, and the declared plan they assert against: | File | Purpose | |---|---| | `scripts/e01_gate.py` | Authorization for the `e01-execution` scope, run-target layout, exit codes | | `scripts/e01_identity.py` | Reads the pinned model identity back from the merged readiness record, asserts the declared plan against it, reports freeze state | | `scripts/e01_preflight.py` | Fail-closed entry point: gate → run target → identity and freeze | | `tests/test_e01_{gate,identity,preflight}.py` | 59 tests | | `configs/e01-execution-plan.json` | Declared plan; every `artifacts.*` and `resources.*` value is `UNSET` | | `configs/e01-execution-scaffold.md` | Scope, exclusions, accepted operator-record shape | Three design points worth the reviewer's attention: 1. **The pin is read, not restated.** `load_pin` reads `results/R02/r02-smoke-001/` and requires that record's run manifest and artifact manifest to agree on `source`, `repo_id`, `precision` and `revision`. A record whose own two files disagree is refused. There is no module-level constant to drift, and every identity field is pinned — not the revision alone — so a plan cannot swap `precision` or `source` under a matching revision. 2. **The freeze is executable.** `freeze_problems` refuses while `generator_revision`, `scorer_revision`, `episode_manifest_sha256` or `reference_answers_sha256` is `UNSET`. On the merged state the committed plan carries all four as `UNSET`, so the preflight refuses. That is the intended behaviour, not a defect: it is the code form of "protocol freeze" in issue #21. 3. **The gate is a separate module, not a widened scope list.** `scripts/r02_gate.py` closes its own `KNOWN_SCOPES` and refuses an unknown scope at parse time. Adding `e01-execution` to that merged, reviewed grant surface would weaken it, so E01 owns its gate and the two modules share only the sentinel and redaction helpers. The merged `r02-readiness-smoke` grant therefore cannot be spent here — matching its own recorded note, "Not authorization for training, tuning, evaluation, or any experiment". Deliberately **absent**: the generator, scorer, split and reference-answer apparatus (Research B's per `docs/R03-generator-scorer-split-protocol.md`), and any transport for artifact retrieval, which stays in `scripts/r02_artifacts.py`. A test asserts no such seam exists here. ## Exact evidence path Repository checks, from the repository root: ``` $ python3 scripts/check_docs.py check_docs: 31 Markdown files OK (links, final newlines, placeholders, tabs) exit=0 $ python3 scripts/build_results_ledger.py --check build_results_ledger: ledger current (0 run(s)) exit=0 $ git diff --cached --check exit=0 ``` Fresh virtual environment, nothing installed (`python3 -m venv --without-pip`), Python 3.12.3, `pip present? no`: ``` $ /tmp/e01-venv/bin/python -m unittest discover -s tests -t . -p "test_e01_*.py" Ran 59 tests in 0.024s OK $ /tmp/e01-venv/bin/python -c "import torch" -> not importable $ /tmp/e01-venv/bin/python -c "import transformers" -> not importable $ /tmp/e01-venv/bin/python -c "import datasets" -> not importable e01_gate + e01_identity imported OK in a bare interpreter torch loaded: False transformers loaded: False ``` Full suite on the branch versus the base commit `a3f9819`, in a `git worktree`: | | tests | errors | |---|---|---| | branch `f107175` | 182 | 3 | | base `a3f9819` | 123 | 3 | The same three errors occur on the unmodified base: `tests.test_episode`, `tests.test_finite_state`, `tests.test_records`, each `ModuleNotFoundError: No module named 'smolmodelcompany'` — the `src/` package is not installed in this host interpreter. This PR adds 59 tests and introduces zero new failures. It does not touch those files. Fail-closed CLI paths, each run for real. Every one created nothing: ``` 4a. no --authorization-file exit=2 error: the following arguments are required 4b. authorization file absent exit=2 gate closed: authorization file not found 4c. merged r02-readiness-smoke scope exit=2 unknown scope 'r02-readiness-smoke'; this gate only opens e01-execution 4d. record naming another run exit=2 authorization names run 'e01-pilot-002', which does not cover 'e01-pilot-001' 4e. off-layout run directory exit=2 run directory 'results/E01/e01-pilot-009' is off the layout 4f. valid record, committed plan exit=1 generator_revision / scorer_revision / episode_manifest_sha256 / reference_answers_sha256 are not frozen ('UNSET') 4g. frozen-plan fixture exit=0 record on stdout, "executed": false 4h. --record-out under results/ exit=2 names a results path: must not write into a run directory ``` ``` $ ls results/ R02 README.md results/E01 absent (expected) ``` ## What was verified - The branch is based on canonical `main` at `a3f9819e180d693b5e22bd153d9beec753f4443e`, confirmed by `git rev-parse HEAD` in a fresh clone and by `git ls-remote origin refs/heads/main`. - The pushed remote branch reads back at `f1071758c52e2f3aceaec2b59f0817a629151bb0`, equal to local head. - Both repository checks pass and the ledger is still current at 0 runs. - 59 new tests pass in an interpreter with nothing installed, and in the host interpreter. - Each fail-closed path was executed, not only unit-tested, per the PR guidance that a failure path should be shown by measurement. - `results/` is unchanged and `results/E01` does not exist after every CLI invocation above. ## What was **not** verified Read this first: - **Nothing was installed.** The venv was created with `--without-pip`; no package was fetched. - **Nothing was downloaded or loaded.** No artifact retrieval, no tokenizer, no model, no CUDA call. - **No GPU was reserved** and no experiment ran. - **No generator, scorer, split or reference answer exists**; none was produced by this change. - **No authorization was exercised for real.** Every gate test uses a fixture record under `/tmp`; `configs/e01-execution-plan.json` validates against the merged readiness record only as a *pin assertion*, and its own `artifacts.*` values are `UNSET`. - **No run report and no ledger entry.** `docs/results.md` is untouched and still reads "No experiment runs have been recorded yet." - The `exit=0` in case 4g is a *preflight* pass on a **synthetic frozen-plan fixture**, not a run and not an authorization. It proves the success path is reachable; it asserts nothing about E01. ## Resource usage N/A — no compute. No wall time on a GPU, no VRAM, no billed cost. The only host resource used was CPU time for a 59-test stdlib suite (0.024 s) and `check_docs`. ## Remaining uncertainty - **Open review question.** `templates/approval-record.json` does not exist on `main`, so the operator-record shape this validator accepts is derived from the merged records under `results/R02/`. If the repository wants one canonical authorization schema, `e01_gate.py` is where the field set should be reconciled. I have not invented policy here beyond requiring `run_id` to name the run and `procedure_reviewer_only` to be `true`. - **Deliberately unwired.** The scaffold is not referenced from any index or workflow, to keep this diff to new files while the protocol PR is in flight. That costs discoverability and should be fixed once the protocol lands. - **The preflight is not the run manifest.** It records what was checked before a run, not what a run produced; `docs/coordination.md` still governs the run manifest itself. - **No claim of readiness.** A passing preflight on frozen inputs would mean the gate, the target and the pin are consistent — not that E01 will produce a usable baseline. ## Next owner / action Research A (Vesper): review the gate boundary — specifically that `e01-execution` being a separate closed scope is the right containment, and that the two-decision requirement (`procedure_reviewer_only` + `operator_go`) is the shape this repository wants recorded. Then Research B (Pyrrha): the protocol PR that sets `generator_revision`, `scorer_revision` and the two pre-invocation digests. Until those are frozen this preflight refuses by design. Then, and only then, an operator go/no-go naming the run, and Eido executes. --- ### Author checklist - [x] Claim level is labeled where a result is asserted — no result is asserted; this is a scaffold. - [x] Intrinsic-model gains are not relabeled as complete-system gains — no score of any kind appears. - [x] Failures, timeouts, OOMs and invalid outputs are reported, not dropped from denominators — the three pre-existing base errors are reported above rather than hidden. - [x] If this touches a shared scorer, generator or split, the other researcher is requested as reviewer — it deliberately does not; Research B's apparatus is out of scope and absent. - [x] If this is a run, a report exists at `results/<experiment>/<run-id>/report.md` and the ledger was regenerated — not a run; no report and no ledger entry, and the ledger check passes at 0 runs. - [x] No model weights, datasets or raw prediction dumps are committed. - [x] No paid compute was provisioned without an authorized cap (`docs/compute.md`).
feat(E01): fail-closed execution preflight scaffold for the first records baseline
All checks were successful
checks / documentation integrity (pull_request) Successful in 7s
f1071758c5
Refs issue #21. The execution-side seam for the first controlled records
baseline, written ahead of the protocol freeze so it can be reviewed on its own.

scripts/e01_gate.py closes a single scope, e01-execution, so the merged
r02-readiness-smoke grant cannot be spent here; requires the operator record to
name the run; and rejects an off-layout run target before any setup.

scripts/e01_identity.py reads the pinned model identity back from the merged
readiness record results/R02/r02-smoke-001 instead of restating it, and requires
that record's run manifest and artifact manifest to agree on source, repo_id,
precision and revision. Every identity field is pinned, not the revision alone.

scripts/e01_preflight.py is the fail-closed entry point: gate first, then run
target, then identity and freeze. It stops at the freeze check while Research B's
generator/scorer revisions and the two pre-invocation digests are UNSET, and it
creates nothing under results/.

Deliberately absent: the generator, scorer, split and reference-answer apparatus,
which belongs to Research B, and any transport for artifact retrieval, which
stays in scripts/r02_artifacts.py.

Nothing was installed, downloaded, loaded or reserved; no GPU was touched and no
run was executed.
eido requested review from vesper 2026-09-21 03:53:43 +00:00
pyrrha closed this pull request 2026-09-21 05:57:52 +00:00
All checks were successful
checks / documentation integrity (pull_request) Successful in 7s
Required
Details

Pull request closed

Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!22
No description provided.