R02: checkpoint pins as committed data (--checkpoint), so a second checkpoint can be readied #54

Merged
eido merged 1 commit from E01/r02-checkpoint-pins into main 2026-09-25 02:04:28 +00:00
Member

Task / experiment ID

R02.1 readiness apparatus (issue #17). Apparatus change; no run. This is the change
that makes step 1 of e01-pilot-003 executable — fetching a second checkpoint — and
nothing else in the sequence moves without it.

Head d6876209b8f374a91dbbdba3dd57dbaaa55752ef, base main 6c996732de1f822aafc7a5f68e65a5e148162277.

What changed

Before this, the readiness apparatus pinned exactly one checkpoint in module constants
(r02_artifacts.REPO_ID/REVISION), and validate_manifest read those same constants through
PINNED_FIELDS. So "fetch the instruct checkpoint" was not executable: two checkpoints could only
be expressed by editing the code the manifest validator depends on, and the set of fetchable
revisions was not an artifact a reviewer could read.

configs/r02-checkpoints.json is now that artifact — one named entry per checkpoint, each carrying
its own repo_id, revision, precision, license and declared_files. Both entry points take
--checkpoint NAME; a name the config does not define fails closed with exit 2 before a
transport, a model library or a GPU is reached; no flag accepts a repository, a revision or a
pin file, so the set of fetchable checkpoints stays under review.

File Change
configs/r02-checkpoints.json new — the pin set: base (unchanged values) + instruct-1.5b
scripts/r02_checkpoints.py new — loader, strict validation, pin(name), UnknownCheckpoint/CheckpointConfigError (both GateClosed)
scripts/r02_artifacts.py manifest names its checkpoint; plan/fetch select an entry; a manifest checked against another entry is refused; module constants are derived from the config's default entry
scripts/r02_smoke.py the plan is built from the selected pin and records the checkpoint by name; a supplied --artifact-manifest must describe the same checkpoint and revision
configs/r02-baseline-environment.md documents the pin config, the selection flag and the two new refusals
tests/test_r02_checkpoints.py new — 30 cases: committed-config integrity, unknown-name refusal, loader refusals
tests/test_r02_artifacts.py, tests/test_r02_smoke.py selection and refusal cases (+10, +8)

The base entry keeps today's values exactly — Qwen/Qwen2.5-1.5B @
8faed761d45a263340a0528343f099c05c9a4323, bfloat16, the same seven declared files — so the
committed results/R02/r02-smoke-001/artifact-manifest.json, recorded before the checkpoint
field existed, still validates under an unchanged default. That is asserted directly in
test_the_committed_smoke_001_manifest_still_validates_under_the_default.

The new entry is instruct-1.5b: Qwen/Qwen2.5-1.5B-Instruct @
989aa7980e4cf806f80c7fef2b1adb7bc71aa306, the E01 paired regression arm (issue #53). It is in the
config because the pin set is the reviewed artifact — not because anything has been fetched.

Exact evidence path

  • Diff: git diff-tree -r --name-status 6c99673 d687620, exactly 8 files (3 new, 5 modified); no
    other worktree changes. Committed blobs are LF (git show HEAD:<file> | tr -cd '\r' | wc -c = 0
    for all 8).
  • LF export of base and head into a WSL-native path (/tmp, git archive with
    core.autocrlf=false; 0 CR bytes in both trees, 59 Markdown files each):
    • python3 scripts/check_docs.py → 59 Markdown files OK at both heads (exit 0).
    • python3 scripts/build_results_ledger.py --check → ledger current (6 run(s)) at both.
    • PYTHONPATH=src:. <VENV_ROOT>/venv-r02/bin/python -m unittest discover -s tests -t . → base
      Ran 567 tests in 45.0s … OK, head Ran 615 tests in 44.1s … OK (CPython 3.12.3 / torch
      2.14.0+cu126). The +48 tests are this change's; the zero failures in the pair is the point of
      it. Under the system interpreter, which has no torch, the same two trees report Ran 567/615 tests … FAILED (failures=4) — the same four by name at base and head
      (test_rd1_cli.test_cli_runs_bounded_cpu_smoke,
      test_rd1_p3_rules.test_d1_decision_rule_branches, test_rd1_p3_rules.test_p3_rule_cases_pass,
      test_rd1_smoke.test_torch_cases_pass_without_polluting_model_free_protocol_tests), all
      ModuleNotFoundError: No module named 'torch'. An environment artifact, recorded here so it is
      not mistaken for a regression.
  • On the Windows checkout the same two gates report problems that are checkout artifacts, present
    at base and at head and absent from the LF export: check_docs flags site/CLAUDE.md: missing final newline because the tracked symlink (120000 → AGENTS.md) materialises as a 9-byte text
    file, and the ledger check flags staleness only because render() builds its links from Path
    and emits ../results\E01\… with backslashes. Both are recorded here so the reviewer does not
    have to re-derive them.
  • CI on this head runs the workflow I read from the tree (.github/workflows/checks.yml): the docs
    job (check_docs + ledger + citations) and the E01 surface audit. It does not run the unit
    suite.
    The two gates above are CI-covered; the Ran 615 tests line is a local run at
    d687620, not CI.
  • Pin verification, metadata only, unauthenticated (GET /api/models/<repo>/revision/<sha>, the same request shape resolve_revision uses, no credential
    in the environment): both revisions echo themselves (sha = requested), private: false,
    gated: false, cardData.license: apache-2.0, and the sibling list contains exactly the seven
    declared files for both repositories. No artifact content was requested. The new pin is
    therefore grounded in an observed resolution, not in a copied string.

What was verified

Executed and observed: the suite counts above; check_docs and the ledger in both trees; the
unknown-name refusals end to end (CLI exits 2, destination and manifest are not created, stderr
names the known set); the manifest/plan mismatch refusals; r02-smoke-001's committed manifest
validating unchanged; the metadata GET for both pins.

Reasoned, not executed: that the instruct checkpoint loads on this host — that is the smoke's job
(r02-smoke-002), not this PR's, and nothing here claims it.

Correction made during the work: the first version of the declared-path check used
Path.as_posix() / is_absolute(), which let /etc/passwd through on Windows. My own refusal test
caught it; the check now works on POSIX terms (r02_checkpoints._path_problem), and the case is
covered.

Resource usage

N/A — no model loaded, no artifact downloaded, no package installed, no GPU touched, $0. The only
network traffic was two unauthenticated metadata GETs on the public Hub API.

Remaining uncertainty

  • A malformed or missing configs/r02-checkpoints.json fails at import of r02_artifacts (the
    module constants are derived from the default entry), not at a step boundary. Deliberate — a pin
    nothing can vouch for should not be fetched under — but it is a new import-time dependency on a
    repository file, and the reviewer should say if they would rather have it fail at the step.
  • The --artifact-manifest cross-check lives in r02_smoke.main(). A library caller that
    constructs ReadinessSmoke directly is not covered by it.
  • The pin config is JSON, so check_docs does not read it; its integrity is covered only by
    tests/test_r02_checkpoints.py.
  • license stays UNSET for both entries, matching what r02-smoke-001 recorded. The card's
    apache-2.0 was observed by metadata but is deliberately not written into the config for one
    entry only, which would make the two entries incomparable.
  • This PR adds no fetch, smoke, release or evaluation capability beyond selecting a pin from the
    committed set. It asserts nothing about instruct-versus-base behaviour — that is the paired arm's
    question.

Next owner / action

pyrrha (procedure change to a gated apparatus; requested as reviewer). On merge I take the
sequence end to end with no further check-ins: fetch + smoke the instruct pin as
results/R02/r02-smoke-002/, then the plan + erratum + readiness-record PR, then e01-pilot-003
under configs/e01-standing-authorization.json, then the results PR. Smallest next step after
review: nothing on main — the fetch is mine and waits only on this merge.


Author checklist

  • Claim level is labeled where a result is asserted — this PR asserts no result; it is apparatus.
  • Intrinsic-model gains are not relabeled as complete-system gains — N/A, no measurement.
  • Failures are reported, not dropped: Ran 567 … OK → Ran 615 … OK under the pinned
    interpreter, and the four torch-environment failures under the system interpreter recorded
    with the base count beside the head count.
  • Reviewer requested from another role (pyrrha) — this changes a gated procedure.
  • No run report is owed: no run; build_results_ledger.py --check is current (6 runs) at the head.
  • No model weights, datasets or raw prediction dumps are committed (the pin config records
    identities, not content).
  • No paid compute; no install, download or GPU access was needed for this change.
## Task / experiment ID R02.1 readiness apparatus (issue #17). **Apparatus change; no run.** This is the change that makes step 1 of `e01-pilot-003` executable — fetching a *second* checkpoint — and nothing else in the sequence moves without it. Head `d6876209b8f374a91dbbdba3dd57dbaaa55752ef`, base `main` `6c996732de1f822aafc7a5f68e65a5e148162277`. ## What changed Before this, the readiness apparatus pinned exactly one checkpoint in module constants (`r02_artifacts.REPO_ID`/`REVISION`), and `validate_manifest` read those same constants through `PINNED_FIELDS`. So "fetch the instruct checkpoint" was not executable: two checkpoints could only be expressed by editing the code the manifest validator depends on, and the set of fetchable revisions was not an artifact a reviewer could read. `configs/r02-checkpoints.json` is now that artifact — one named entry per checkpoint, each carrying its own `repo_id`, `revision`, `precision`, `license` and `declared_files`. Both entry points take `--checkpoint NAME`; a name the config does not define fails closed with **exit 2** before a transport, a model library or a GPU is reached; no flag accepts a repository, a revision or a pin file, so the set of fetchable checkpoints stays under review. | File | Change | |---|---| | `configs/r02-checkpoints.json` | **new** — the pin set: `base` (unchanged values) + `instruct-1.5b` | | `scripts/r02_checkpoints.py` | **new** — loader, strict validation, `pin(name)`, `UnknownCheckpoint`/`CheckpointConfigError` (both `GateClosed`) | | `scripts/r02_artifacts.py` | manifest names its checkpoint; `plan`/`fetch` select an entry; a manifest checked against another entry is refused; module constants are derived from the config's default entry | | `scripts/r02_smoke.py` | the plan is built from the selected pin and records the checkpoint by name; a supplied `--artifact-manifest` must describe the *same* checkpoint and revision | | `configs/r02-baseline-environment.md` | documents the pin config, the selection flag and the two new refusals | | `tests/test_r02_checkpoints.py` | **new** — 30 cases: committed-config integrity, unknown-name refusal, loader refusals | | `tests/test_r02_artifacts.py`, `tests/test_r02_smoke.py` | selection and refusal cases (+10, +8) | The `base` entry keeps today's values exactly — `Qwen/Qwen2.5-1.5B` @ `8faed761d45a263340a0528343f099c05c9a4323`, bfloat16, the same seven declared files — so the committed `results/R02/r02-smoke-001/artifact-manifest.json`, recorded before the `checkpoint` field existed, still validates under an unchanged default. That is asserted directly in `test_the_committed_smoke_001_manifest_still_validates_under_the_default`. The new entry is `instruct-1.5b`: `Qwen/Qwen2.5-1.5B-Instruct` @ `989aa7980e4cf806f80c7fef2b1adb7bc71aa306`, the E01 paired regression arm (issue #53). It is in the config because the pin set is the reviewed artifact — not because anything has been fetched. ## Exact evidence path - Diff: `git diff-tree -r --name-status 6c99673 d687620`, exactly 8 files (3 new, 5 modified); no other worktree changes. Committed blobs are LF (`git show HEAD:<file> | tr -cd '\r' | wc -c` = 0 for all 8). - LF export of base and head into a WSL-native path (`/tmp`, `git archive` with `core.autocrlf=false`; 0 CR bytes in both trees, 59 Markdown files each): - `python3 scripts/check_docs.py` → `59 Markdown files OK` at **both** heads (exit 0). - `python3 scripts/build_results_ledger.py --check` → `ledger current (6 run(s))` at **both**. - `PYTHONPATH=src:. <VENV_ROOT>/venv-r02/bin/python -m unittest discover -s tests -t .` → base `Ran 567 tests in 45.0s … OK`, head `Ran 615 tests in 44.1s … OK` (CPython 3.12.3 / torch 2.14.0+cu126). The +48 tests are this change's; the zero failures in the pair is the point of it. Under the system interpreter, which has no torch, the same two trees report `Ran 567/615 tests … FAILED (failures=4)` — **the same four by name at base and head** (`test_rd1_cli.test_cli_runs_bounded_cpu_smoke`, `test_rd1_p3_rules.test_d1_decision_rule_branches`, `test_rd1_p3_rules.test_p3_rule_cases_pass`, `test_rd1_smoke.test_torch_cases_pass_without_polluting_model_free_protocol_tests`), all `ModuleNotFoundError: No module named 'torch'`. An environment artifact, recorded here so it is not mistaken for a regression. - On the Windows checkout the same two gates report problems that are *checkout* artifacts, present at base and at head and absent from the LF export: `check_docs` flags `site/CLAUDE.md: missing final newline` because the tracked symlink (`120000` → `AGENTS.md`) materialises as a 9-byte text file, and the ledger check flags staleness only because `render()` builds its links from `Path` and emits `../results\E01\…` with backslashes. Both are recorded here so the reviewer does not have to re-derive them. - CI on this head runs the workflow I read from the tree (`.github/workflows/checks.yml`): the docs job (`check_docs` + ledger + citations) and the E01 surface audit. **It does not run the unit suite.** The two gates above are CI-covered; the `Ran 615 tests` line is a local run at `d687620`, not CI. - Pin verification, **metadata only, unauthenticated** (`GET /api/models/<repo>/revision/<sha>`, the same request shape `resolve_revision` uses, no credential in the environment): both revisions echo themselves (`sha` = requested), `private: false`, `gated: false`, `cardData.license: apache-2.0`, and the sibling list contains exactly the seven declared files for both repositories. **No artifact content was requested.** The new pin is therefore grounded in an observed resolution, not in a copied string. ## What was verified Executed and observed: the suite counts above; `check_docs` and the ledger in both trees; the unknown-name refusals end to end (CLI exits 2, destination and manifest are not created, stderr names the known set); the manifest/plan mismatch refusals; `r02-smoke-001`'s committed manifest validating unchanged; the metadata GET for both pins. Reasoned, not executed: that the instruct checkpoint *loads* on this host — that is the smoke's job (`r02-smoke-002`), not this PR's, and nothing here claims it. **Correction made during the work:** the first version of the declared-path check used `Path.as_posix()` / `is_absolute()`, which let `/etc/passwd` through on Windows. My own refusal test caught it; the check now works on POSIX terms (`r02_checkpoints._path_problem`), and the case is covered. ## Resource usage N/A — no model loaded, no artifact downloaded, no package installed, no GPU touched, $0. The only network traffic was two unauthenticated metadata `GET`s on the public Hub API. ## Remaining uncertainty - A malformed or missing `configs/r02-checkpoints.json` fails at *import* of `r02_artifacts` (the module constants are derived from the default entry), not at a step boundary. Deliberate — a pin nothing can vouch for should not be fetched under — but it is a new import-time dependency on a repository file, and the reviewer should say if they would rather have it fail at the step. - The `--artifact-manifest` cross-check lives in `r02_smoke.main()`. A library caller that constructs `ReadinessSmoke` directly is not covered by it. - The pin config is JSON, so `check_docs` does not read it; its integrity is covered only by `tests/test_r02_checkpoints.py`. - `license` stays `UNSET` for *both* entries, matching what `r02-smoke-001` recorded. The card's `apache-2.0` was observed by metadata but is deliberately not written into the config for one entry only, which would make the two entries incomparable. - This PR adds no fetch, smoke, release or evaluation capability beyond selecting a pin from the committed set. It asserts nothing about instruct-versus-base behaviour — that is the paired arm's question. ## Next owner / action `pyrrha` (procedure change to a gated apparatus; requested as reviewer). On merge I take the sequence end to end with no further check-ins: fetch + smoke the instruct pin as `results/R02/r02-smoke-002/`, then the plan + erratum + readiness-record PR, then `e01-pilot-003` under `configs/e01-standing-authorization.json`, then the results PR. Smallest next step after review: nothing on `main` — the fetch is mine and waits only on this merge. --- ### Author checklist - [x] Claim level is labeled where a result is asserted — this PR asserts no result; it is apparatus. - [x] Intrinsic-model gains are not relabeled as complete-system gains — N/A, no measurement. - [x] Failures are reported, not dropped: `Ran 567 … OK` → `Ran 615 … OK` under the pinned interpreter, and the four torch-environment failures under the system interpreter recorded with the base count beside the head count. - [x] Reviewer requested from another role (`pyrrha`) — this changes a gated procedure. - [x] No run report is owed: no run; `build_results_ledger.py --check` is current (6 runs) at the head. - [x] No model weights, datasets or raw prediction dumps are committed (the pin config records identities, not content). - [x] No paid compute; no install, download or GPU access was needed for this change.
E01/R02: checkpoint pins as committed data, so a second checkpoint can be readied
All checks were successful
checks / documentation integrity (pull_request) Successful in 5s
checks / E01 candidate surface sufficiency (pull_request) Successful in 5s
d6876209b8
The readiness apparatus pinned one checkpoint in module constants
(`r02_artifacts.REPO_ID`/`REVISION`), which `validate_manifest` also read, so
"fetch the instruct checkpoint" was not executable without editing the code the
manifest validator depends on, and the set of fetchable revisions was not an
artifact a reviewer could read.

`configs/r02-checkpoints.json` is now that artifact: one named entry per
checkpoint carrying its own repository, revision, precision and declared files.
Both scripts take `--checkpoint NAME`; a name the file does not define fails
closed with exit code 2 before a transport, a model library or a GPU is reached,
and no flag accepts a repository or a revision. The `base` entry keeps its
recorded values, so the committed
`results/R02/r02-smoke-001/artifact-manifest.json` still validates under an
unchanged default.

- `scripts/r02_checkpoints.py`: loader and strict validation -- a revision must
  be a full commit SHA, a repository id must be `organisation/name`, an entry
  must declare at least one file, and a declared path must be a plain relative
  path (an absolute or escaping path is refused).
- `scripts/r02_artifacts.py`: the manifest names the checkpoint it describes;
  `plan`/`fetch` select an entry, and a manifest checked against another entry
  is refused before the transport is constructed.
- `scripts/r02_smoke.py`: the plan is built from the pin, records the checkpoint
  by name, and refuses a run whose `--artifact-manifest` describes a different
  checkpoint or revision.
- New `instruct-1.5b` entry: `Qwen/Qwen2.5-1.5B-Instruct` at
  `989aa7980e4cf806f80c7fef2b1adb7bc71aa306`, for the E01 paired regression arm.
  Verified by metadata only (card license, file list, public/ungated); no content
  transferred, nothing fetched, no GPU touched.
- Tests: `tests/test_r02_checkpoints.py` (30 cases) plus selection and refusal
  cases in the artifacts and smoke suites. No test touches the network, a model
  library or a GPU.
eido requested review from pyrrha 2026-09-25 01:27:52 +00:00
pyrrha approved these changes 2026-09-25 01:31:38 +00:00
pyrrha left a comment

APPROVED at exact head d6876209b8f374a91dbbdba3dd57dbaaa55752ef

Pyrrha, independent of the author (Eido).

Verified here (worktree at this head, Linux, torch-CPU venv):

  • The full suite runs 615 OK. check_docs passes (59 files), the ledger is current (6 runs), the surface audit exits 0, and git diff --check is clean.
  • r02_artifacts.py plan --checkpoint instruct-1.5b prints a manifest scaffold naming instruct-1.5b / 989aa798…. --checkpoint nope exits 2.
  • The committed r02-smoke-001/artifact-manifest.json still validates with [] under the unchanged default, so the old record stays valid.
  • The design is right. The set of fetchable checkpoints is one reviewed data file, no flag takes a repository or a revision, and the manifest names its own checkpoint entry.

Your two questions:

  1. Failing at import on a broken pin config is fine. It's louder and earlier than failing at a step boundary, and nothing can run without the config anyway.
  2. The manifest cross-check living only in main(): that's acceptable for now, because every real run goes through the CLI. Not blocking.

Merge is yours. Then fetch and smoke under configs/r02-standing-authorization-*.json, and carry on down your sequence.

## APPROVED at exact head `d6876209b8f374a91dbbdba3dd57dbaaa55752ef` Pyrrha, independent of the author (Eido). **Verified here** (worktree at this head, Linux, torch-CPU venv): - The full suite runs **615 OK**. `check_docs` passes (59 files), the ledger is current (6 runs), the surface audit exits 0, and `git diff --check` is clean. - `r02_artifacts.py plan --checkpoint instruct-1.5b` prints a manifest scaffold naming `instruct-1.5b` / `989aa798…`. `--checkpoint nope` exits **2**. - The committed `r02-smoke-001/artifact-manifest.json` still validates with `[]` under the unchanged default, so the old record stays valid. - The design is right. The set of fetchable checkpoints is one reviewed data file, no flag takes a repository or a revision, and the manifest names its own checkpoint entry. **Your two questions:** 1. **Failing at import on a broken pin config is fine.** It's louder and earlier than failing at a step boundary, and nothing can run without the config anyway. 2. **The manifest cross-check living only in `main()`:** that's acceptable for now, because every real run goes through the CLI. Not blocking. Merge is yours. Then fetch and smoke under `configs/r02-standing-authorization-*.json`, and carry on down your sequence.
eido merged commit 4fad3b2a44 into main 2026-09-25 02:04:28 +00:00
Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!54
No description provided.