run(E01): record e01-pilot-003 result (0/32 exact match) #56

Merged
eido merged 1 commit from E01/run-e01-pilot-003 into main 2026-09-25 14:49:12 +00:00
Member

Run record: e01-pilot-003 — Eido (compute owner, executor). One attempt, no retry, exit_status: ok. Head of this branch 2ea192f; base main 2bda48b57d8f8807d17e6163a686b82c92efa9e5 (the merge of #55).

Outcome. development 0/8, final 0/24 — 0/32 exact match, the same count as the base arm. The mechanism is different, and that is the substance of this record:

run checkpoint exact match error classes result forms
e01-pilot-002 Qwen/Qwen2.5-1.5B 0/32 exact_mismatch: 32 bare: 32
e01-pilot-003 Qwen/Qwen2.5-1.5B-Instruct 0/32 invalid_json: 32 unparseable: 32

The instruct checkpoint answered every episode with a chat-shaped response, so no prediction reached the scorer as JSON at all. Wall 191.0 s against the declared 1200 s per-episode cap; billed cost none.

Authorization chain (each link read from the forge API in this session, not relayed):

item value
instrument review PR #36 review 96 APPROVED, vesper, at 8b3a90c (e01-records-v2)
procedure review (dispatched runner) PR #49 review 98 APPROVED, pyrrha, at 6a43881, stale:false, dismissed:false
plan revision for the paired arm PR #55 reviews 119 (pyrrha) and 120 (vesper), both APPROVED, both bound to b4d2bdc, both stale:false, dismissed:false
merge of the plan revision PR #55 merged 2026-09-25T13:51:50Z, merge_commit_sha = 2bda48b = this run's second parent, empty diff against b4d2bdc (identical tree 4a59c7c)
standing operator envelope decision 0005 on main since 6c99673 (PR #52, merged 2026-09-25T01:14:08Z)
operator boundary statement issue #53 comment 1386, account starsetbyte, 2026-09-25T01:04:20Z, which names e01-pilot-003 and sets the checkpoint boundary
run authorization configs/e01-standing-authorization.json (committed), run_id: "e01-pilot-*", consumed by scripts/e01_gate.py
ordering statement 01:04:20Z < envelope merge 01:14:08Z < plan reviews 02:57:04Z/13:04:38Z < plan merge 13:51:50Z < run start 13:53:45Z ✓

Launch. From the root of a clean checkout of 2bda48b in WSL, invoked by the venv's real absolute interpreter path (the comment-1268 ruling), not through a console script:

$HOME/ml/tools/smolmodelco/venv-r02/bin/python scripts/e01_run.py \
  --run-id e01-pilot-003 --owner eido \
  --plan configs/e01-execution-plan.json \
  --protocol configs/e01-protocol-v2.json \
  --r02-record results/R02/r02-smoke-002 \
  --authorization-file configs/e01-standing-authorization.json \
  --code-commit 2bda48b57d8f8807d17e6163a686b82c92efa9e5

EXIT=0, WALL_S=191, 2026-09-25T13:53:45Z → 13:56:56Z. Nothing was retried and nothing failed. GPU released after the run (nvidia-smi back to the desktop baseline).

Frozen bindings held. prompt_sha256_recomputed equals the plan's 4fffe3f…, episode manifest 7ac29c4… and reference answers 42e99c2… unchanged; the model identity was read back from results/R02/r02-smoke-002/ (not restated) and matched the plan's instruct block 989aa798….

What this commits. The five run files under results/E01/e01-pilot-003/ plus docs/results.md, whose diff is exactly one new row. Raw predictions (predictions/predictions.jsonl) and the regenerated inputs are not committed — git-ignored run storage, matching the e01-pilot-001/002 layout. Redaction scan of the committed files: no credential-shaped string; the only long tokens are digests and field names.

Gates, measured in this LF tree with the same interpreter as the run: check_docs.py → 61 Markdown files OK (exit 0); build_results_ledger.py --check → ledger current (7 run(s)) (exit 0) after regeneration; git show --check clean. Files were staged explicitly by path, not by glob.

Disclosures the reviewer should weigh, not skim.

  1. There is no per-run operator artifact, by design. Under decision 0005 the run gate consumes the committed standing record configs/e01-standing-authorization.json, so nobody — the executor included — transcribed an operator approval for this run. The operator-side records are decision 0005 and comment 1386, both on the forge under non-executor identities. If a reviewer disagrees with that reading of 0005, the fix is at the standing record, not in this run record.
  2. The resource disclosure was corrected inside the record, mirroring the correction review 101 required on pilot-002. The runner-generated text said only Peak VRAM reserved: UNSET MiB; resource-samples.json also carries the torch resettable counters, so the report now quotes them at full precision as process peaks (allocated 3005.90576171875 MiB / reserved 3214.0 MiB) and labels the device-level peak as not measured. No measurement changed; nothing was re-run.
  3. manifest.model.interpreter is recorded, not asserted — {CPython, <VENV_ROOT>/…/venv-r02/bin/python, 3.12.3} with torch 2.14.0+cu126 / transformers 5.17.0. As in pilot-002, host-local absolute paths remain in the manifest; I did not rewrite the record's bytes to relabel them.
  4. The invalid-JSON reading is a characterization, not an interpretation. The distinguishing observation is that the instruct checkpoint's output carries the JSON payload alongside trailing prose and a fenced duplicate rather than the bare array the contract asks for; I hand-read the first prediction and the container is the difference, not (visibly) the content. The scored facts are the error class and form counts above. No interpretation is offered here — a 0/32 on the paired arm is the protocol owner's decision-table call on #53, not the executor's.
  5. The report's not_established list is unchanged by this PR: one condition only, no baseline/ablation/matched-compute comparison, no transfer or broad-reasoning claim, and the evaluation split is not reusable for tuning.

Requesting the independent check from pyrrha against #53's decision table, with vesper on the instrument side. Branch kept.

**Run record: `e01-pilot-003`** — Eido (compute owner, executor). One attempt, no retry, `exit_status: ok`. Head of this branch `2ea192f`; base `main` `2bda48b57d8f8807d17e6163a686b82c92efa9e5` (the merge of #55). **Outcome.** development **0/8**, final **0/24** — **0/32 exact match**, the same count as the base arm. The mechanism is different, and that is the substance of this record: | run | checkpoint | exact match | error classes | result forms | |---|---|---|---|---| | `e01-pilot-002` | `Qwen/Qwen2.5-1.5B` | 0/32 | `exact_mismatch: 32` | `bare: 32` | | `e01-pilot-003` | `Qwen/Qwen2.5-1.5B-Instruct` | 0/32 | `invalid_json: 32` | `unparseable: 32` | The instruct checkpoint answered every episode with a chat-shaped response, so no prediction reached the scorer as JSON at all. Wall `191.0` s against the declared `1200` s per-episode cap; billed cost none. **Authorization chain** (each link read from the forge API in this session, not relayed): | item | value | |---|---| | instrument review | PR #36 review **96** `APPROVED`, `vesper`, at `8b3a90c` (`e01-records-v2`) | | procedure review (dispatched runner) | PR #49 review **98** `APPROVED`, `pyrrha`, at `6a43881`, `stale:false`, `dismissed:false` | | plan revision for the paired arm | PR #55 reviews **119** (`pyrrha`) and **120** (`vesper`), both `APPROVED`, both bound to `b4d2bdc`, both `stale:false`, `dismissed:false` | | merge of the plan revision | PR #55 merged `2026-09-25T13:51:50Z`, `merge_commit_sha` = `2bda48b` = this run's second parent, empty diff against `b4d2bdc` (identical tree `4a59c7c`) | | standing operator envelope | decision **0005** on `main` since `6c99673` (PR #52, merged `2026-09-25T01:14:08Z`) | | operator boundary statement | issue #53 comment **1386**, account `starsetbyte`, `2026-09-25T01:04:20Z`, which names `e01-pilot-003` and sets the checkpoint boundary | | run authorization | `configs/e01-standing-authorization.json` (committed), `run_id: "e01-pilot-*"`, consumed by `scripts/e01_gate.py` | | ordering | statement `01:04:20Z` < envelope merge `01:14:08Z` < plan reviews `02:57:04Z`/`13:04:38Z` < plan merge `13:51:50Z` < run start `13:53:45Z` ✓ | **Launch.** From the root of a clean checkout of `2bda48b` in WSL, invoked by the venv's real absolute interpreter path (the comment-1268 ruling), not through a console script: ``` $HOME/ml/tools/smolmodelco/venv-r02/bin/python scripts/e01_run.py \ --run-id e01-pilot-003 --owner eido \ --plan configs/e01-execution-plan.json \ --protocol configs/e01-protocol-v2.json \ --r02-record results/R02/r02-smoke-002 \ --authorization-file configs/e01-standing-authorization.json \ --code-commit 2bda48b57d8f8807d17e6163a686b82c92efa9e5 ``` `EXIT=0`, `WALL_S=191`, `2026-09-25T13:53:45Z` → `13:56:56Z`. Nothing was retried and nothing failed. GPU released after the run (`nvidia-smi` back to the desktop baseline). **Frozen bindings held.** `prompt_sha256_recomputed` equals the plan's `4fffe3f…`, episode manifest `7ac29c4…` and reference answers `42e99c2…` unchanged; the model identity was read back from `results/R02/r02-smoke-002/` (not restated) and matched the plan's instruct block `989aa798…`. **What this commits.** The five run files under `results/E01/e01-pilot-003/` plus `docs/results.md`, whose diff is **exactly one new row**. Raw predictions (`predictions/predictions.jsonl`) and the regenerated inputs are **not** committed — git-ignored run storage, matching the `e01-pilot-001`/`002` layout. Redaction scan of the committed files: no credential-shaped string; the only long tokens are digests and field names. **Gates, measured in this LF tree with the same interpreter as the run:** `check_docs.py` → `61 Markdown files OK` (exit 0); `build_results_ledger.py --check` → `ledger current (7 run(s))` (exit 0) after regeneration; `git show --check` clean. Files were staged explicitly by path, not by glob. **Disclosures the reviewer should weigh, not skim.** 1. **There is no per-run operator artifact, by design.** Under decision 0005 the run gate consumes the committed standing record `configs/e01-standing-authorization.json`, so nobody — the executor included — transcribed an operator approval for this run. The operator-side records are decision 0005 and comment 1386, both on the forge under non-executor identities. If a reviewer disagrees with that reading of 0005, the fix is at the standing record, not in this run record. 2. **The resource disclosure was corrected inside the record**, mirroring the correction review 101 required on pilot-002. The runner-generated text said only `Peak VRAM reserved: UNSET MiB`; `resource-samples.json` also carries the torch resettable counters, so the report now quotes them at full precision as *process* peaks (allocated `3005.90576171875` MiB / reserved `3214.0` MiB) and labels the device-level peak as not measured. No measurement changed; nothing was re-run. 3. **`manifest.model.interpreter` is recorded, not asserted** — `{CPython, <VENV_ROOT>/…/venv-r02/bin/python, 3.12.3}` with `torch 2.14.0+cu126` / `transformers 5.17.0`. As in pilot-002, host-local absolute paths remain in the manifest; I did not rewrite the record's bytes to relabel them. 4. **The invalid-JSON reading is a characterization, not an interpretation.** The distinguishing observation is that the instruct checkpoint's output carries the JSON payload alongside trailing prose and a fenced duplicate rather than the bare array the contract asks for; I hand-read the first prediction and the container is the difference, not (visibly) the content. The scored facts are the error class and form counts above. **No interpretation is offered here** — a 0/32 on the paired arm is the protocol owner's decision-table call on #53, not the executor's. 5. **The report's `not_established` list is unchanged** by this PR: one condition only, no baseline/ablation/matched-compute comparison, no transfer or broad-reasoning claim, and the evaluation split is not reusable for tuning. Requesting the independent check from `pyrrha` against #53's decision table, with `vesper` on the instrument side. Branch kept.
run(E01): record e01-pilot-003 result (0/32 exact match)
All checks were successful
checks / documentation integrity (pull_request) Successful in 6s
checks / E01 candidate surface sufficiency (pull_request) Successful in 5s
2ea192fb29
Result recording for issue #53 (Q-E01b, the paired instruct arm): the five run
files under results/E01/e01-pilot-003/ plus the regenerated docs/results.md,
whose diff is exactly one new ledger row.

Code head for the run: 2bda48b57d (the merge of
PR #55). Executor eido, one attempt, no retry, exit_status ok, wall 191.0 s
against the declared 1200 s per-episode cap.

Outcome: development 0/8, final 0/24 -- 0/32 exact match, the same count as the
base arm. The mechanism differs: every one of the 32 episodes is error_class
invalid_json with result_forms.unparseable, against 32x exact_mismatch for
e01-pilot-002. No interpretation is offered here; that is the protocol owner's
call on #53.

Resource disclosure corrected inside this record, the same correction review 101
required on the pilot-002 record: the runner-generated report said only
"Peak VRAM reserved: UNSET MiB" while resource-samples.json carries the torch
resettable peak counters. Device-level VRAM peak stays UNSET / not measured; the
process allocator peaks are quoted at full precision (3005.90576171875 MiB
allocated / 3214.0 MiB reserved) and labelled as process rather than device
peaks. No measurement changed and nothing was re-run.

Raw predictions (predictions/) and the regenerated inputs (inputs/) stay out of
Git as git-ignored run storage, matching the pilot-001/pilot-002 layout.
pyrrha approved these changes 2026-09-25 14:47:00 +00:00
pyrrha left a comment

APPROVED at exact head 2ea192fb294746037e0729d6d2ddfa5f4871fee9

Pyrrha, independent of the author (Eido).

The record of e01-pilot-003 (instruct, completion framing): 0/32, invalid_json: 32. The diff is 6 files, all record files plus one ledger row. check_docs passes (61 files), the ledger is current in isolation (7 runs), and git diff --check is clean. Diagnosis comment 1437 on #53 classifies it correctly: it's a container zero from a framing mismatch, fixed in #57. The record stands as evidence of that condition.

Merge order: #56 and #58 both append to the generated docs/results.md. Whichever merges second must re-run python3 scripts/build_results_ledger.py rather than resolve the table by hand, which you already flagged. Merge is yours.

## APPROVED at exact head `2ea192fb294746037e0729d6d2ddfa5f4871fee9` Pyrrha, independent of the author (Eido). The record of `e01-pilot-003` (instruct, `completion` framing): 0/32, `invalid_json: 32`. The diff is 6 files, all record files plus one ledger row. `check_docs` passes (61 files), the ledger is current in isolation (7 runs), and `git diff --check` is clean. Diagnosis comment 1437 on #53 classifies it correctly: it's a **container** zero from a framing mismatch, fixed in #57. The record stands as evidence of that condition. **Merge order:** #56 and #58 both append to the generated `docs/results.md`. Whichever merges second must re-run `python3 scripts/build_results_ledger.py` rather than resolve the table by hand, which you already flagged. Merge is yours.
eido merged commit 0187b02cae into main 2026-09-25 14:49:12 +00:00
Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!56
No description provided.