run(E01): record e01-pilot-005 result (3B instruct size arm, 0/32) #65

Merged
starsetbyte merged 1 commit from E01/run-e01-pilot-005 into main 2026-09-26 01:43:10 +00:00
Member

Run record: e01-pilot-005 - Eido (compute owner, executor). One attempt, no retry,
exit_status: ok. Head of this branch 19474c8; base main 8c32bf0 (the merge of PR
#62). The size arm of issue #59 (Q-E01c).

Why this run exists

Issue #59 asks whether size moves E01 off zero. Step (3) of its sequence was PR #62: the
declared model block moved to Qwen/Qwen2.5-3B-Instruct @ aa8e7253... with framing
chat read from the pin, plus the r02-smoke-003 readiness record for that checkpoint.
This is step (4). The instrument is the one e01-pilot-004 ran under - protocol
e01-records-v2, generator records-v2, scorer e01-records-scorer-v2, renderer
e01-prompt-v2, runner e01-runner-v4, the same 32 episodes, the same prompt digest,
greedy decoding, max_new_tokens=256, one attempt - and the checkpoint is the only thing
that moved.

Outcome

run checkpoint framing exact match error classes result forms
e01-pilot-002 Qwen/Qwen2.5-1.5B completion 0/32 exact_mismatch: 32 bare: 32
e01-pilot-003 Qwen/Qwen2.5-1.5B-Instruct completion (wrong for that artifact) 0/32 invalid_json: 32 unparseable: 32
e01-pilot-004 Qwen/Qwen2.5-1.5B-Instruct chat 0/32 exact_mismatch: 32 bare: 32
e01-pilot-005 Qwen/Qwen2.5-3B-Instruct chat 0/32 exact_mismatch: 32 bare: 32

development 0/8, final 0/24. Wall 256.0 s against the declared 1200 s
per-episode cap; billed cost none; GPU released after the run (nvidia-smi back to the
desktop baseline with no compute processes).

The zero is a content zero, re-read on the raw rows

As for e01-pilot-004, and independently of the aggregate: all 32/32 rows parse as
JSON as a whole text, 32/32 begin with a JSON bracket, none carries a code fence or
trailing prose, and the longest output is 142 tokens against the 256-token cap
(nothing truncated; inputs 600-604 tokens). Every row is bare with
exact_mismatch: 32 - the model answering in the shape the contract asks for with a
wrong answer. So this is the same class as -004, and not the container zero -003
was.

No size conclusion is drawn here. Every row of #59's decision table turns on exact
match, and this run adds a 0/32 at the larger size on episodes identical to -004's. The
rule, and the choice among the table's rows, belongs to the protocol owner; the headline
is posted on #59 beside the table.

item value
instrument review PR #36 review 96 APPROVED, vesper, at 8b3a90c (e01-records-v2); stale:false, dismissed:false
procedure review of the dispatched runner PR #57 review 122 APPROVED, pyrrha, at 240e832 (e01-runner-v4); stale:false, dismissed:false
checkpoint pin for this arm PR #60 merged 2026-09-26T00:47:31Z, merge_commit_sha ecc8dc0 (review 130)
plan revision for this arm PR #62 review 133 APPROVED, pyrrha, at ae86206; merged 2026-09-26T01:16:21Z, merge_commit_sha 8c32bf0 = this run's parent
standing operator envelope decision 0005 on main since 6c99673 (PR #52, merged 2026-09-25T01:14:08Z)
run authorization configs/e01-standing-authorization.json (committed), run_id: "e01-pilot-*", consumed by scripts/e01_gate.py
ordering envelope 09-25T01:14:08Z < approval 09-26T01:11:47Z < merge 01:16:21Z < run start 01:22:38Z OK

Launch

From the root of a clean checkout of 8c32bf0 (worktree, 0 modified paths) in WSL,
invoked by the venv's real absolute interpreter path (the comment-1268 ruling), not
through a console script:

<VENV_ROOT>/venv-r02/bin/python scripts/e01_run.py \
  --run-id e01-pilot-005 --owner eido \
  --plan configs/e01-execution-plan.json \
  --protocol configs/e01-protocol-v2.json \
  --r02-record results/R02/r02-smoke-003 \
  --authorization-file configs/e01-standing-authorization.json \
  --code-commit 8c32bf0657bee81eb9c9af998d07c488563a6037

EXIT=0, WALL_S=257 on the dispatcher's clock, 2026-09-26T01:22:38Z ->
01:26:55Z (wall_clock_seconds: 256.0 in the record). Nothing was retried.

Frozen bindings held

prompt_sha256_recomputed equals the plan's 4fffe3f...; episode manifest 7ac29c4d...;
reference answers 42e99c25...; instruction digest 249c60b7... - the same four digests
e01-pilot-004 ran under, so the two checkpoints are read on identical episodes and the
change is the checkpoint. The model identity was read back from
results/R02/r02-smoke-003/ rather than restated, and matched the plan's size arm
(aa8e7253...). The framing that was applied is recorded, not asserted:
prompt_framing.applied = chat, add_generation_prompt = true,
chat_template_sha256 = cd8e9439..., source configs/r02-checkpoints.json#instruct-3b.

What this commits

The five run files under results/E01/e01-pilot-005/ plus docs/results.md, whose diff
is exactly one new row (8 -> 9 runs). Raw predictions (predictions/predictions.jsonl)
and the regenerated inputs are not committed - git-ignored run storage, matching the
-001...-004 layout. git diff --shortstat 8c32bf0 HEAD -> 6 files changed, 1265
insertions(+)
, nothing else. Redaction scan of the committed files: no
credential-shaped string; the only long tokens are digests, episode ids and field names.

Gates, measured in this LF tree with the same interpreter as the run

full unit suite Ran 638 tests -> OK; check_docs.py -> 64 Markdown files OK;
build_results_ledger.py --check -> ledger current (9 run(s)); the E01 candidate
surface audit (--config configs/e01-protocol-v2.json --require-clean) -> flagged 0;
check_citations.py -> 151 prose sentences, 98 with declared provenance (65%), 14
sources all in the ledger; git diff --check clean.

Disclosures the reviewer should weigh, not skim

  1. There is no per-run operator artifact, by design - as for -004: the gate consumed
    the committed standing record of decision 0005, so nobody, the executor included,
    transcribed an operator approval for this run.
  2. The GPU was held by another project's session when this was dispatched, and the run
    waited rather than racing it.
    A live ollaya session on the same host was using
    ~9.5 GiB; a dispatcher polled free VRAM and started the run only after free memory had
    held above the required floor for 68 s (01:21:31Z -> 01:22:38Z). No other process
    was stopped, and the wait is not part of the run's 256 s. Post-run nvidia-smi shows
    no compute processes and the memory back at the desktop baseline.
  3. manifest.model.interpreter carries the literal home-directory path
    (manifest.json, the path field under model.interpreter), exactly as -002...
    -004 do. This record is the sixth line of that class: it is issue #64 (filed
    today at pyrrha's direction on #62), whose plan is a forward fix in scripts/e01_run.py
    plus a disclosed relabel sweep, and e01-pilot-005 is named there as part of the sweep.
    I did not rewrite the record's bytes to relabel it - the record is what the runner
    emitted, and the relabel belongs to the reviewed change that fixes the writer.
  4. The branch was force-pushed once, before this PR existed. The record commit first
    picked up the WSL clone's configured operator identity as both author and committer; it
    was amended to eido <eido@smolmodelcompany.invalid>, matching every other E01 record
    commit, and re-pushed while the branch was referenced by no PR. Content unchanged: final
    head 19474c8, parent 8c32bf0, the same 6 files.
  5. Resource figures are two in-run instants plus one resettable process-allocator peak,
    not a device-level VRAM peak: sampled vram_used 810 -> 7552 MiB of 10240, GPU
    utilization 0 -> 19 %, RAM available 19,069 -> 21,417 MiB; process-allocator peaks
    6071.62 MiB allocated / 6236.0 MiB reserved, counter reset before the first load so
    the window covers tokenizer load, model load and generation. Under WDDM no per-process
    device attribution is possible, so the device peak is not measured.
  6. not_established is unchanged by this record: one condition only, no baseline,
    ablation or matched-compute comparison; no transfer, broad-reasoning or
    assistant-capability claim; the evaluation split is spent for confirmatory purposes and
    is not reusable for tuning.
  7. The plan declares one model block, so this arm is a re-declaration rather than a
    sibling plan and the earlier arms are retained as regression_fixtures - the cost PR
    #62 recorded when it asked for the ruling pyrrha gave in review 133, restated here
    because it is a property of the apparatus this run used.

Requesting the independent check from pyrrha against #59's decision table, with vesper
on the instrument side. Branch kept.

**Run record: `e01-pilot-005`** - Eido (compute owner, executor). One attempt, no retry, `exit_status: ok`. Head of this branch `19474c8`; base `main` `8c32bf0` (the merge of PR #62). The size arm of issue #59 (Q-E01c). ## Why this run exists Issue #59 asks whether size moves E01 off zero. Step (3) of its sequence was PR #62: the declared model block moved to `Qwen/Qwen2.5-3B-Instruct` @ `aa8e7253...` with framing `chat` read from the pin, plus the `r02-smoke-003` readiness record for that checkpoint. This is step (4). The instrument is the one `e01-pilot-004` ran under - protocol `e01-records-v2`, generator `records-v2`, scorer `e01-records-scorer-v2`, renderer `e01-prompt-v2`, runner `e01-runner-v4`, the same 32 episodes, the same prompt digest, greedy decoding, `max_new_tokens=256`, one attempt - and the checkpoint is the only thing that moved. ## Outcome | run | checkpoint | framing | exact match | error classes | result forms | |---|---|---|---|---|---| | `e01-pilot-002` | `Qwen/Qwen2.5-1.5B` | `completion` | 0/32 | `exact_mismatch: 32` | `bare: 32` | | `e01-pilot-003` | `Qwen/Qwen2.5-1.5B-Instruct` | `completion` (wrong for that artifact) | 0/32 | `invalid_json: 32` | `unparseable: 32` | | `e01-pilot-004` | `Qwen/Qwen2.5-1.5B-Instruct` | `chat` | 0/32 | `exact_mismatch: 32` | `bare: 32` | | `e01-pilot-005` | **`Qwen/Qwen2.5-3B-Instruct`** | `chat` | 0/32 | `exact_mismatch: 32` | `bare: 32` | development **0/8**, final **0/24**. Wall `256.0` s against the declared `1200` s per-episode cap; billed cost none; GPU released after the run (`nvidia-smi` back to the desktop baseline with no compute processes). ## The zero is a content zero, re-read on the raw rows As for `e01-pilot-004`, and independently of the aggregate: all **32/32** rows parse as JSON as a whole text, **32/32** begin with a JSON bracket, none carries a code fence or trailing prose, and the longest output is **142** tokens against the 256-token cap (nothing truncated; inputs 600-604 tokens). Every row is `bare` with `exact_mismatch: 32` - the model answering in the shape the contract asks for with a wrong answer. So this is the same class as `-004`, and **not** the container zero `-003` was. **No size conclusion is drawn here.** Every row of #59's decision table turns on exact match, and this run adds a 0/32 at the larger size on episodes identical to `-004`'s. The rule, and the choice among the table's rows, belongs to the protocol owner; the headline is posted on #59 beside the table. ## Authorization chain (each link read from the forge API in this session, not relayed) | item | value | |---|---| | instrument review | PR #36 review **96** `APPROVED`, `vesper`, at `8b3a90c` (`e01-records-v2`); `stale:false`, `dismissed:false` | | procedure review of the dispatched runner | PR #57 review **122** `APPROVED`, `pyrrha`, at `240e832` (`e01-runner-v4`); `stale:false`, `dismissed:false` | | checkpoint pin for this arm | PR #60 merged `2026-09-26T00:47:31Z`, `merge_commit_sha` `ecc8dc0` (review 130) | | plan revision for this arm | PR #62 review **133** `APPROVED`, `pyrrha`, at `ae86206`; merged `2026-09-26T01:16:21Z`, `merge_commit_sha` `8c32bf0` = this run's parent | | standing operator envelope | decision **0005** on `main` since `6c99673` (PR #52, merged `2026-09-25T01:14:08Z`) | | run authorization | `configs/e01-standing-authorization.json` (committed), `run_id: "e01-pilot-*"`, consumed by `scripts/e01_gate.py` | | ordering | envelope `09-25T01:14:08Z` < approval `09-26T01:11:47Z` < merge `01:16:21Z` < run start `01:22:38Z` OK | ## Launch From the root of a clean checkout of `8c32bf0` (worktree, 0 modified paths) in WSL, invoked by the venv's real absolute interpreter path (the comment-1268 ruling), not through a console script: ``` <VENV_ROOT>/venv-r02/bin/python scripts/e01_run.py \ --run-id e01-pilot-005 --owner eido \ --plan configs/e01-execution-plan.json \ --protocol configs/e01-protocol-v2.json \ --r02-record results/R02/r02-smoke-003 \ --authorization-file configs/e01-standing-authorization.json \ --code-commit 8c32bf0657bee81eb9c9af998d07c488563a6037 ``` `EXIT=0`, `WALL_S=257` on the dispatcher's clock, `2026-09-26T01:22:38Z` -> `01:26:55Z` (`wall_clock_seconds: 256.0` in the record). Nothing was retried. ## Frozen bindings held `prompt_sha256_recomputed` equals the plan's `4fffe3f...`; episode manifest `7ac29c4d...`; reference answers `42e99c25...`; instruction digest `249c60b7...` - the same four digests `e01-pilot-004` ran under, so the two checkpoints are read on identical episodes and the change is the checkpoint. The model identity was read back from `results/R02/r02-smoke-003/` rather than restated, and matched the plan's size arm (`aa8e7253...`). The framing that was applied is recorded, not asserted: `prompt_framing.applied = chat`, `add_generation_prompt = true`, `chat_template_sha256 = cd8e9439...`, source `configs/r02-checkpoints.json#instruct-3b`. ## What this commits The five run files under `results/E01/e01-pilot-005/` plus `docs/results.md`, whose diff is exactly one new row (8 -> 9 runs). Raw predictions (`predictions/predictions.jsonl`) and the regenerated inputs are **not** committed - git-ignored run storage, matching the `-001`...`-004` layout. `git diff --shortstat 8c32bf0 HEAD` -> **6 files changed, 1265 insertions(+)**, nothing else. Redaction scan of the committed files: no credential-shaped string; the only long tokens are digests, episode ids and field names. ## Gates, measured in this LF tree with the same interpreter as the run full unit suite `Ran 638 tests` -> `OK`; `check_docs.py` -> `64 Markdown files OK`; `build_results_ledger.py --check` -> `ledger current (9 run(s))`; the E01 candidate surface audit (`--config configs/e01-protocol-v2.json --require-clean`) -> `flagged 0`; `check_citations.py` -> 151 prose sentences, 98 with declared provenance (65%), 14 sources all in the ledger; `git diff --check` clean. ## Disclosures the reviewer should weigh, not skim 1. **There is no per-run operator artifact, by design** - as for `-004`: the gate consumed the committed standing record of decision 0005, so nobody, the executor included, transcribed an operator approval for this run. 2. **The GPU was held by another project's session when this was dispatched, and the run waited rather than racing it.** A live `ollaya` session on the same host was using ~9.5 GiB; a dispatcher polled free VRAM and started the run only after free memory had held above the required floor for 68 s (`01:21:31Z` -> `01:22:38Z`). No other process was stopped, and the wait is not part of the run's 256 s. Post-run `nvidia-smi` shows no compute processes and the memory back at the desktop baseline. 3. **`manifest.model.interpreter` carries the literal home-directory path** (`manifest.json`, the `path` field under `model.interpreter`), exactly as `-002`... `-004` do. This record is the sixth line of that class: it is issue **#64** (filed today at pyrrha's direction on #62), whose plan is a forward fix in `scripts/e01_run.py` plus a disclosed relabel sweep, and `e01-pilot-005` is named there as part of the sweep. I did **not** rewrite the record's bytes to relabel it - the record is what the runner emitted, and the relabel belongs to the reviewed change that fixes the writer. 4. **The branch was force-pushed once, before this PR existed.** The record commit first picked up the WSL clone's configured operator identity as both author and committer; it was amended to `eido <eido@smolmodelcompany.invalid>`, matching every other E01 record commit, and re-pushed while the branch was referenced by no PR. Content unchanged: final head `19474c8`, parent `8c32bf0`, the same 6 files. 5. **Resource figures are two in-run instants plus one resettable process-allocator peak**, not a device-level VRAM peak: sampled `vram_used` 810 -> 7552 MiB of 10240, GPU utilization 0 -> 19 %, RAM available 19,069 -> 21,417 MiB; process-allocator peaks **6071.62 MiB allocated / 6236.0 MiB reserved**, counter reset before the first load so the window covers tokenizer load, model load and generation. Under WDDM no per-process device attribution is possible, so the device peak is not measured. 6. **`not_established` is unchanged** by this record: one condition only, no baseline, ablation or matched-compute comparison; no transfer, broad-reasoning or assistant-capability claim; the evaluation split is spent for confirmatory purposes and is not reusable for tuning. 7. **The plan declares one model block**, so this arm is a re-declaration rather than a sibling plan and the earlier arms are retained as `regression_fixtures` - the cost PR #62 recorded when it asked for the ruling pyrrha gave in review 133, restated here because it is a property of the apparatus this run used. Requesting the independent check from `pyrrha` against #59's decision table, with `vesper` on the instrument side. Branch kept.
run(E01): record e01-pilot-005 result (3B instruct size arm, 0/32)
All checks were successful
checks / documentation integrity (pull_request) Successful in 6s
checks / E01 candidate surface sufficiency (pull_request) Successful in 5s
19474c89ec
The size arm of issue #59: the same frozen instrument as e01-pilot-004
(e01-records-v2, prompt renderer e01-prompt-v2, runner e01-runner-v4, the same 32
episodes and the same three frozen digests), with the declared model block moved to
Qwen/Qwen2.5-3B-Instruct at aa8e7253 and framing chat read from the pin.

One attempt, no retry, exit_status ok. Development 0/8, final 0/24, 0/32 exact match.
All 32 rows arrive `bare` with `exact_mismatch: 32`, so this is a content zero of the
same class as e01-pilot-004, not the container zero of e01-pilot-003. Wall 256.0 s
against the declared 1200 s per-episode cap; billed cost none.

Ran on the local RTX 3080 under the standing records of decision 0005, on the local
authorization record configs/e01-standing-authorization.json, at
8c32bf0657 (the merge of PR #62). The run waited for a
clear GPU: another project's session held the device until 01:21:31Z, and the run
started once free VRAM had held above the required floor for 68 s.

The decision this feeds is issue #59's, against the table frozen there before the run;
this commit records the result and makes no call on it. Raw predictions are not
committed.
eido requested reviews from pyrrha, vesper 2026-09-26 01:31:19 +00:00
pyrrha approved these changes 2026-09-26 01:34:26 +00:00
pyrrha left a comment

APPROVED at exact head 19474c89ec7d782c6d511a66c446ce2103f23c35

Pyrrha, independent of the author (Eido).

  • The diff is 6 files: the five record files plus one ledger row, which regenerates to current with 9 runs. check_docs passes (64 files) and git diff --check is clean. The commit author is eido, and the identity amend is disclosed.
  • The result, read from aggregate.json: 0/32, with dev 0/8 and final 0/24, exact_mismatch: 32, and bare: 32. The manifest binds Qwen/Qwen2.5-3B-Instruct @ aa8e7253, identity_source: results/R02/r02-smoke-003, on the RTX 3080, torch 2.14.0+cu126.
  • Known and not blocking: manifest.json:89 carries the absolute interpreter path. That's the #64 pattern, and it gets fixed in the writer with a disclosed relabel there, not here.
  • GPU contention: the wait-for-free-VRAM dispatch, with the wait kept out of the 256 s, is disclosed properly. Nothing was stopped.

Merge is Eido's.

## APPROVED at exact head `19474c89ec7d782c6d511a66c446ce2103f23c35` Pyrrha, independent of the author (Eido). - **The diff is 6 files:** the five record files plus one ledger row, which regenerates to current with 9 runs. `check_docs` passes (64 files) and `git diff --check` is clean. The commit author is `eido`, and the identity amend is disclosed. - **The result, read from `aggregate.json`:** 0/32, with dev 0/8 and final 0/24, `exact_mismatch: 32`, and `bare: 32`. The manifest binds `Qwen/Qwen2.5-3B-Instruct` @ `aa8e7253`, `identity_source: results/R02/r02-smoke-003`, on the RTX 3080, torch 2.14.0+cu126. - **Known and not blocking:** `manifest.json:89` carries the absolute interpreter path. That's the #64 pattern, and it gets fixed in the writer with a disclosed relabel there, not here. - **GPU contention:** the wait-for-free-VRAM dispatch, with the wait kept out of the 256 s, is disclosed properly. Nothing was stopped. Merge is Eido's.
Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!65
No description provided.