fix(E01): read the peak-VRAM probe after the generation step (review #61 follow-up 1, item 2) #33

Merged
starsetbyte merged 1 commit from E01/runner-peak-probe-timing into main 2026-09-21 15:29:44 +00:00
Member

Apparatus-only follow-up to review #61 on #28 (follow-up 1, item 2 — "file the runner fix as a separate apparatus PR"). Base fa858b1 (main). Head b44ef1b3478bb94e084e5ab178f66445a54e098d.

Nothing was run. No model, no GPU, no artifact retrieval, no run directory, no dispatch, no download, no reservation, no ledger entry. This PR does not touch results/E01/e01-pilot-001/: #28's record keeps its own UNSET and its own number.

The defect, in the runner's terms

scripts/e01_run.py sampled exactly once, at stage resource-sample-pre, before any load, and registered the torch peak probe after that instant — grep -n sample_once scripts/e01_run.py returned one call. So the probe was never read: a record from that path lists it in method.probes (with the basis string "read after the generation step") while peak_claims carries no vram_peak_* entry and probe_errors is empty. The record reads as a sampling success with a missing number.

The change

One added sampling instant after the generation loop, stage resource-sample-post, mirroring scripts/r02_smoke.py (sample → register probe → generate → sample again), which is why results/R02/r02-smoke-001 carries 3003.083 MiB allocated / 3178.0 MiB reserved on this host.

I did not take the other shape the review offered ("register the peak probe before the single sampling instant"). Registering it earlier only moves the registration: the counter would then be read at the pre-load instant, before the window it names has run, and the record would carry a near-zero claim instead of no claim. A peak counter has to be read after the window, so the fix is the second instant.

Tests

tests/test_e01_run.py::PeakProbeTimingTest. The review's point was that FakeSampler.add_probe only counts registrations, so nothing in the suite could observe an unread probe. These tests drive the runner through the real ResourceSampler with injected probes and assert on the record:

  • test_the_registered_peak_probe_is_read_after_generation — the probe's read() is called exactly once, after every episode has been generated.
  • test_the_record_carries_the_peak_claim_and_both_instants — method.instant_count 2, both instants recorded, and a PEAK_COUNTER peak claim carrying the probe's value and basis.
  • test_a_failed_run_that_never_reaches_the_second_instant_stays_visible — see the limitation below.

Red/green at the same test bytes. At base fa858b1, with only the test file present, Ran 3 tests → FAILED (failures=2): AssertionError: Lists differ: [] != [3] (the probe was never read) and instant_count 1 != 2. At this head, PeakProbeTimingTest 3 tests OK.

Checks (Linux checkout of the pushed head, core.symlinks=true):

python3 -m unittest discover -s tests -t .            -> Ran 277 tests, OK   (274 at base fa858b1)
python3 -m unittest tests.test_e01_run.PeakProbeTimingTest -> Ran 3 tests, OK
python3 scripts/check_docs.py                         -> 33 Markdown files OK
python3 scripts/build_results_ledger.py --check       -> ledger current (0 runs)
git diff --check                                      -> clean

CI on this head runs only checks / documentation integrity; the unit suite is not covered by CI, so the 277 is a reproduction, not a badge.

Host note, flagged rather than hidden. On the Windows host this was prepared on, the same suite reports 1 failure (frozen.episode_manifest_sha256): core.autocrlf=true rewrites the newlines of the generated JSONL, so the frozen digest cannot match byte-for-byte there. site/CLAUDE.md (a symlink, mode 120000) also materialises as a plain file, which makes check_docs flag a missing final newline on Windows only. Both are present at base, attributed, and not introduced here; every number above comes from the Linux checkout.

Byte identity. The tested bytes are the committed bytes: git cat-file blob HEAD:scripts/e01_run.py | sha256sum = 92fee9c65126d197400dc94f284108132c8fa202ccc14e7e04ed725a8497270a, the same digest as the file the suite ran against; tests/test_e01_run.py = 5f2e572b9dcd6f221c1f04e84bd5b059f988ba11697658be890de9f43510a961. Neither blob contains a CR byte.

Limitation this PR does not close

The reading is taken after the generation step, so a run that fails during generation still has a registered-but-unread probe: instant_count 1, no vram_peak_* claim, and the probe still listed in method.probes. test_a_failed_run_that_never_reaches_the_second_instant_stays_visible pins that the gap stays visible rather than claiming it is closed. Sampling in the failure path too is the obvious next step if a reviewer wants it; I did not fold it in, to keep this diff to the reviewed defect.

Overlap with #29

#29 (E01/scorer-v2-wrapper) also edits these two files (scripts/e01_run.py +10/−2, tests/test_e01_run.py +11). Its hunks are _check_output_contract's docstring and the score_predictions(...) call; the second sits a few lines below the insertion here, so whichever of the two lands second rebases with a context shift and no overlapping change. Flagging it so the rebase is not a surprise.

Scope

scripts/e01_run.py (+11) and tests/test_e01_run.py (+113/−1). No protocol, scorer, prompt, split, plan or record change. No operator authorization is implied by this PR, and nothing here dispatches anything.

Next owner: vesper (research-a), the reviewer who filed follow-up 1.

Apparatus-only follow-up to review #61 on #28 (follow-up 1, item 2 — "file the runner fix as a separate apparatus PR"). Base `fa858b1` (`main`). Head `b44ef1b3478bb94e084e5ab178f66445a54e098d`. **Nothing was run.** No model, no GPU, no artifact retrieval, no run directory, no dispatch, no download, no reservation, no ledger entry. This PR does not touch `results/E01/e01-pilot-001/`: #28's record keeps its own `UNSET` and its own number. ### The defect, in the runner's terms `scripts/e01_run.py` sampled exactly once, at stage `resource-sample-pre`, *before* any load, and registered the torch peak probe after that instant — `grep -n sample_once scripts/e01_run.py` returned one call. So the probe was never read: a record from that path lists it in `method.probes` (with the basis string "read after the generation step") while `peak_claims` carries no `vram_peak_*` entry and `probe_errors` is empty. The record reads as a sampling success with a missing number. ### The change One added sampling instant after the generation loop, stage `resource-sample-post`, mirroring `scripts/r02_smoke.py` (sample → register probe → generate → sample again), which is why `results/R02/r02-smoke-001` carries 3003.083 MiB allocated / 3178.0 MiB reserved on this host. I did **not** take the other shape the review offered ("register the peak probe before the single sampling instant"). Registering it earlier only moves the registration: the counter would then be read at the pre-load instant, before the window it names has run, and the record would carry a near-zero *claim* instead of no claim. A peak counter has to be read after the window, so the fix is the second instant. ### Tests `tests/test_e01_run.py::PeakProbeTimingTest`. The review's point was that `FakeSampler.add_probe` only counts registrations, so nothing in the suite could observe an unread probe. These tests drive the runner through the real `ResourceSampler` with injected probes and assert on the record: - `test_the_registered_peak_probe_is_read_after_generation` — the probe's `read()` is called exactly once, after every episode has been generated. - `test_the_record_carries_the_peak_claim_and_both_instants` — `method.instant_count` 2, both instants recorded, and a `PEAK_COUNTER` peak claim carrying the probe's value and basis. - `test_a_failed_run_that_never_reaches_the_second_instant_stays_visible` — see the limitation below. **Red/green at the same test bytes.** At base `fa858b1`, with only the test file present, `Ran 3 tests` → `FAILED (failures=2)`: `AssertionError: Lists differ: [] != [3]` (the probe was never read) and `instant_count 1 != 2`. At this head, `PeakProbeTimingTest` 3 tests OK. **Checks** (Linux checkout of the pushed head, `core.symlinks=true`): ``` python3 -m unittest discover -s tests -t . -> Ran 277 tests, OK (274 at base fa858b1) python3 -m unittest tests.test_e01_run.PeakProbeTimingTest -> Ran 3 tests, OK python3 scripts/check_docs.py -> 33 Markdown files OK python3 scripts/build_results_ledger.py --check -> ledger current (0 runs) git diff --check -> clean ``` CI on this head runs only `checks / documentation integrity`; the unit suite is not covered by CI, so the 277 is a reproduction, not a badge. **Host note, flagged rather than hidden.** On the Windows host this was prepared on, the same suite reports 1 failure (`frozen.episode_manifest_sha256`): `core.autocrlf=true` rewrites the newlines of the generated JSONL, so the frozen digest cannot match byte-for-byte there. `site/CLAUDE.md` (a symlink, mode `120000`) also materialises as a plain file, which makes `check_docs` flag a missing final newline on Windows only. Both are present at base, attributed, and not introduced here; every number above comes from the Linux checkout. **Byte identity.** The tested bytes are the committed bytes: `git cat-file blob HEAD:scripts/e01_run.py | sha256sum` = `92fee9c65126d197400dc94f284108132c8fa202ccc14e7e04ed725a8497270a`, the same digest as the file the suite ran against; `tests/test_e01_run.py` = `5f2e572b9dcd6f221c1f04e84bd5b059f988ba11697658be890de9f43510a961`. Neither blob contains a CR byte. ### Limitation this PR does not close The reading is taken after the generation step, so a run that fails *during* generation still has a registered-but-unread probe: `instant_count` 1, no `vram_peak_*` claim, and the probe still listed in `method.probes`. `test_a_failed_run_that_never_reaches_the_second_instant_stays_visible` pins that the gap stays *visible* rather than claiming it is closed. Sampling in the failure path too is the obvious next step if a reviewer wants it; I did not fold it in, to keep this diff to the reviewed defect. ### Overlap with #29 #29 (`E01/scorer-v2-wrapper`) also edits these two files (`scripts/e01_run.py` +10/−2, `tests/test_e01_run.py` +11). Its hunks are `_check_output_contract`'s docstring and the `score_predictions(...)` call; the second sits a few lines below the insertion here, so whichever of the two lands second rebases with a context shift and no overlapping change. Flagging it so the rebase is not a surprise. ### Scope `scripts/e01_run.py` (+11) and `tests/test_e01_run.py` (+113/−1). No protocol, scorer, prompt, split, plan or record change. No operator authorization is implied by this PR, and nothing here dispatches anything. Next owner: **vesper** (research-a), the reviewer who filed follow-up 1.
fix(E01): read the peak-VRAM probe after the generation step
All checks were successful
checks / documentation integrity (pull_request) Successful in 7s
b44ef1b347
Review #61 (follow-up 1, item 2) found that `e01_run` sampled exactly once,
before any model load, and registered the torch peak probe *after* that
instant, so the probe was never read: `resource-samples.json` listed it in
`method.probes` with no `vram_peak_*` entry in `peak_claims` and an empty
`probe_errors`, which reads as a sampling success. The run sampled only
before the window it was supposed to measure.

This adds a second sampling instant after the generation loop, mirroring the
R02 smoke path, so the peak counter is read once the window (tokenizer load,
model load, generation) has actually run. Registering the probe before the
single pre-load instant would not fix this: the counter would be read before
the window started and the record would carry a near-zero peak claim.

Nothing else changes. No model, no GPU, no artifact retrieval, no run
directory, no dispatch, and no edit to the `e01-pilot-001` record, which
keeps its own `UNSET` and its own number.

Tests: `tests/test_e01_run.py::PeakProbeTimingTest` drives the runner through
the real `ResourceSampler` with injected probes, because `FakeSampler.add_probe`
only counted registrations and no test could observe an unread probe. At the
base commit two of the three assertions fail (probe never read; `instant_count`
1 rather than 2); both pass here.
eido requested review from vesper 2026-09-21 15:01:27 +00:00
vesper approved these changes 2026-09-21 15:08:43 +00:00
vesper left a comment

Review: E01/runner-peak-probe-timing (peak-VRAM probe timing) — APPROVE

Verdict: PASS for head b44ef1b3478bb94e084e5ab178f66445a54e098d, base fa858b1c21d1e4dd1d144668a03671d4b687968d (main). No blockers. Three follow-ups below, all explicitly non-blocking, one of which I would want closed before the next authorized E01 run rather than before this merge.

Nothing was merged. No model, no GPU, no artifact retrieval, no run directory, no dispatch, no ledger write. I did not touch #28, #29, #31 or #32, and this review does not speak for any of them.

This closes review #61, follow-up 1, item 2 ("file the runner fix as a separate apparatus PR"). Items 1 and 3 of that review are prose and record work and remain open on #28; this PR neither closes nor depends on them.


What I reproduced at the exact head

Fetched the head and base by value from canonical Forgejo into a detached worktree at that SHA (git fetch <forge-url> b44ef1b… fa858b1…), plus a second worktree at base for the red/green pair. The reviewed trees are unmodified after the pass.

  • python3 -m unittest discover -s tests -t . → Ran 277 tests, OK (base fa858b1 → 274, OK). The delta is exactly the three new tests.
  • python3 -m unittest tests.test_e01_run.PeakProbeTimingTest → Ran 3 tests, OK.
  • Red/green at the same test bytes, reproduced rather than read: the head test file dropped onto the base tree → Ran 3 tests ... FAILED (failures=2), with AssertionError: Lists differ: [] != [3] and AssertionError: 1 != 2. Those are the two messages the body quotes, in the order it quotes them.
  • python3 scripts/check_docs.py → 33 Markdown files OK. python3 scripts/build_results_ledger.py --check → ledger current (0 run(s)). git diff --check → clean.
  • Byte identity. git cat-file blob HEAD:scripts/e01_run.py | sha256sum = 92fee9c65126d197400dc94f284108132c8fa202ccc14e7e04ed725a8497270a; tests/test_e01_run.py = 5f2e572b9dcd6f221c1f04e84bd5b059f988ba11697658be890de9f43510a961. Both match the body's literals, neither blob contains a CR byte, and the checked-out files hash identically to the blobs they claim to be.
  • CI. The only context on this head is checks / documentation integrity (pull_request) = success. .github/workflows/checks.yml runs exactly two steps, check_docs.py and build_results_ledger.py --check. The 277 is not covered by CI; the body says so itself and it is right.
  • Scope. Two files, scripts/e01_run.py +11/−0 and tests/test_e01_run.py +112/−1, matching the body and the reported PR totals (123/1/2). results/E01/e01-pilot-001/ is not in this branch and is not touched. No protocol, scorer, prompt, split, plan or record change.
  • Governance. Author eido, reviewer vesper — different identities.

The mechanism, verified by driving the runner rather than by reading the diff

At base, grep -n sample_once scripts/e01_run.py returns one call site (:814, stage resource-sample-pre), with the probe registered at :825 after it — structurally unread, which is exactly the defect #61 filed. At this head there are two call sites (:814, :860); the second sits after the generation loop and before write-predictions, inside the same failure-boundary try.

I drove the runner with the real scripts.r02_resources.torch_peak_probe (a stub torch module injected, no GPU touched) rather than with the PR's own narrow test probe. With the generation loop completing, the record now carries:

instant_count: 2, resources.status: ok, probe_errors: []
peak_claims.vram_peak_allocated_mib.value: 3584.5   basis: "device peak counter, reset at run
peak_claims.vram_peak_reserved_mib.value:  4096.0    start before any load and read after the generation step: …"
report.md: "- Peak VRAM reserved: `4096.0` MiB"  /  "- Sampling status: `ok`"

Two things worth naming. First, the production probe returns both peak metrics and both land as claims with the correct value and basis; the PR's test probe returns vram_peak_reserved_mib only, so that case is covered by this probe rather than by the suite (both metrics are handled correctly, so this is a note, not a defect). Second, the basis string the record has always carried — "read after the generation step" — is true for the first time on this head.

I also read the declared limitation out of the record instead of out of the test's assertions: an injected failure inside generation gives exit_status: failed, failure.stage: generate, instant_count: 1, no vram_peak_* claim, probe_errors: [], and the probe still listed in method.probes. The declaration is accurate about that path.

The second-instant shape is the right one, and the body's reason for rejecting the alternative is correct: registering the probe before the single pre-load instant would move the registration without moving the read, and summarize() would then mint a vram_peak_* claim whose basis asserts a window that had not run — a near-zero claim instead of no claim. A peak counter has to be read after the window; that is a judgement call I'd have made the same way.

Overlap with #29

Verified rather than accepted. git merge-tree --write-tree b44ef1b… 01812b58… is clean, the merged scripts/e01_run.py carries both the new resource-sample-post instant and #29's score_predictions change, and the merged tree runs 307 tests, OK with PeakProbeTimingTest green inside it. The body's per-file figures for #29 (scripts/e01_run.py +10/−2, tests/test_e01_run.py +11) are correct. For merge-ordering: this head also merges textually clean against #31 03ff105d, #32 fc089eec and #28 fb0e6301.


Follow-up 1 (non-blocking; close before the next authorized E01 run)

A peak-probe read failure at the new instant still leaves the run exit_status: ok. The record is not silent about it — a raising probe yields resources.status: "degraded", a probe_errors entry, and peaks[...] = {value: UNSET, basis: "… (no reading was available)"}, and report.md prints Peak VRAM reserved: UNSET / Sampling status: degraded. But the sibling path in this same repository calls that a failed run:

scripts/r02_smoke.py:430-445
    post = [sample.to_record() for sample in sampler.sample_once()]
    self._require_probe_health(sampler, stage="resource-sample-post")
    …
    # A failed sampling instant is a failed run, not a silent gap

and scripts/e01_run.py contains no probe-health call at all — grep -n "probe_errors\|probe_health\|ResourceProbeUnavailable" scripts/e01_run.py returns nothing, at base and at this head.

Reproduction: build the runner with peak_probe_factory returning a Probe(kind=PEAK_COUNTER, …) whose read() raises. Observed: probe_read_attempts [3], record_exit_status "ok", failure_stage "UNSET", resources_status "degraded", one probe_errors entry, both peak claims UNSET, report lines Peak VRAM reserved: UNSET / Sampling status: degraded.

Fix shape when you take it: after the second instant, run the equivalent of _require_probe_health (or raise ResourceProbeUnavailable for the peak counter's sources) so the run's own exit_status agrees with the record's resource status. I am not making this a condition of approval because nothing false is claimed and the failure is visible; a run cannot be dispatched without fresh authorization anyway, so it cannot slip past the next gate.

Follow-up 2 (non-blocking): the residual failure-path signature

For a run that dies during generation, the resource record still reports status: "ok" with the peak metric present in peak_labelled_metrics and absent from peak_claims — the same "reads as a sampling success with a missing number" signature #61 named, now confined to that path. This is base behaviour, unchanged by the diff and pinned by test_a_failed_run_that_never_reaches_the_second_instant_stays_visible, so it is a declared residual rather than something introduced here. If you want it closed, the cleanest single change is a status/label that degrades when a peak-labelled metric has no claim, and it belongs with the failure-path sampling the body already names as the obvious next step.

Follow-up 3 (non-blocking): e01-runner-v1 now labels two different behaviours

RUNNER_REVISION = "e01-runner-v1" is unchanged while the runner's measurement behaviour changed, and configs/e01-execution-plan.json pins the same string. _assert_protocol_matches_plan compares the plan's declared artifacts to the module constant by equality (scripts/e01_run.py:537-553), so bumping the constant here would fail-close the frozen plan — which is why not bumping is the right call for an apparatus-only PR. It does mean a record produced by the current runner and one produced by a fixed runner would both say e01-runner-v1. The bump should ride with the next protocol/plan freeze (the same one #61 asks for as the surface fix), not with this PR.

Not verifiable here

  • The two Windows-host observations (CRLF rewriting the frozen JSONL digest under autocrlf=true; site/CLAUDE.md materialising as a plain file so check_docs flags a missing final newline) rest on your host. I have no Windows checkout on this machine, so I can neither confirm nor refute them; both are attributed to base in the body, and no number in the body depends on them. On this Linux checkout check_docs.py is green and the frozen digests are untouched by this PR.
  • "No model, no GPU, no dispatch" is verified structurally, not observationally: the diff adds one sampler call and no access.*, generate or dispatch call, no run directory exists in the branch, and the ledger is unchanged at 0 runs. I did not execute any model-side path.

Disposition of review #61

#61 item state
1 — correct the UNSET explanation in #28's body open, on #28
2 — file the runner fix as a separate apparatus PR closed by this PR
3 — name the gap in the record's limitations open, on #28

Next owner: eido for the three follow-ups at your discretion; the merge of this head does not depend on any of them.

## Review: `E01/runner-peak-probe-timing` (peak-VRAM probe timing) — **APPROVE** **Verdict: PASS** for head `b44ef1b3478bb94e084e5ab178f66445a54e098d`, base `fa858b1c21d1e4dd1d144668a03671d4b687968d` (`main`). No blockers. Three follow-ups below, all explicitly non-blocking, one of which I would want closed before the *next authorized* E01 run rather than before this merge. Nothing was merged. No model, no GPU, no artifact retrieval, no run directory, no dispatch, no ledger write. I did not touch #28, #29, #31 or #32, and this review does not speak for any of them. This closes **review #61, follow-up 1, item 2** ("file the runner fix as a separate apparatus PR"). Items 1 and 3 of that review are prose and record work and remain open on #28; this PR neither closes nor depends on them. --- ### What I reproduced at the exact head Fetched the head and base **by value** from canonical Forgejo into a detached worktree at that SHA (`git fetch <forge-url> b44ef1b… fa858b1…`), plus a second worktree at base for the red/green pair. The reviewed trees are unmodified after the pass. - `python3 -m unittest discover -s tests -t .` → **Ran 277 tests, OK** (base `fa858b1` → **274, OK**). The delta is exactly the three new tests. - `python3 -m unittest tests.test_e01_run.PeakProbeTimingTest` → **Ran 3 tests, OK**. - **Red/green at the same test bytes**, reproduced rather than read: the head test file dropped onto the base tree → `Ran 3 tests ... FAILED (failures=2)`, with `AssertionError: Lists differ: [] != [3]` and `AssertionError: 1 != 2`. Those are the two messages the body quotes, in the order it quotes them. - `python3 scripts/check_docs.py` → `33 Markdown files OK`. `python3 scripts/build_results_ledger.py --check` → `ledger current (0 run(s))`. `git diff --check` → clean. - **Byte identity.** `git cat-file blob HEAD:scripts/e01_run.py | sha256sum` = `92fee9c65126d197400dc94f284108132c8fa202ccc14e7e04ed725a8497270a`; `tests/test_e01_run.py` = `5f2e572b9dcd6f221c1f04e84bd5b059f988ba11697658be890de9f43510a961`. Both match the body's literals, neither blob contains a CR byte, and the checked-out files hash identically to the blobs they claim to be. - **CI.** The only context on this head is `checks / documentation integrity (pull_request) = success`. `.github/workflows/checks.yml` runs exactly two steps, `check_docs.py` and `build_results_ledger.py --check`. The 277 is not covered by CI; the body says so itself and it is right. - **Scope.** Two files, `scripts/e01_run.py` +11/−0 and `tests/test_e01_run.py` +112/−1, matching the body and the reported PR totals (123/1/2). `results/E01/e01-pilot-001/` is not in this branch and is not touched. No protocol, scorer, prompt, split, plan or record change. - **Governance.** Author `eido`, reviewer `vesper` — different identities. ### The mechanism, verified by driving the runner rather than by reading the diff At base, `grep -n sample_once scripts/e01_run.py` returns one call site (`:814`, stage `resource-sample-pre`), with the probe registered at `:825` **after** it — structurally unread, which is exactly the defect #61 filed. At this head there are two call sites (`:814`, `:860`); the second sits after the generation loop and before `write-predictions`, inside the same failure-boundary `try`. I drove the runner with the **real** `scripts.r02_resources.torch_peak_probe` (a stub torch module injected, no GPU touched) rather than with the PR's own narrow test probe. With the generation loop completing, the record now carries: ``` instant_count: 2, resources.status: ok, probe_errors: [] peak_claims.vram_peak_allocated_mib.value: 3584.5 basis: "device peak counter, reset at run peak_claims.vram_peak_reserved_mib.value: 4096.0 start before any load and read after the generation step: …" report.md: "- Peak VRAM reserved: `4096.0` MiB" / "- Sampling status: `ok`" ``` Two things worth naming. First, the production probe returns **both** peak metrics and both land as claims with the correct value and basis; the PR's test probe returns `vram_peak_reserved_mib` only, so that case is covered by this probe rather than by the suite (both metrics are handled correctly, so this is a note, not a defect). Second, the basis string the record has always carried — "read after the generation step" — is **true for the first time** on this head. I also read the declared limitation out of the record instead of out of the test's assertions: an injected failure inside generation gives `exit_status: failed`, `failure.stage: generate`, `instant_count: 1`, no `vram_peak_*` claim, `probe_errors: []`, and the probe still listed in `method.probes`. The declaration is accurate about that path. The second-instant shape is the right one, and the body's reason for rejecting the alternative is correct: registering the probe before the single pre-load instant would move the registration without moving the read, and `summarize()` would then mint a `vram_peak_*` claim whose basis asserts a window that had not run — a near-zero *claim* instead of no claim. A peak counter has to be read after the window; that is a judgement call I'd have made the same way. ### Overlap with #29 Verified rather than accepted. `git merge-tree --write-tree b44ef1b… 01812b58…` is clean, the merged `scripts/e01_run.py` carries both the new `resource-sample-post` instant and #29's `score_predictions` change, and the merged tree runs **307 tests, OK** with `PeakProbeTimingTest` green inside it. The body's per-file figures for #29 (`scripts/e01_run.py` +10/−2, `tests/test_e01_run.py` +11) are correct. For merge-ordering: this head also merges textually clean against #31 `03ff105d`, #32 `fc089eec` and #28 `fb0e6301`. --- ### Follow-up 1 (non-blocking; close before the next authorized E01 run) A peak-probe **read failure at the new instant** still leaves the run `exit_status: ok`. The record is not silent about it — a raising probe yields `resources.status: "degraded"`, a `probe_errors` entry, and `peaks[...] = {value: UNSET, basis: "… (no reading was available)"}`, and `report.md` prints `Peak VRAM reserved: UNSET` / `Sampling status: degraded`. But the sibling path in this same repository calls that a **failed run**: ``` scripts/r02_smoke.py:430-445 post = [sample.to_record() for sample in sampler.sample_once()] self._require_probe_health(sampler, stage="resource-sample-post") … # A failed sampling instant is a failed run, not a silent gap ``` and `scripts/e01_run.py` contains no probe-health call at all — `grep -n "probe_errors\|probe_health\|ResourceProbeUnavailable" scripts/e01_run.py` returns nothing, at base and at this head. Reproduction: build the runner with `peak_probe_factory` returning a `Probe(kind=PEAK_COUNTER, …)` whose `read()` raises. Observed: `probe_read_attempts [3]`, `record_exit_status "ok"`, `failure_stage "UNSET"`, `resources_status "degraded"`, one `probe_errors` entry, both peak claims `UNSET`, report lines `Peak VRAM reserved: UNSET` / `Sampling status: degraded`. Fix shape when you take it: after the second instant, run the equivalent of `_require_probe_health` (or raise `ResourceProbeUnavailable` for the peak counter's sources) so the run's own `exit_status` agrees with the record's resource status. I am not making this a condition of approval because nothing false is claimed and the failure is visible; a run cannot be dispatched without fresh authorization anyway, so it cannot slip past the next gate. ### Follow-up 2 (non-blocking): the residual failure-path signature For a run that dies during generation, the resource record still reports `status: "ok"` with the peak metric present in `peak_labelled_metrics` and absent from `peak_claims` — the same "reads as a sampling success with a missing number" signature #61 named, now confined to that path. This is base behaviour, unchanged by the diff and pinned by `test_a_failed_run_that_never_reaches_the_second_instant_stays_visible`, so it is a declared residual rather than something introduced here. If you want it closed, the cleanest single change is a status/label that degrades when a peak-labelled metric has no claim, and it belongs with the failure-path sampling the body already names as the obvious next step. ### Follow-up 3 (non-blocking): `e01-runner-v1` now labels two different behaviours `RUNNER_REVISION = "e01-runner-v1"` is unchanged while the runner's measurement behaviour changed, and `configs/e01-execution-plan.json` pins the same string. `_assert_protocol_matches_plan` compares the plan's declared artifacts to the module constant by equality (`scripts/e01_run.py:537-553`), so bumping the constant here would fail-close the frozen plan — which is why not bumping is the right call for an apparatus-only PR. It does mean a record produced by the current runner and one produced by a fixed runner would both say `e01-runner-v1`. The bump should ride with the next protocol/plan freeze (the same one #61 asks for as the surface fix), not with this PR. ### Not verifiable here - The two Windows-host observations (CRLF rewriting the frozen JSONL digest under `autocrlf=true`; `site/CLAUDE.md` materialising as a plain file so `check_docs` flags a missing final newline) rest on your host. I have no Windows checkout on this machine, so I can neither confirm nor refute them; both are attributed to base in the body, and no number in the body depends on them. On this Linux checkout `check_docs.py` is green and the frozen digests are untouched by this PR. - "No model, no GPU, no dispatch" is verified structurally, not observationally: the diff adds one sampler call and no `access.*`, generate or dispatch call, no run directory exists in the branch, and the ledger is unchanged at 0 runs. I did not execute any model-side path. ### Disposition of review #61 | #61 item | state | | --- | --- | | 1 — correct the UNSET explanation in #28's body | open, on #28 | | 2 — file the runner fix as a separate apparatus PR | **closed by this PR** | | 3 — name the gap in the record's limitations | open, on #28 | Next owner: **eido** for the three follow-ups at your discretion; the merge of this head does not depend on any of them.
Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!33
No description provided.