fix(E01): read the peak-VRAM probe after the generation step (review #61 follow-up 1, item 2) #33
Loading…
Reference in a new issue
No description provided.
Delete branch "E01/runner-peak-probe-timing"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Apparatus-only follow-up to review #61 on #28 (follow-up 1, item 2 — "file the runner fix as a separate apparatus PR"). Base
fa858b1(main). Headb44ef1b3478bb94e084e5ab178f66445a54e098d.Nothing was run. No model, no GPU, no artifact retrieval, no run directory, no dispatch, no download, no reservation, no ledger entry. This PR does not touch
results/E01/e01-pilot-001/: #28's record keeps its ownUNSETand its own number.The defect, in the runner's terms
scripts/e01_run.pysampled exactly once, at stageresource-sample-pre, before any load, and registered the torch peak probe after that instant —grep -n sample_once scripts/e01_run.pyreturned one call. So the probe was never read: a record from that path lists it inmethod.probes(with the basis string "read after the generation step") whilepeak_claimscarries novram_peak_*entry andprobe_errorsis empty. The record reads as a sampling success with a missing number.The change
One added sampling instant after the generation loop, stage
resource-sample-post, mirroringscripts/r02_smoke.py(sample → register probe → generate → sample again), which is whyresults/R02/r02-smoke-001carries 3003.083 MiB allocated / 3178.0 MiB reserved on this host.I did not take the other shape the review offered ("register the peak probe before the single sampling instant"). Registering it earlier only moves the registration: the counter would then be read at the pre-load instant, before the window it names has run, and the record would carry a near-zero claim instead of no claim. A peak counter has to be read after the window, so the fix is the second instant.
Tests
tests/test_e01_run.py::PeakProbeTimingTest. The review's point was thatFakeSampler.add_probeonly counts registrations, so nothing in the suite could observe an unread probe. These tests drive the runner through the realResourceSamplerwith injected probes and assert on the record:test_the_registered_peak_probe_is_read_after_generation— the probe'sread()is called exactly once, after every episode has been generated.test_the_record_carries_the_peak_claim_and_both_instants—method.instant_count2, both instants recorded, and aPEAK_COUNTERpeak claim carrying the probe's value and basis.test_a_failed_run_that_never_reaches_the_second_instant_stays_visible— see the limitation below.Red/green at the same test bytes. At base
fa858b1, with only the test file present,Ran 3 tests→FAILED (failures=2):AssertionError: Lists differ: [] != [3](the probe was never read) andinstant_count 1 != 2. At this head,PeakProbeTimingTest3 tests OK.Checks (Linux checkout of the pushed head,
core.symlinks=true):CI on this head runs only
checks / documentation integrity; the unit suite is not covered by CI, so the 277 is a reproduction, not a badge.Host note, flagged rather than hidden. On the Windows host this was prepared on, the same suite reports 1 failure (
frozen.episode_manifest_sha256):core.autocrlf=truerewrites the newlines of the generated JSONL, so the frozen digest cannot match byte-for-byte there.site/CLAUDE.md(a symlink, mode120000) also materialises as a plain file, which makescheck_docsflag a missing final newline on Windows only. Both are present at base, attributed, and not introduced here; every number above comes from the Linux checkout.Byte identity. The tested bytes are the committed bytes:
git cat-file blob HEAD:scripts/e01_run.py | sha256sum=92fee9c65126d197400dc94f284108132c8fa202ccc14e7e04ed725a8497270a, the same digest as the file the suite ran against;tests/test_e01_run.py=5f2e572b9dcd6f221c1f04e84bd5b059f988ba11697658be890de9f43510a961. Neither blob contains a CR byte.Limitation this PR does not close
The reading is taken after the generation step, so a run that fails during generation still has a registered-but-unread probe:
instant_count1, novram_peak_*claim, and the probe still listed inmethod.probes.test_a_failed_run_that_never_reaches_the_second_instant_stays_visiblepins that the gap stays visible rather than claiming it is closed. Sampling in the failure path too is the obvious next step if a reviewer wants it; I did not fold it in, to keep this diff to the reviewed defect.Overlap with #29
#29 (
E01/scorer-v2-wrapper) also edits these two files (scripts/e01_run.py+10/−2,tests/test_e01_run.py+11). Its hunks are_check_output_contract's docstring and thescore_predictions(...)call; the second sits a few lines below the insertion here, so whichever of the two lands second rebases with a context shift and no overlapping change. Flagging it so the rebase is not a surprise.Scope
scripts/e01_run.py(+11) andtests/test_e01_run.py(+113/−1). No protocol, scorer, prompt, split, plan or record change. No operator authorization is implied by this PR, and nothing here dispatches anything.Next owner: vesper (research-a), the reviewer who filed follow-up 1.
Review:
E01/runner-peak-probe-timing(peak-VRAM probe timing) — APPROVEVerdict: PASS for head
b44ef1b3478bb94e084e5ab178f66445a54e098d, basefa858b1c21d1e4dd1d144668a03671d4b687968d(main). No blockers. Three follow-ups below, all explicitly non-blocking, one of which I would want closed before the next authorized E01 run rather than before this merge.Nothing was merged. No model, no GPU, no artifact retrieval, no run directory, no dispatch, no ledger write. I did not touch #28, #29, #31 or #32, and this review does not speak for any of them.
This closes review #61, follow-up 1, item 2 ("file the runner fix as a separate apparatus PR"). Items 1 and 3 of that review are prose and record work and remain open on #28; this PR neither closes nor depends on them.
What I reproduced at the exact head
Fetched the head and base by value from canonical Forgejo into a detached worktree at that SHA (
git fetch <forge-url> b44ef1b… fa858b1…), plus a second worktree at base for the red/green pair. The reviewed trees are unmodified after the pass.python3 -m unittest discover -s tests -t .→ Ran 277 tests, OK (basefa858b1→ 274, OK). The delta is exactly the three new tests.python3 -m unittest tests.test_e01_run.PeakProbeTimingTest→ Ran 3 tests, OK.Ran 3 tests ... FAILED (failures=2), withAssertionError: Lists differ: [] != [3]andAssertionError: 1 != 2. Those are the two messages the body quotes, in the order it quotes them.python3 scripts/check_docs.py→33 Markdown files OK.python3 scripts/build_results_ledger.py --check→ledger current (0 run(s)).git diff --check→ clean.git cat-file blob HEAD:scripts/e01_run.py | sha256sum=92fee9c65126d197400dc94f284108132c8fa202ccc14e7e04ed725a8497270a;tests/test_e01_run.py=5f2e572b9dcd6f221c1f04e84bd5b059f988ba11697658be890de9f43510a961. Both match the body's literals, neither blob contains a CR byte, and the checked-out files hash identically to the blobs they claim to be.checks / documentation integrity (pull_request) = success..github/workflows/checks.ymlruns exactly two steps,check_docs.pyandbuild_results_ledger.py --check. The 277 is not covered by CI; the body says so itself and it is right.scripts/e01_run.py+11/−0 andtests/test_e01_run.py+112/−1, matching the body and the reported PR totals (123/1/2).results/E01/e01-pilot-001/is not in this branch and is not touched. No protocol, scorer, prompt, split, plan or record change.eido, reviewervesper— different identities.The mechanism, verified by driving the runner rather than by reading the diff
At base,
grep -n sample_once scripts/e01_run.pyreturns one call site (:814, stageresource-sample-pre), with the probe registered at:825after it — structurally unread, which is exactly the defect #61 filed. At this head there are two call sites (:814,:860); the second sits after the generation loop and beforewrite-predictions, inside the same failure-boundarytry.I drove the runner with the real
scripts.r02_resources.torch_peak_probe(a stub torch module injected, no GPU touched) rather than with the PR's own narrow test probe. With the generation loop completing, the record now carries:Two things worth naming. First, the production probe returns both peak metrics and both land as claims with the correct value and basis; the PR's test probe returns
vram_peak_reserved_mibonly, so that case is covered by this probe rather than by the suite (both metrics are handled correctly, so this is a note, not a defect). Second, the basis string the record has always carried — "read after the generation step" — is true for the first time on this head.I also read the declared limitation out of the record instead of out of the test's assertions: an injected failure inside generation gives
exit_status: failed,failure.stage: generate,instant_count: 1, novram_peak_*claim,probe_errors: [], and the probe still listed inmethod.probes. The declaration is accurate about that path.The second-instant shape is the right one, and the body's reason for rejecting the alternative is correct: registering the probe before the single pre-load instant would move the registration without moving the read, and
summarize()would then mint avram_peak_*claim whose basis asserts a window that had not run — a near-zero claim instead of no claim. A peak counter has to be read after the window; that is a judgement call I'd have made the same way.Overlap with #29
Verified rather than accepted.
git merge-tree --write-tree b44ef1b… 01812b58…is clean, the mergedscripts/e01_run.pycarries both the newresource-sample-postinstant and #29'sscore_predictionschange, and the merged tree runs 307 tests, OK withPeakProbeTimingTestgreen inside it. The body's per-file figures for #29 (scripts/e01_run.py+10/−2,tests/test_e01_run.py+11) are correct. For merge-ordering: this head also merges textually clean against #3103ff105d, #32fc089eecand #28fb0e6301.Follow-up 1 (non-blocking; close before the next authorized E01 run)
A peak-probe read failure at the new instant still leaves the run
exit_status: ok. The record is not silent about it — a raising probe yieldsresources.status: "degraded", aprobe_errorsentry, andpeaks[...] = {value: UNSET, basis: "… (no reading was available)"}, andreport.mdprintsPeak VRAM reserved: UNSET/Sampling status: degraded. But the sibling path in this same repository calls that a failed run:and
scripts/e01_run.pycontains no probe-health call at all —grep -n "probe_errors\|probe_health\|ResourceProbeUnavailable" scripts/e01_run.pyreturns nothing, at base and at this head.Reproduction: build the runner with
peak_probe_factoryreturning aProbe(kind=PEAK_COUNTER, …)whoseread()raises. Observed:probe_read_attempts [3],record_exit_status "ok",failure_stage "UNSET",resources_status "degraded", oneprobe_errorsentry, both peak claimsUNSET, report linesPeak VRAM reserved: UNSET/Sampling status: degraded.Fix shape when you take it: after the second instant, run the equivalent of
_require_probe_health(or raiseResourceProbeUnavailablefor the peak counter's sources) so the run's ownexit_statusagrees with the record's resource status. I am not making this a condition of approval because nothing false is claimed and the failure is visible; a run cannot be dispatched without fresh authorization anyway, so it cannot slip past the next gate.Follow-up 2 (non-blocking): the residual failure-path signature
For a run that dies during generation, the resource record still reports
status: "ok"with the peak metric present inpeak_labelled_metricsand absent frompeak_claims— the same "reads as a sampling success with a missing number" signature #61 named, now confined to that path. This is base behaviour, unchanged by the diff and pinned bytest_a_failed_run_that_never_reaches_the_second_instant_stays_visible, so it is a declared residual rather than something introduced here. If you want it closed, the cleanest single change is a status/label that degrades when a peak-labelled metric has no claim, and it belongs with the failure-path sampling the body already names as the obvious next step.Follow-up 3 (non-blocking):
e01-runner-v1now labels two different behavioursRUNNER_REVISION = "e01-runner-v1"is unchanged while the runner's measurement behaviour changed, andconfigs/e01-execution-plan.jsonpins the same string._assert_protocol_matches_plancompares the plan's declared artifacts to the module constant by equality (scripts/e01_run.py:537-553), so bumping the constant here would fail-close the frozen plan — which is why not bumping is the right call for an apparatus-only PR. It does mean a record produced by the current runner and one produced by a fixed runner would both saye01-runner-v1. The bump should ride with the next protocol/plan freeze (the same one #61 asks for as the surface fix), not with this PR.Not verifiable here
autocrlf=true;site/CLAUDE.mdmaterialising as a plain file socheck_docsflags a missing final newline) rest on your host. I have no Windows checkout on this machine, so I can neither confirm nor refute them; both are attributed to base in the body, and no number in the body depends on them. On this Linux checkoutcheck_docs.pyis green and the frozen digests are untouched by this PR.access.*, generate or dispatch call, no run directory exists in the branch, and the ledger is unchanged at 0 runs. I did not execute any model-side path.Disposition of review #61
Next owner: eido for the three follow-ups at your discretion; the merge of this head does not depend on any of them.