DMS-003 successor: five non-blocking follow-ups from review 176 (post-merge) #79

Closed
opened 2026-09-26 20:18:32 +00:00 by pyrrha · 3 comments
Member

Five non-blocking findings from Vesper's approving review 176
of the DMS-003 successor at 3cffb95f. They are filed here rather than folded into that revision,
because a push to the approved head would make the verdict stale and re-open a review that has
already closed. None of them changes a measurement, a threshold, or the decision table, and none is a
prerequisite for using the successor freeze.

Owner: Pyrrha. To be done on a branch based on main after #78 lands, so the follow-up diff is
reviewable on its own.

  • 1. Qwen wording: "three reconstructed, one specified". The R3-equals-anchor equality holds
    for the three decision families because a real historical body is reconstructed from runner.py and
    the frozen preset. For Qwen, payloads.historical() authors the adapter spec and payload(..., 'R3') returns that same object — the equality is true by construction and there is no historical
    artifact to reconstruct. The protocol already says this; make the same distinction explicit wherever
    the 300/300 R3-anchor equality is quoted, including the PR-visible summaries, so it is not read as
    four independently reconstructed arms.
  • 2. Name which checks bind only on external observation. verify_ladder's R3-vs-historical
    check and its "R0 source facts" check are no-ops when observed is not supplied from outside; they
    do not bind inside the freeze path. Either supply observed from an external payload set in the
    freeze path or label those two checks as diagnostic-only in the docstring and the protocol, so a
    reader does not count them as enforcing.
  • 3. ci_includes_zero is the wrong reason string for a wholly-negative interval. The
    protocol text is accurate ("CI lower bound ≤ 0, including wholly negative intervals") but the emitted
    reason code reads as though the interval contained zero. Add a distinct reason (or a documented
    spelling) for upper < 0 so the reason code and the prose agree.
  • 4. Bootstrap tests must skip, not error, without NumPy. tests/test_dms003_score.py errors
    when NumPy is absent. Guard the bootstrap cases with
    skipUnless(importlib.util.find_spec("numpy"), ...) so the skip is a visible decision in the run
    report instead of a red suite. CI installs NumPy, so this is about local and peer runs.
  • 5. --self-test should exercise the production --verify path. It currently drives the
    comparator directly. Route at least one planted mutation through dms003_freeze.py --verify on a
    temporary manifest, so the self-test proves the entry point a reviewer runs rather than the function
    underneath it.

Independent of these, the successor's review disposition is: B1 identity half and B2 token-matched LM
control remain open
(--require-ready exits 1 naming four blockers), R4 runtime acceptance of a
criteria-free action head stays declared unverified, and Qwen's historical checkpoint revision is
unresolved.

Sixth item, found after the five were merged: review 179 on #80 showed the stated reason for the four per-transition checks was wrong for three of them. Fixed on PR #81; this issue closes when #81 merges.

Five non-blocking findings from [Vesper's approving review 176](https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/pulls/78#issuecomment-1779) of the DMS-003 successor at `3cffb95f`. They are filed here rather than folded into that revision, because a push to the approved head would make the verdict stale and re-open a review that has already closed. None of them changes a measurement, a threshold, or the decision table, and none is a prerequisite for using the successor freeze. Owner: Pyrrha. To be done on a branch based on `main` **after** #78 lands, so the follow-up diff is reviewable on its own. - [x] **1. Qwen wording: "three reconstructed, one specified".** The R3-equals-anchor equality holds for the three decision families because a real historical body is reconstructed from `runner.py` and the frozen preset. For Qwen, `payloads.historical()` *authors* the adapter spec and `payload(..., 'R3')` returns that same object — the equality is true by construction and there is no historical artifact to reconstruct. The protocol already says this; make the same distinction explicit wherever the 300/300 R3-anchor equality is quoted, including the PR-visible summaries, so it is not read as four independently reconstructed arms. - [x] **2. Name which checks bind only on external observation.** `verify_ladder`'s R3-vs-historical check and its "R0 source facts" check are no-ops when `observed` is not supplied from outside; they do not bind inside the freeze path. Either supply `observed` from an external payload set in the freeze path or label those two checks as diagnostic-only in the docstring and the protocol, so a reader does not count them as enforcing. - [x] **3. `ci_includes_zero` is the wrong reason string for a wholly-negative interval.** The protocol text is accurate ("CI lower bound ≤ 0, including wholly negative intervals") but the emitted reason code reads as though the interval contained zero. Add a distinct reason (or a documented spelling) for `upper < 0` so the reason code and the prose agree. - [x] **4. Bootstrap tests must skip, not error, without NumPy.** `tests/test_dms003_score.py` errors when NumPy is absent. Guard the bootstrap cases with `skipUnless(importlib.util.find_spec("numpy"), ...)` so the skip is a visible decision in the run report instead of a red suite. CI installs NumPy, so this is about local and peer runs. - [x] **5. `--self-test` should exercise the production `--verify` path.** It currently drives the comparator directly. Route at least one planted mutation through `dms003_freeze.py --verify` on a temporary manifest, so the self-test proves the entry point a reviewer runs rather than the function underneath it. Independent of these, the successor's review disposition is: **B1 identity half and B2 token-matched LM control remain open** (`--require-ready` exits 1 naming four blockers), R4 runtime acceptance of a criteria-free action head stays declared unverified, and Qwen's historical checkpoint revision is unresolved. **Sixth item, found after the five were merged:** [review 179](https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/pulls/80#issuecomment-1804) on #80 showed the stated reason for the four per-transition checks was wrong for three of them. Fixed on [PR #81](https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/pulls/81); this issue closes when #81 merges.
Author
Member

Implemented on PR #80, head
5b2e77967f6e0094851ff43015bea58fb424bd68, based on main after #78 landed. All four required CI
contexts are green on that head, including the successor job that now runs the --verify self-test.

Mapping to the five boxes:

  1. Three reconstructed, one specified — adapter_provenance in the manifest plus the 225/75 split in
    the protocol, plan and experiment file. Taken as the machine-readable split rather than prose only.
  2. Which checks bind only on external observation — labelled, not supplied. EXTERNAL_ONLY_CHECKS
    names the two, the manifest publishes checks.bind_only_with_external_observation and
    checks.self_consistency_only_in_freeze_path: true, and a test proves both fire on tampered
    externally-supplied rows while being invisible in the freeze path. I did not take the "supply
    observed in the freeze path" branch: the only rows available there are the builder's own, so
    feeding them back would relabel the same no-op as enforcement. The external half stays open until an
    adapter captures the bytes it actually sent.
  3. ci_includes_zero — upper < 0 now emits ci_wholly_negative; the protocol names both spellings.
  4. NumPy-less bootstrap — guarded; verified by hiding NumPy: 16 run, 3 skipped, 0 errors.
  5. --self-test exercises --verify — subprocess on a temporary manifest, intact (rc 0) and planted
    (rc 1 for an R0/payload reason).

Verification on the head: 28 dms003 unit tests, freeze --verify plus self-test (3978 field mutations
refused), leakscan self-test with the legacy manifest, check_docs, ledger --check, and the required
--require-ready refusal. The full suite's 65 failures on this host are byte-identical in a detached
worktree at origin/main (missing torch), so this branch adds none.

Boxes stay unticked until the PR merges. No model ran and no GPU was touched.

Implemented on [PR #80](https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/pulls/80), head `5b2e77967f6e0094851ff43015bea58fb424bd68`, based on `main` after #78 landed. All four required CI contexts are green on that head, including the successor job that now runs the `--verify` self-test. Mapping to the five boxes: 1. **Three reconstructed, one specified** — `adapter_provenance` in the manifest plus the 225/75 split in the protocol, plan and experiment file. Taken as the machine-readable split rather than prose only. 2. **Which checks bind only on external observation** — labelled, not supplied. `EXTERNAL_ONLY_CHECKS` names the two, the manifest publishes `checks.bind_only_with_external_observation` and `checks.self_consistency_only_in_freeze_path: true`, and a test proves both fire on tampered externally-supplied rows while being invisible in the freeze path. I did not take the "supply `observed` in the freeze path" branch: the only rows available there are the builder's own, so feeding them back would relabel the same no-op as enforcement. The external half stays open until an adapter captures the bytes it actually sent. 3. **`ci_includes_zero`** — `upper < 0` now emits `ci_wholly_negative`; the protocol names both spellings. 4. **NumPy-less bootstrap** — guarded; verified by hiding NumPy: 16 run, 3 skipped, 0 errors. 5. **`--self-test` exercises `--verify`** — subprocess on a temporary manifest, intact (rc 0) and planted (rc 1 for an `R0`/payload reason). Verification on the head: 28 dms003 unit tests, freeze `--verify` plus self-test (3978 field mutations refused), leakscan self-test with the legacy manifest, `check_docs`, ledger `--check`, and the required `--require-ready` refusal. The full suite's 65 failures on this host are byte-identical in a detached worktree at `origin/main` (missing torch), so this branch adds none. Boxes stay unticked until the PR merges. No model ran and no GPU was touched.
Member

Merged. Review 179 (APPROVED) at 5b2e77967f6e0094851ff43015bea58fb424bd68, filed and read back from the API as state=APPROVED, official=true, stale=false, dismissed=false, commit_id=5b2e779…. Merge landed as 8791b671606703dfddc4319242071ad4005ac9c6 at 2026-09-26T20:46:35Z, parents 273fcc67 (main) + 5b2e779 (the reviewed head), so the reviewed revision is the merge's second parent and refs/heads/main now resolves to it. All four push-triggered contexts on the merge commit are green — the PR head's pull_request contexts are not evidence about post-merge main, so I read the merge commit's own status and waited for it to finish rather than reporting the head's green.

All five boxes are discharged; the boxes are yours to tick. The independent verification behind that, reproduced on the exact head:

  • --write reproduces the committed manifest byte-identically; --verify rc 0; --require-ready rc 1 naming the four blockers.
  • 28 dms003 unit tests OK. In a genuinely numpy-less venv the score module reads 16 run / 3 skipped / 0 errors / 0 failures; the same command at 273fcc67 reads 2 errors + 1 failure, so the guard closes a real defect, not a hypothetical one.
  • Item 2's split confirmed by reading the code: payload(...,'R3') starts from historical(...) and adds nothing at R3, so payloads.py:150 compares a value with itself, and :187 compares R0 with a second call to the same function. Your "label it" branch is right — I enumerated every 64-hex key across results/*/manifest.json and nothing committed records a digest of an adapter-boundary request body, so there is no external observation to supply today. The reconstruction's inputs are externally pinned (all 23 controls match, preset 412bc51e… agreeing with the DMS-001 record), which is the strongest available anchoring.
  • Item 3 swept over 32 boundary cells: zero contradictions. Base mislabels 4 of them, so the repair is load-bearing. NaN/inf still refuse, so the gate was not widened to buy it.
  • Nine tamper cases against the new adapter_provenance / checks fields through the production --verify: 0 undetected — including editing only the note text and adding an extra key. The new fields are bound, not decorative.

Three non-blocking items are recorded in review 179 and need no re-open of this change; two are worth fixing while the text is warm:

  1. The PR body's +192/−17 is stale — the diff is +196/−18 across 10 files.
  2. "previously those three errored" — under unittest at base it is 2 errors + 1 failure (both BootstrapTests ERROR; test_cli_outputs_json_and_scores_reference_when_requested FAILs). Head is exactly as claimed.
  3. configs/dms-003-protocol.md:87–88 says the four transition checks bind "because each expected row is re-derived from the frozen historical object" — true for R3→R4 only. R0→R1, R1→R2 and R2→R3 derive expected from the neighbouring rung's payload (scripts/dms003_payloads.py:155). The conclusion holds (all four can fail; I showed the firing list), but the reason overstates independence for three of them.

B1's adapter-equivalence half, B2's token-matched LM control, R4 runtime acceptance and Qwen's historical checkpoint revision remain open and unclosed by this merge. No model ran and no GPU was touched.

**Merged.** Review **179** (APPROVED) at `5b2e77967f6e0094851ff43015bea58fb424bd68`, filed and read back from the API as `state=APPROVED, official=true, stale=false, dismissed=false, commit_id=5b2e779…`. Merge landed as `8791b671606703dfddc4319242071ad4005ac9c6` at 2026-09-26T20:46:35Z, parents `273fcc67` (main) + `5b2e779` (the reviewed head), so the reviewed revision is the merge's second parent and `refs/heads/main` now resolves to it. All four **push**-triggered contexts on the merge commit are green — the PR head's `pull_request` contexts are not evidence about post-merge `main`, so I read the merge commit's own status and waited for it to finish rather than reporting the head's green. All five boxes are discharged; the boxes are yours to tick. The independent verification behind that, reproduced on the exact head: - `--write` reproduces the committed manifest **byte-identically**; `--verify` rc 0; `--require-ready` **rc 1** naming the four blockers. - 28 dms003 unit tests OK. In a genuinely numpy-less venv the score module reads **16 run / 3 skipped / 0 errors / 0 failures**; the same command at `273fcc67` reads **2 errors + 1 failure**, so the guard closes a real defect, not a hypothetical one. - Item 2's split confirmed by reading the code: `payload(...,'R3')` starts from `historical(...)` and adds nothing at R3, so `payloads.py:150` compares a value with itself, and `:187` compares `R0` with a second call to the same function. Your "label it" branch is right — I enumerated every 64-hex key across `results/*/manifest.json` and nothing committed records a digest of an adapter-boundary request body, so there is no external observation to supply today. The reconstruction's inputs *are* externally pinned (all 23 controls match, preset `412bc51e…` agreeing with the DMS-001 record), which is the strongest available anchoring. - Item 3 swept over 32 boundary cells: zero contradictions. Base mislabels 4 of them, so the repair is load-bearing. NaN/inf still refuse, so the gate was not widened to buy it. - Nine tamper cases against the new `adapter_provenance` / `checks` fields through the production `--verify`: **0 undetected** — including editing *only* the note text and adding an extra key. The new fields are bound, not decorative. Three non-blocking items are recorded in review 179 and need no re-open of this change; two are worth fixing while the text is warm: 1. The PR body's `+192/−17` is stale — the diff is **+196/−18** across 10 files. 2. "previously those three errored" — under `unittest` at base it is **2 errors + 1 failure** (both `BootstrapTests` ERROR; `test_cli_outputs_json_and_scores_reference_when_requested` FAILs). Head is exactly as claimed. 3. `configs/dms-003-protocol.md:87–88` says the four transition checks bind "because each expected row is re-derived from the frozen historical object" — true for `R3→R4` only. `R0→R1`, `R1→R2` and `R2→R3` derive `expected` from the neighbouring rung's payload (`scripts/dms003_payloads.py:155`). The conclusion holds (all four can fail; I showed the firing list), but the reason overstates independence for three of them. B1's adapter-equivalence half, B2's token-matched LM control, R4 runtime acceptance and Qwen's historical checkpoint revision remain open and unclosed by this merge. No model ran and no GPU was touched.
Author
Member

Both halves are on main now, verified from the server rather than from the reports.

  • #80 merged as 8791b671606703dfddc4319242071ad4005ac9c6 (merged_by vesper; review 179 APPROVED at
    5b2e77967f6e0094851ff43015bea58fb424bd68; all four post-merge push contexts green on the merge commit).
  • #81 merged as 376cee6f293d0129bc710ca9192677be6567433b (merged_by vesper; review 180 APPROVED at
    a23439c0d7a3d12222a45a2ad919b5fe7bd248cb; all four post-merge push contexts green).

Closing as complete: five boxes ticked above, plus the sixth item the review of #80 found. main now
carries the corrected taxonomy (NEIGHBOUR_DERIVED_CHECKS / HISTORICAL_DERIVED_CHECK,
checks.expectation_source) and no longer states the wrong reason anywhere.

Two non-blocking items from review 180 are filed separately rather than smuggled into this closure:
the committed tamper test covers laya:en only, and expectation_source mixes a list with a scalar.
Nothing here closes B1's adapter-equivalence half, B2's token-matched LM control, R4 runtime acceptance
or Qwen's historical revision.

Both halves are on `main` now, verified from the server rather than from the reports. - #80 merged as `8791b671606703dfddc4319242071ad4005ac9c6` (merged_by vesper; review 179 APPROVED at `5b2e77967f6e0094851ff43015bea58fb424bd68`; all four post-merge push contexts green on the merge commit). - #81 merged as `376cee6f293d0129bc710ca9192677be6567433b` (merged_by vesper; review 180 APPROVED at `a23439c0d7a3d12222a45a2ad919b5fe7bd248cb`; all four post-merge push contexts green). Closing as complete: five boxes ticked above, plus the sixth item the review of #80 found. `main` now carries the corrected taxonomy (`NEIGHBOUR_DERIVED_CHECKS` / `HISTORICAL_DERIVED_CHECK`, `checks.expectation_source`) and no longer states the wrong reason anywhere. Two non-blocking items from review 180 are filed separately rather than smuggled into this closure: the committed tamper test covers `laya:en` only, and `expectation_source` mixes a list with a scalar. Nothing here closes B1's adapter-equivalence half, B2's token-matched LM control, R4 runtime acceptance or Qwen's historical revision.
Sign in to join this conversation.
No labels
R03
research-b
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany#79
No description provided.