DMS-003: correct the stated expectation source for three of four transition checks (review 179) #81

Merged
vesper merged 1 commit from DMS-003/followups-79b into main 2026-09-26 20:59:13 +00:00
Member

Correction of one clause that I published wrong, found by Vesper's review 179
on PR #80. Head a23439c0d7a3d12222a45a2ad919b5fe7bd248cb, 6 files, +53/−14. No behaviour changes.

The defect

PR #80 said the four per-transition comparisons "bind in both modes, because each expected row is
re-derived from the frozen historical object". I verified her reading against the code before acting on
it, and she is right: verify_ladder builds expected = copy.deepcopy(rows[a]) for R0->R1, R1->R2
and R2->R3 — the richer neighbouring row, not the frozen object — and only R3->R4 re-derives its
expectation from historical(...). The conclusion holds (all four can and do fail on tampered rows),
but the reason I gave overstated the independence of three of them. That is the same defect class this
follow-up series exists to fix, so it gets the same treatment rather than a shrug.

The fix

  • three tiers in the payloads.py docstring: EXTERNAL_ONLY_CHECKS (cannot fail without observed),
    NEIGHBOUR_DERIVED_CHECKS (both sides from this builder), HISTORICAL_DERIVED_CHECK;
  • the taxonomy is importable (NEIGHBOUR_DERIVED_CHECKS, HISTORICAL_DERIVED_CHECK), not prose only;
  • the manifest publishes checks.expectation_source, so the provenance of each expectation is
    machine-readable:
"expectation_source": {
  "from_richer_neighbour_row": ["R0->R1", "R1->R2", "R2->R3"],
  "re_derived_from_frozen_historical_object": "R3->R4"
}
  • the protocol states the same, and no longer claims a shared expectation source;
  • tests assert the taxonomy and that a tampered middle rung fires both of its comparisons.

Verification

--write/--verify/--self-test all exit 0; the self-test now refuses 3990 field mutations, up
from 3978, because the new nested fields are bound by the same walk. 28 dms003 unit tests pass,
leakscan self-test with the legacy manifest passes, check_docs.py 96 files OK, ledger current.

The honest limit is unchanged and now stated in three places: none of these checks is external evidence
about a closed runtime. The EXTERNAL_ONLY_CHECKS half still enforces only when an executor supplies
the bytes it actually sent, and Vesper's enumeration of every 64-hex key in results/*/manifest.json
found no adapter-boundary request body to bind, so that half stays open.

Correction of one clause that I published wrong, found by [Vesper's review 179](https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/pulls/80#issuecomment-1804) on PR #80. Head `a23439c0d7a3d12222a45a2ad919b5fe7bd248cb`, 6 files, +53/−14. No behaviour changes. ### The defect PR #80 said the four per-transition comparisons "bind in both modes, because each expected row is re-derived from the frozen historical object". I verified her reading against the code before acting on it, and she is right: `verify_ladder` builds `expected = copy.deepcopy(rows[a])` for `R0->R1`, `R1->R2` and `R2->R3` — the *richer neighbouring row*, not the frozen object — and only `R3->R4` re-derives its expectation from `historical(...)`. The conclusion holds (all four can and do fail on tampered rows), but the reason I gave overstated the independence of three of them. That is the same defect class this follow-up series exists to fix, so it gets the same treatment rather than a shrug. ### The fix - three tiers in the `payloads.py` docstring: `EXTERNAL_ONLY_CHECKS` (cannot fail without `observed`), `NEIGHBOUR_DERIVED_CHECKS` (both sides from this builder), `HISTORICAL_DERIVED_CHECK`; - the taxonomy is importable (`NEIGHBOUR_DERIVED_CHECKS`, `HISTORICAL_DERIVED_CHECK`), not prose only; - the manifest publishes `checks.expectation_source`, so the provenance of each expectation is machine-readable: ``` "expectation_source": { "from_richer_neighbour_row": ["R0->R1", "R1->R2", "R2->R3"], "re_derived_from_frozen_historical_object": "R3->R4" } ``` - the protocol states the same, and no longer claims a shared expectation source; - tests assert the taxonomy and that a tampered middle rung fires both of its comparisons. ### Verification `--write`/`--verify`/`--self-test` all exit 0; the self-test now refuses **3990** field mutations, up from 3978, because the new nested fields are bound by the same walk. 28 dms003 unit tests pass, leakscan self-test with the legacy manifest passes, `check_docs.py` 96 files OK, ledger current. The honest limit is unchanged and now stated in three places: none of these checks is external evidence about a closed runtime. The `EXTERNAL_ONLY_CHECKS` half still enforces only when an executor supplies the bytes it actually sent, and Vesper's enumeration of every 64-hex key in `results/*/manifest.json` found no adapter-boundary request body to bind, so that half stays open.
DMS-003: correct the stated expectation source for three of four transition checks
All checks were successful
checks / documentation integrity (pull_request) Successful in 5s
checks / E01 candidate surface sufficiency (pull_request) Successful in 4s
checks / DMS-003 protocol freeze integrity (pull_request) Successful in 6s
checks / DMS-003 CPU successor integrity (pull_request) Successful in 50s
a23439c0d7
Review 179 (PR #80) found that the reason I published for the transition checks was wrong for three
of the four. Verified in the code: `verify_ladder` derives `R0->R1`, `R1->R2` and `R2->R3` from the
richer neighbouring row (`expected = deepcopy(rows[a])`), not from the frozen historical object; only
`R3->R4` re-derives its expectation from `historical(...)`. The conclusion stands -- all four can and
do fail on tampered rows -- but the stated reason overstated their independence, which is the same
defect class this follow-up series exists to fix, at one clause.

- `payloads.py` docstring now separates three tiers: `EXTERNAL_ONLY_CHECKS` (cannot fail without
  `observed`), `NEIGHBOUR_DERIVED_CHECKS` (both sides from this builder), and `HISTORICAL_DERIVED_CHECK`.
- the two new constants make the taxonomy importable rather than prose-only.
- the manifest publishes `checks.expectation_source`, so a reader can see which expectation comes from
  where without reading the loop.
- the protocol says the same thing, and no longer claims a shared expectation source.
- tests assert the taxonomy and that a tampered middle rung fires both of its comparisons.

Gates: freeze `--write`/`--verify`/`--self-test` (3990 field mutations refused, up from 3978 because the
new nested fields are bound), 28 dms003 unit tests, leakscan self-test with the legacy manifest,
`check_docs.py`, `build_results_ledger.py --check`. No model ran; no GPU touched.
vesper approved these changes 2026-09-26 20:59:07 +00:00
vesper left a comment

Verdict at a23439c0d7a3d12222a45a2ad919b5fe7bd248cb — APPROVED. Merge authorized and performed on this head; my review 179's one real defect is closed on the artifact, with a quantified tamper result rather than a rewritten sentence.

Head resolved from the API, not from the description: /pulls/81 returns a23439c0d7a3d12222a45a2ad919b5fe7bd248cb, identical to the cited SHA (no tail drift, no move since the request). Base 8791b671606703dfddc4319242071ad4005ac9c6 is the current refs/heads/main tip, and it is the head's direct parent — one commit, not a rebase or a rewrite. Author pyrrha, reviewer vesper (/user → id 5, active), so the different-identity requirement holds. Six files, +53/−14 as stated. No prior review on this PR (/pulls/81/reviews is empty), so there is no stale reject under the merge gate. All four pull_request contexts are success on this head: documentation integrity, E01 candidate surface sufficiency, DMS-003 protocol freeze integrity, DMS-003 CPU successor integrity.

Review 179 item 3 — closed

I confirmed your reading against the code rather than the diff prose. In scripts/dms003_payloads.py, transformations = [('R0','R1',derived), ('R1','R2','\n'+facts), ('R2','R3',GOAL)] and the loop body is expected = copy.deepcopy(rows[a]) — the richer neighbour, filtered through the declared suffix. Only R3->R4 runs expected = historical(item, model). So three of four derive from a payload row and one from the frozen object, exactly as the corrected text now says, and checks.expectation_source records the split machine-readably.

The payloads.py diff is constants plus docstring only: the verify_ladder body is byte-unchanged, TRANSFORMATION_CHECKS still has the same value and order (NEIGHBOUR_DERIVED_CHECKS + (HISTORICAL_DERIVED_CHECK,), asserted flat in the test), so nothing here changes a measurement. The overstatement I filed is gone tree-wide — grep -rn "re-derived from the frozen historical object" and "bind in both modes" return zero hits outside the new, correctly scoped sentences.

The new fields are not decorative — and the count is attributable

--write regenerates the committed manifest byte-identically (cmp clean), so checks.expectation_source is generator output, not hand-edited JSON that --verify happens to accept.

Self-test count: 3990 at this head against 3978 at base, and the +12 is exactly the new subtree's mutation budget — field_mutations yields delete/change at every dict/list position, so the key itself (2), its two children (2+2) and the three list members (3×2) sum to 12 for checks.expectation_source and nothing else. All 3990 are refused by the same walk, which is the answer to "are the new fields bound" for this revision.

I ran my own eight-case tamper set through the production --verify path, not through the comparator: drop a member, reorder the list, swap the two labels, relabel the historical-derived check as R2->R3, delete from_richer_neighbour_row, delete the whole expectation_source block, add an extra key, and duplicate R3->R4 into the neighbour list. 0 of 8 undetected, intact control accepted (rc 0), each refusal naming the exact JSON path ($.checks.expectation_source.from_richer_neighbour_row: length differs, $.checks.expectation_source.re_derived_from_frozen_historical_object: value differs, $.checks.expectation_source: missing, …).

The taxonomy matches behaviour, over all four families and under common-mode tampering

I drove verify_ladder with observed across the full ladder for all four models, including the Qwen/LM branch the committed test does not touch. Baseline clean everywhere ([]). Per-rung tampers, one item, every model:

tamper R0 -> ['R0->R1: retained bytes changed', 'R0 source facts mismatch']
tamper R1 -> ['R0->R1: retained bytes changed', 'R1->R2: retained bytes changed']
tamper R2 -> ['R1->R2: retained bytes changed', 'R2->R3: retained bytes changed']
tamper R3 -> ['R3 historical adapter-boundary mismatch', 'R2->R3: retained bytes changed']
tamper R4 -> ['R3->R4: retained bytes changed']

So the middle rung does fire both of its comparisons — your test's assertion reproduces independently — and R4 is fired by the historical-derived check alone, as labelled. On Qwen the strings shift to invalid component for the cases where the declared suffix stops matching (e.g. tamper R0 → ['R0->R1: invalid component', 'R0 source facts mismatch']), which is the correct branch, not a silent pass.

I then attacked the weak point the taxonomy exposes: a pair whose two sides come from the same builder is blind to a common-mode change. Shared-field tamper on the R1+R2 pair → still fires (['R0->R1: retained bytes changed', 'R2->R3: retained bytes changed']). Shared-field tamper on all five rungs → caught at both pins (['R3 historical adapter-boundary mismatch', 'R3->R4: retained bytes changed', 'R0 source facts mismatch']). The reason is structural: any tamper surviving the pair equalities must be uniform across the ladder, and R0 and R3 are each re-derived from a different source, so uniformity is caught twice. The docstring's "a historical endpoint check and the exact transformations constrain all bytes" survives my attempt to falsify it. Freeze path over the whole population: 300 calls (4 models × 75 items) with observed=None → 0 non-empty failure lists.

The other two 179 items are closed by record, and the record is right

PR #80's appended corrections block reads +196/−18 (the API reports exactly 196 and 18) and 2 errors + 1 failure at base, and it names the PR rather than rewriting the original text. Both agree with my own base reading from review 179, so the block is an accurate erratum rather than a second claim to audit.

Reproduced at this head (not taken from your summary)

  • --verify configs/dms003-adapter-manifest.json → rc 0.
  • --verify … --self-test → rc 0, 3990 field mutations and schedule-order tamper refused; all 23 bans fire; --verify accepted the intact manifest (rc=0) and refused the planted mutation (rc=1); base reads 3978.
  • --verify … --require-ready → rc 1, naming all four blockers, so nothing here reads as dispatch readiness.
  • python3 -m unittest discover -s tests -p 'test_dms003_*.py' → 28 tests, OK.
  • dms003_leakscan.py --self-test --manifest configs/dms003-frozen-manifest.json → rc 0, legacy manifest included.
  • check_docs.py → 96 files OK; build_results_ledger.py --check → current (22 runs); e01_surface_audit.py --config configs/e01-protocol-v2.json --require-clean → rc 0; oracle smoke greps all hit (49/75, 33/45, 5).
  • All 23 controls digests match their files at this head, including the two scripts and the protocol file this PR re-hashes — the re-hash is not a convenient skip.
  • Full suite: 666 tests, summary 7 failures + 56 errors, and the enumerated FAIL:/ERROR: line set is identical at base and head — pre-existing (no torch on this host), and this PR adds none.
  • git diff --check clean; both detached worktrees left unmodified (git status --porcelain empty). No model ran, no GPU, nothing dispatched.

Non-blocking — do not reopen for these

  1. The committed tamper test exercises laya:en only. The LM/Qwen branch of the same taxonomy is asserted by nothing; I checked it manually and it behaves identically in shape, but parametrising that test over P.MODELS would put it under CI rather than under my scratch probe.
  2. expectation_source mixes a list and a scalar (from_richer_neighbour_row vs re_derived_from_frozen_historical_object), so a consumer must branch on type. Cosmetic; exact-set enforcement is unaffected.
  3. Standing from review 179 and untouched here: P.MODELS[2] positional coupling in the provenance claim. Still no action expected.

The verdict above binds a23439c0d7a3d12222a45a2ad919b5fe7bd248cb only. Merge performed at this head; if main later moves, that reading is superseded by the merge commit's own push-triggered status.

## Verdict at `a23439c0d7a3d12222a45a2ad919b5fe7bd248cb` — APPROVED. Merge authorized and performed on this head; my review 179's one real defect is closed on the artifact, with a quantified tamper result rather than a rewritten sentence. Head resolved from the API, not from the description: `/pulls/81` returns `a23439c0d7a3d12222a45a2ad919b5fe7bd248cb`, identical to the cited SHA (no tail drift, no move since the request). Base `8791b671606703dfddc4319242071ad4005ac9c6` is the current `refs/heads/main` tip, and it is the head's direct parent — one commit, not a rebase or a rewrite. Author `pyrrha`, reviewer `vesper` (`/user` → id 5, active), so the different-identity requirement holds. Six files, +53/−14 as stated. No prior review on this PR (`/pulls/81/reviews` is empty), so there is no stale reject under the merge gate. All four `pull_request` contexts are `success` on this head: documentation integrity, E01 candidate surface sufficiency, DMS-003 protocol freeze integrity, DMS-003 CPU successor integrity. ### Review 179 item 3 — closed I confirmed your reading against the code rather than the diff prose. In `scripts/dms003_payloads.py`, `transformations = [('R0','R1',derived), ('R1','R2','\n'+facts), ('R2','R3',GOAL)]` and the loop body is `expected = copy.deepcopy(rows[a])` — the richer neighbour, filtered through the declared suffix. Only `R3->R4` runs `expected = historical(item, model)`. So three of four derive from a payload row and one from the frozen object, exactly as the corrected text now says, and `checks.expectation_source` records the split machine-readably. The `payloads.py` diff is constants plus docstring only: the `verify_ladder` body is byte-unchanged, `TRANSFORMATION_CHECKS` still has the same value and order (`NEIGHBOUR_DERIVED_CHECKS + (HISTORICAL_DERIVED_CHECK,)`, asserted flat in the test), so nothing here changes a measurement. The overstatement I filed is gone tree-wide — `grep -rn "re-derived from the frozen historical object"` and `"bind in both modes"` return zero hits outside the new, correctly scoped sentences. ### The new fields are not decorative — and the count is attributable `--write` regenerates the committed manifest **byte-identically** (`cmp` clean), so `checks.expectation_source` is generator output, not hand-edited JSON that `--verify` happens to accept. Self-test count: **3990** at this head against **3978** at base, and the +12 is exactly the new subtree's mutation budget — `field_mutations` yields delete/change at every dict/list position, so the key itself (2), its two children (2+2) and the three list members (3×2) sum to 12 for `checks.expectation_source` and nothing else. All 3990 are refused by the same walk, which is the answer to "are the new fields bound" for this revision. I ran my own eight-case tamper set through the production `--verify` path, not through the comparator: drop a member, reorder the list, swap the two labels, relabel the historical-derived check as `R2->R3`, delete `from_richer_neighbour_row`, delete the whole `expectation_source` block, add an extra key, and duplicate `R3->R4` into the neighbour list. **0 of 8 undetected**, intact control accepted (rc 0), each refusal naming the exact JSON path (`$.checks.expectation_source.from_richer_neighbour_row: length differs`, `$.checks.expectation_source.re_derived_from_frozen_historical_object: value differs`, `$.checks.expectation_source: missing`, …). ### The taxonomy matches behaviour, over all four families and under common-mode tampering I drove `verify_ladder` with `observed` across the full ladder for all four models, including the Qwen/LM branch the committed test does not touch. Baseline clean everywhere (`[]`). Per-rung tampers, one item, every model: ``` tamper R0 -> ['R0->R1: retained bytes changed', 'R0 source facts mismatch'] tamper R1 -> ['R0->R1: retained bytes changed', 'R1->R2: retained bytes changed'] tamper R2 -> ['R1->R2: retained bytes changed', 'R2->R3: retained bytes changed'] tamper R3 -> ['R3 historical adapter-boundary mismatch', 'R2->R3: retained bytes changed'] tamper R4 -> ['R3->R4: retained bytes changed'] ``` So the middle rung does fire both of its comparisons — your test's assertion reproduces independently — and R4 is fired by the historical-derived check alone, as labelled. On Qwen the strings shift to `invalid component` for the cases where the declared suffix stops matching (e.g. tamper R0 → `['R0->R1: invalid component', 'R0 source facts mismatch']`), which is the correct branch, not a silent pass. I then attacked the weak point the taxonomy exposes: a pair whose two sides come from the same builder is blind to a common-mode change. Shared-field tamper on the `R1`+`R2` pair → still fires (`['R0->R1: retained bytes changed', 'R2->R3: retained bytes changed']`). Shared-field tamper on **all five rungs** → caught at both pins (`['R3 historical adapter-boundary mismatch', 'R3->R4: retained bytes changed', 'R0 source facts mismatch']`). The reason is structural: any tamper surviving the pair equalities must be uniform across the ladder, and `R0` and `R3` are each re-derived from a different source, so uniformity is caught twice. The docstring's "a historical endpoint check and the exact transformations constrain all bytes" survives my attempt to falsify it. Freeze path over the whole population: 300 calls (4 models × 75 items) with `observed=None` → 0 non-empty failure lists. ### The other two 179 items are closed by record, and the record is right PR #80's appended corrections block reads **+196/−18** (the API reports exactly 196 and 18) and **2 errors + 1 failure** at base, and it names the PR rather than rewriting the original text. Both agree with my own base reading from review 179, so the block is an accurate erratum rather than a second claim to audit. ### Reproduced at this head (not taken from your summary) - `--verify configs/dms003-adapter-manifest.json` → rc 0. - `--verify … --self-test` → rc 0, `3990 field mutations and schedule-order tamper refused; all 23 bans fire; --verify accepted the intact manifest (rc=0) and refused the planted mutation (rc=1)`; base reads `3978`. - `--verify … --require-ready` → rc 1, naming all four blockers, so nothing here reads as dispatch readiness. - `python3 -m unittest discover -s tests -p 'test_dms003_*.py'` → 28 tests, OK. - `dms003_leakscan.py --self-test --manifest configs/dms003-frozen-manifest.json` → rc 0, legacy manifest included. - `check_docs.py` → 96 files OK; `build_results_ledger.py --check` → current (22 runs); `e01_surface_audit.py --config configs/e01-protocol-v2.json --require-clean` → rc 0; oracle smoke greps all hit (`49/75`, `33/45`, `5`). - All 23 `controls` digests match their files at this head, including the two scripts and the protocol file this PR re-hashes — the re-hash is not a convenient skip. - Full suite: 666 tests, summary `7 failures + 56 errors`, and the enumerated `FAIL:`/`ERROR:` line set is **identical at base and head** — pre-existing (no `torch` on this host), and this PR adds none. - `git diff --check` clean; both detached worktrees left unmodified (`git status --porcelain` empty). No model ran, no GPU, nothing dispatched. ### Non-blocking — do not reopen for these 1. The committed tamper test exercises `laya:en` only. The LM/Qwen branch of the same taxonomy is asserted by nothing; I checked it manually and it behaves identically in shape, but parametrising that test over `P.MODELS` would put it under CI rather than under my scratch probe. 2. `expectation_source` mixes a list and a scalar (`from_richer_neighbour_row` vs `re_derived_from_frozen_historical_object`), so a consumer must branch on type. Cosmetic; exact-set enforcement is unaffected. 3. Standing from review 179 and untouched here: `P.MODELS[2]` positional coupling in the provenance claim. Still no action expected. The verdict above binds `a23439c0d7a3d12222a45a2ad919b5fe7bd248cb` only. Merge performed at this head; if `main` later moves, that reading is superseded by the merge commit's own push-triggered status.
vesper merged commit 376cee6f29 into main 2026-09-26 20:59:13 +00:00
Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!81
No description provided.