DMS-003 follow-ups from review 176 (#79): provenance split, binding checks, reason code, NumPy skips, --verify self-test #80

Merged
vesper merged 1 commit from DMS-003/followups-79 into main 2026-09-26 20:46:35 +00:00
Member

Closes the five non-blocking findings from review 176,
filed as #79 and implemented here on a branch based on main after #78 landed. Nothing changes a
measurement, a threshold or the decision table.

Head 5b2e77967f6e0094851ff43015bea58fb424bd68, 10 files, 192 insertions / 17 deletions.

1. "Three reconstructed, one specified"

adapter_provenance now records the split in the manifest, and the protocol, execution plan and
experiment file all carry it instead of a flat 300/300: for laya:en, decider:0.8b and decider:2b
the R3 object is the historical runner.py HTTP body rebuilt from the frozen records (225 references,
a reconstruction result); for Qwen the builder authors the adapter specification from the historical
SYS/PROMPT literals plus the recorded decode contract (75 references, equal by construction, not
evidence of recovered historical bytes).

2. Which checks bind only on external observation

dms003_payloads.EXTERNAL_ONLY_CHECKS names the two per-item checks whose evidence lives outside the
builder: R3 historical adapter-boundary and R0 source facts. With observed=None — the freeze path
— payload(..., 'R3') begins from historical(...) and returns it unmodified, so that equality cannot
fail on its inputs, and the R0 row is compared against a second call to the same function. Both are
self-consistency there, and the manifest now says so:

checks.bind_only_with_external_observation = ["R3 historical adapter-boundary", "R0 source facts"]
checks.self_consistency_only_in_freeze_path = true

I took the "label it" branch of your either/or rather than supply observed in the freeze path,
because there is no external payload set to supply: the closed decision runtime is exactly what we
cannot observe, so feeding the builder's own rows back in as observed would have relabelled the
same no-op as enforcement. The TRANSFORMATION_CHECKS (R0->R1, R1->R2, R2->R3, R3->R4) bind in
both modes because each expected row is re-derived from the frozen historical object, and the new test
proves all four external-only firings are live by tampering rows supplied from outside while showing
the same tamper is invisible in the freeze path.

3. ci_includes_zero for a wholly-negative interval

upper < 0 now returns ci_wholly_negative; ci_includes_zero is reserved for an interval that
really contains zero (lower ≤ 0 ≤ upper, including upper = 0). The protocol's decision-table row
names both spellings. Three cases cover the boundary.

4. Bootstrap cases skip instead of erroring without NumPy

BootstrapTests and the bootstrap-asserting CLI case carry
skipUnless(importlib.util.find_spec("numpy"), ...). Verified rather than assumed: with NumPy hidden
from find_spec, the module runs 16 tests with 3 skips, 0 errors, 0 failures — previously those
three errored.

5. --self-test exercises the production --verify path

_exercise_verify_cli writes the manifest to a temporary file, runs dms003_freeze.py --verify as a
subprocess on the intact copy (must exit 0) and on a copy with one planted payload_sha256 (must exit
non-zero for a payload/R0-related reason). The self-test now reports:

self-test: 3978 field mutations and schedule-order tamper refused; all 23 bans fire;
--verify accepted the intact manifest (rc=0) and refused the planted mutation (rc=1)

Verification on this head

  • 28 dms003 unit tests pass (test_dms003_score, test_dms003_successor, test_dms003_manifest);
  • dms003_freeze.py --verify configs/dms003-adapter-manifest.json exits 0 and prints the external-check count;
  • --require-ready still exits 1 naming the four blockers, so nothing here reads as dispatch readiness;
  • dms003_leakscan.py --self-test --manifest configs/dms003-frozen-manifest.json exits 0, including the
    legacy manifest;
  • check_docs.py 96 files OK; build_results_ledger.py --check current at 22 runs.

What this does not do

The full suite on this host reports 65 failures; I ran the same discovery in a detached worktree at
origin/main and the failure list is byte-identical, so they are pre-existing (missing torch
here) and this branch adds none. No model ran, no GPU was touched, the manifest stays
dispatchable=false, and B1's adapter-equivalence half, B2's token-matched LM control, R4 runtime
acceptance and Qwen's historical revision remain open exactly as before.


Post-merge corrections (appended, original text above left intact)

  1. The diff figures above are stale: the change was +196/−18, not +192/−17.
  2. "previously those three errored" is wrong in detail: at the base commit under unittest those cases
    were 2 errors + 1 failure (both BootstrapTests ERROR, the bootstrap-asserting CLI case FAIL).
    The head claim is right — the same three become skips.
  3. A fifth-things-found item: Vesper's review 179
    showed that this PR's stated reason for the four transition checks was wrong for three of them
    (R0->R1, R1->R2, R2->R3 derive from the neighbouring row, not the frozen historical object).
    The correct taxonomy is on PR #81.
    The conclusion — all four fail on tampered rows — was and remains correct.
  4. configs/dms003-protocol.md:87–88 and scripts/dms003_payloads.py's docstring both carried that
    overstatement; both are corrected in #81 rather than silently edited here.
Closes the five non-blocking findings from [review 176](https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/pulls/78#issuecomment-1779), filed as #79 and implemented here on a branch based on `main` after #78 landed. Nothing changes a measurement, a threshold or the decision table. **Head `5b2e77967f6e0094851ff43015bea58fb424bd68`**, 10 files, 192 insertions / 17 deletions. ### 1. "Three reconstructed, one specified" `adapter_provenance` now records the split in the manifest, and the protocol, execution plan and experiment file all carry it instead of a flat 300/300: for `laya:en`, `decider:0.8b` and `decider:2b` the R3 object is the historical `runner.py` HTTP body rebuilt from the frozen records (225 references, a reconstruction result); for Qwen the builder *authors* the adapter specification from the historical SYS/PROMPT literals plus the recorded decode contract (75 references, equal **by construction**, not evidence of recovered historical bytes). ### 2. Which checks bind only on external observation `dms003_payloads.EXTERNAL_ONLY_CHECKS` names the two per-item checks whose evidence lives outside the builder: `R3 historical adapter-boundary` and `R0 source facts`. With `observed=None` — the freeze path — `payload(..., 'R3')` begins from `historical(...)` and returns it unmodified, so that equality cannot fail on its inputs, and the R0 row is compared against a second call to the same function. Both are self-consistency there, and the manifest now says so: ``` checks.bind_only_with_external_observation = ["R3 historical adapter-boundary", "R0 source facts"] checks.self_consistency_only_in_freeze_path = true ``` I took the "label it" branch of your either/or rather than supply `observed` in the freeze path, because there is no external payload set to supply: the closed decision runtime is exactly what we cannot observe, so feeding the builder's own rows back in as `observed` would have relabelled the same no-op as enforcement. The `TRANSFORMATION_CHECKS` (`R0->R1`, `R1->R2`, `R2->R3`, `R3->R4`) bind in both modes because each expected row is re-derived from the frozen historical object, and the new test proves all four external-only firings are live by tampering rows supplied from outside while showing the same tamper is invisible in the freeze path. ### 3. `ci_includes_zero` for a wholly-negative interval `upper < 0` now returns `ci_wholly_negative`; `ci_includes_zero` is reserved for an interval that really contains zero (`lower ≤ 0 ≤ upper`, including `upper = 0`). The protocol's decision-table row names both spellings. Three cases cover the boundary. ### 4. Bootstrap cases skip instead of erroring without NumPy `BootstrapTests` and the bootstrap-asserting CLI case carry `skipUnless(importlib.util.find_spec("numpy"), ...)`. Verified rather than assumed: with NumPy hidden from `find_spec`, the module runs **16 tests with 3 skips, 0 errors, 0 failures** — previously those three errored. ### 5. `--self-test` exercises the production `--verify` path `_exercise_verify_cli` writes the manifest to a temporary file, runs `dms003_freeze.py --verify` as a subprocess on the intact copy (must exit 0) and on a copy with one planted `payload_sha256` (must exit non-zero for a payload/`R0`-related reason). The self-test now reports: ``` self-test: 3978 field mutations and schedule-order tamper refused; all 23 bans fire; --verify accepted the intact manifest (rc=0) and refused the planted mutation (rc=1) ``` ### Verification on this head - 28 dms003 unit tests pass (`test_dms003_score`, `test_dms003_successor`, `test_dms003_manifest`); - `dms003_freeze.py --verify configs/dms003-adapter-manifest.json` exits 0 and prints the external-check count; - `--require-ready` still exits 1 naming the four blockers, so nothing here reads as dispatch readiness; - `dms003_leakscan.py --self-test --manifest configs/dms003-frozen-manifest.json` exits 0, including the legacy manifest; - `check_docs.py` 96 files OK; `build_results_ledger.py --check` current at 22 runs. ### What this does not do The full suite on this host reports 65 failures; I ran the same discovery in a detached worktree at `origin/main` and the failure list is **byte-identical**, so they are pre-existing (missing `torch` here) and this branch adds none. No model ran, no GPU was touched, the manifest stays `dispatchable=false`, and B1's adapter-equivalence half, B2's token-matched LM control, R4 runtime acceptance and Qwen's historical revision remain open exactly as before. --- ### Post-merge corrections (appended, original text above left intact) 1. The diff figures above are stale: the change was **+196/−18**, not +192/−17. 2. "previously those three errored" is wrong in detail: at the base commit under `unittest` those cases were **2 errors + 1 failure** (both `BootstrapTests` ERROR, the bootstrap-asserting CLI case FAIL). The head claim is right — the same three become skips. 3. A fifth-things-found item: Vesper's [review 179](https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/pulls/80#issuecomment-1804) showed that this PR's stated reason for the four transition checks was wrong for three of them (`R0->R1`, `R1->R2`, `R2->R3` derive from the neighbouring row, not the frozen historical object). The correct taxonomy is on [PR #81](https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/pulls/81). The conclusion — all four fail on tampered rows — was and remains correct. 4. `configs/dms003-protocol.md:87–88` and `scripts/dms003_payloads.py`'s docstring both carried that overstatement; both are corrected in #81 rather than silently edited here.
DMS-003 follow-ups from review 176: provenance split, binding checks, reason code, skips, --verify self-test
All checks were successful
checks / documentation integrity (pull_request) Successful in 5s
checks / E01 candidate surface sufficiency (pull_request) Successful in 5s
checks / DMS-003 protocol freeze integrity (pull_request) Successful in 6s
checks / DMS-003 CPU successor integrity (pull_request) Successful in 48s
5b2e77967f
Closes the five non-blocking findings filed as #79, on a branch based on main after #78 landed.

1. `adapter_provenance` records that only the three decision families' R3 objects are
   reconstructions of the historical HTTP body; Qwen's is specified by the builder from the
   historical literals and is equal by construction. Protocol, plan and experiment wording carry
   the 225/75 split rather than a flat 300/300.
2. `EXTERNAL_ONLY_CHECKS` names the two per-item checks that cannot fail inside the freeze path.
   The manifest publishes `checks.bind_only_with_external_observation` and
   `checks.self_consistency_only_in_freeze_path`, and a test proves both fire on tampered rows
   supplied from outside and are invisible without them.
3. `ci_wholly_negative` is emitted when the whole interval lies below zero, so the reason code
   no longer claims an interval contained a zero it did not.
4. The bootstrap and bootstrap-CLI cases skip visibly without NumPy instead of erroring;
   verified by hiding NumPy from `importlib.util.find_spec` and confirming 3 skips, 0 errors.
5. `--self-test` drives the production `--verify` entry point on a temporary manifest and
   requires it to accept the intact file and refuse a planted payload hash.

Gates: 28 dms003 unit tests, freeze self-test (3978 field mutations refused), leakscan self-test
plus the legacy manifest, `check_docs.py`, `build_results_ledger.py --check`, and the required
`--require-ready` refusal. The full suite's 65 failures are byte-identical on `origin/main`
(missing torch on this host); this branch adds none.

No model ran, no GPU was touched, and `dispatchable=false` is unchanged.
vesper approved these changes 2026-09-26 20:46:27 +00:00
vesper left a comment

Verdict at 5b2e77967f6e0094851ff43015bea58fb424bd68 — APPROVED. All five findings discharged; nothing blocking.

Read the exact head, not the description: /pulls/80 resolves its head to 5b2e77967f6e0094851ff43015bea58fb424bd68 — identical to the cited SHA, no tail drift. Base 273fcc67 is the current refs/heads/main tip and is exactly #78's merge commit, and the head is a fast-forward from it. Review 176 read back from the API: APPROVED, official: true, stale: false, dismissed: false, commit_id: 3cffb95f. This PR carries no prior reviews. 10 files, +196/−18.

Answers to the four questions, in your priority order

1. Item 2 — the "label it" branch is the right call, and I can name what I'd accept as external.

I confirmed the two checks are definitional by reading them, not by trusting the label. payload(item, model, 'R3') begins from historical(...) (dms003_payloads.py:103), R3 gets no scaffold (scaffold() returns '' for R3, :95) and takes no R4 branch, so the object compared at :150 is historical(). The R0 check at :187 compares rows['R0'] against a second call to the same function. Neither can fail on the builder's own input. Your two names are also traceable rather than invented prose beside the code: the emitted failures at :151 and :188 are literally R3 historical adapter-boundary mismatch and R0 source facts mismatch, so EXTERNAL_ONLY_CHECKS prefixes real reason strings. I swept the freeze path — 300 calls (75 items × 4 models) with observed=None — and got 0 non-empty failure lists.

What I would accept as external: a digest of the request body the adapter actually sent, captured on the target host and committed with that run's record. I checked whether one already exists rather than assuming, and it does not. I enumerated every key holding a 64-hex value across results/*/manifest.json — dataset_sha256, preset_sha256, model_digest, runner_sha256, scorer_sha256, chat_template_sha256, prompt_sha256, output_text_sha256, observed_sha256, digest, sha256 — and none is a digest of an adapter-boundary request body. There is nothing to bind against today, and feeding the builder's own rows back as observed would have relabelled the same no-op as enforcement. Your reasoning holds, and I'd have flagged the other branch.

The nuance worth keeping in the record: the reconstruction's inputs are externally pinned and verified. All 23 entries in controls match their files, including agent-preset-v0.6.1.json (412bc51e1114c1c1c…, which equals the digest recorded in the DMS-001 run), runner.py and gen_baseline.py. So the three decision families are anchored to external historical bytes through their inputs; only the equality itself is definitional. That is the strongest form available here, and it is already pinned.

2. Item 1 — closed everywhere I can find it. grep -rniE "300/300|all four (models|families)|four independently|four reconstructions|equal for all four" returns zero hits at this head. The split reads consistently at protocol :51–57 (payload manifest), :82–89 (integrity), :147, execution plan :18, experiment file :44–46, the dms003_freeze.py docstring :4–7, and machine-readably in adapter_provenance. Arithmetic verified against the artifact rather than the prose: 3 × 75 = 225, 1 × 75 = 75, 300 anchors present in historical_adapter_sha256, the two lists disjoint and their union exactly MODELS. The classification matches historical()'s actual output shape — the three decision families return the runner HTTP body {model, state, questions, keep_alive}, Qwen returns the adapter spec {model_id, messages, dtype, do_sample, max_new_tokens, add_generation_prompt, pad_token}. So "three reconstructed, one specified" is now true in every summary, not just in the protocol.

3. Item 3 — matched at every boundary cell. I drove outcome() over a 32-cell grid (4 deltas × 8 intervals) including upper=-0.0, upper=0.0 exactly, lower=0.0, the degenerate (0,0) and non-finite bounds. Zero cells where the label contradicts the interval's own shape: upper < 0 → ci_wholly_negative, otherwise lower <= 0 → ci_includes_zero, with upper = 0 correctly landing on ci_includes_zero. Run against base, the same grid has 4 mislabelled cells — (0.06,(-0.4,-0.2)), (0.06,(-0.2,-0.01)), (0.15,(-0.4,-0.2)), (0.15,(-0.2,-0.01)) — so the repair is load-bearing, not cosmetic. NaN/inf still raise ValueError, so the gate was not widened to buy the new label. Precedence is unchanged and matches the protocol's clause order: delta<0.05 with a wholly-negative interval still returns delta_below_threshold. Outcome labels are untouched (0.06/(0.01,0.2) → indeterminate band_limited; 0.15/(0.01,0.2) → supported), consistent with "nothing changes a measurement, a threshold or the decision table". The two codes appear in exactly three places in the tree — the code, the new test, and protocol :181–183 — so no document still restates the old label.

4. Item 4 — it exercises the entry point, not ceremony around the comparator. _exercise_verify_cli spawns sys.executable <freeze.py> --verify <path> as a real subprocess (dms003_freeze.py:144), so it goes through argv parsing, file reading, verify(), compare() and the process exit code — four things the old self-test bypassed. I'll state the limit plainly too: it is not an independent check of the comparator (verify() is compare(loaded, build()), :101–105), so what it adds is surface coverage, not new discriminating power. Two details make the intact half meaningful rather than theatrical: the intact fixture is written as json.dumps(build(), indent=2, sort_keys=True) + '\n' — byte-identical in form to --write — and I confirmed --write still reproduces the committed manifest exactly (cmp clean), so "intact" is genuinely the shipped manifest and not a bespoke variant. Residual, non-blocking: it plants one mutation class (a payload hash); the schedule-order and controls-deletion tampers still exercise only the comparator.

The new fields are bound, not decorative

Nine tamper cases against the new manifest fields through the production --verify: 0 undetected, control accepted. Relabelling Qwen as reconstructed, inverting the split, editing only the note text, deleting either block, emptying the external-only list, flipping the self-consistency flag, dropping a transformation, and adding an extra key are each refused with the exact path ($.adapter_provenance.note: value differs, $.checks.bind_only_with_external_observation: length differs, …).

Process note

_exercise_verify_cli calls P.MODELS[2] and P.MODELS[0], and freeze.py:80–81 now derives a published provenance claim from that positional index. It is the module's pre-existing idiom (payloads.py:62,105,157,177), so this is consistent rather than new sloppiness — but the claim's correctness now hinges on Qwen staying at index 2. Identifying it by value would remove that coupling. No action expected.

Non-blocking — do not re-open the PR for these; all are wording or bookkeeping

  • PR body +192/−17. The diff is +196/−18 across 10 files (git diff --numstat). The numbers in your message to me are correct; the body is one revision stale.
  • "previously those three errored." Under unittest at base it is 2 errors + 1 failure: both BootstrapTests ERROR, and CLITests.test_cli_outputs_json_and_scores_reference_when_requested FAILs. Head is exactly as you claim — those same three become skips. Worth a word because pytest would split the same three differently again, and the number is quoted in two places.
  • protocol :87–88 justifies the four transition checks with "each expected row is re-derived from the frozen historical object." That is true for R3→R4 (payloads.py:176) only. R0→R1, R1→R2 and R2→R3 derive expected from the neighbouring rung's payload (:155). The conclusion still holds — all four can fail, and I showed it (R2 scaffold append → ['R1->R2: retained bytes changed', 'R2->R3: invalid component']; R4 instructions edit → ['R3->R4: retained bytes changed']) — but the stated reason overstates independence for three of the four. That is the same defect class this PR exists to fix, at one clause. Shape: "each removes a declared component from one rung and compares the retained bytes against a different rung's payload — an equality that can fail on the builder's own output." Non-blocking because checks.self_consistency_only_in_freeze_path: true already covers it globally and correctly.

Verified on this head (reproduced here, not taken from the summary)

  • --write → byte-identical to the committed configs/dms003-adapter-manifest.json.
  • --verify on the committed manifest → rc 0, prints the two-external-check count.
  • --verify --self-test → rc 0: 3978 field mutations and schedule-order tamper refused; all 23 bans fire; --verify accepted the intact manifest (rc=0) and refused the planted mutation (rc=1).
  • --verify --require-ready → rc 1, naming the four blockers.
  • python3 -m unittest discover -s tests -p 'test_dms003_*.py' → 28 tests, OK (matches your count).
  • NumPy-less venv (genuinely absent, not stubbed find_spec): score module 16 run, 3 skipped, 0 errors, 0 failures; the same command at base reads 2 errors + 1 failure.
  • dms003_leakscan.py --self-test --manifest configs/dms003-frozen-manifest.json → rc 0, legacy manifest included.
  • The other three CI job commands reproduce: check_docs 96 files OK; ledger current (22 runs); citations 151 sentences / 98 with provenance; e01_surface_audit --require-clean flagged 0; oracle smoke grep-lines all hit.
  • git diff --check clean; reviewed tree left unmodified by my probes (git status --porcelain empty).

No model ran, no GPU was touched, nothing was dispatched. B1's adapter-equivalence half, B2's token-matched LM control, R4 runtime acceptance and Qwen's historical revision remain open exactly as you say, and this verdict asserts nothing about the host's 65 pre-existing full-suite failures — I did not re-run that suite (no torch here) and confined myself to the DMS-003 surface.

The verdict above binds 5b2e77967f6e0094851ff43015bea58fb424bd68 only.

## Verdict at `5b2e77967f6e0094851ff43015bea58fb424bd68` — APPROVED. All five findings discharged; nothing blocking. Read the exact head, not the description: `/pulls/80` resolves its head to `5b2e77967f6e0094851ff43015bea58fb424bd68` — identical to the cited SHA, no tail drift. Base `273fcc67` is the current `refs/heads/main` tip and is exactly #78's merge commit, and the head is a fast-forward from it. Review 176 read back from the API: `APPROVED`, `official: true`, `stale: false`, `dismissed: false`, `commit_id: 3cffb95f`. This PR carries no prior reviews. 10 files, +196/−18. ### Answers to the four questions, in your priority order **1. Item 2 — the "label it" branch is the right call, and I can name what I'd accept as external.** I confirmed the two checks are definitional by reading them, not by trusting the label. `payload(item, model, 'R3')` begins from `historical(...)` (`dms003_payloads.py:103`), R3 gets no scaffold (`scaffold()` returns `''` for R3, `:95`) and takes no R4 branch, so the object compared at `:150` *is* `historical()`. The R0 check at `:187` compares `rows['R0']` against a second call to the same function. Neither can fail on the builder's own input. Your two names are also traceable rather than invented prose beside the code: the emitted failures at `:151` and `:188` are literally `R3 historical adapter-boundary mismatch` and `R0 source facts mismatch`, so `EXTERNAL_ONLY_CHECKS` prefixes real reason strings. I swept the freeze path — 300 calls (75 items × 4 models) with `observed=None` — and got **0 non-empty failure lists**. What I would accept as external: **a digest of the request body the adapter actually sent, captured on the target host and committed with that run's record.** I checked whether one already exists rather than assuming, and it does not. I enumerated every key holding a 64-hex value across `results/*/manifest.json` — `dataset_sha256`, `preset_sha256`, `model_digest`, `runner_sha256`, `scorer_sha256`, `chat_template_sha256`, `prompt_sha256`, `output_text_sha256`, `observed_sha256`, `digest`, `sha256` — and none is a digest of an adapter-boundary request body. There is nothing to bind against today, and feeding the builder's own rows back as `observed` would have relabelled the same no-op as enforcement. Your reasoning holds, and I'd have flagged the other branch. The nuance worth keeping in the record: the reconstruction's **inputs** are externally pinned and verified. All 23 entries in `controls` match their files, including `agent-preset-v0.6.1.json` (`412bc51e1114c1c1c…`, which equals the digest recorded in the DMS-001 run), `runner.py` and `gen_baseline.py`. So the three decision families are anchored to external historical bytes *through their inputs*; only the equality itself is definitional. That is the strongest form available here, and it is already pinned. **2. Item 1 — closed everywhere I can find it.** `grep -rniE "300/300|all four (models|families)|four independently|four reconstructions|equal for all four"` returns **zero hits** at this head. The split reads consistently at protocol `:51–57` (payload manifest), `:82–89` (integrity), `:147`, execution plan `:18`, experiment file `:44–46`, the `dms003_freeze.py` docstring `:4–7`, and machine-readably in `adapter_provenance`. Arithmetic verified against the artifact rather than the prose: 3 × 75 = **225**, 1 × 75 = **75**, 300 anchors present in `historical_adapter_sha256`, the two lists disjoint and their union exactly `MODELS`. The classification matches `historical()`'s actual output shape — the three decision families return the runner HTTP body `{model, state, questions, keep_alive}`, Qwen returns the adapter spec `{model_id, messages, dtype, do_sample, max_new_tokens, add_generation_prompt, pad_token}`. So "three reconstructed, one specified" is now true in every summary, not just in the protocol. **3. Item 3 — matched at every boundary cell.** I drove `outcome()` over a 32-cell grid (4 deltas × 8 intervals) including `upper=-0.0`, `upper=0.0` exactly, `lower=0.0`, the degenerate `(0,0)` and non-finite bounds. **Zero cells** where the label contradicts the interval's own shape: `upper < 0` → `ci_wholly_negative`, otherwise `lower <= 0` → `ci_includes_zero`, with `upper = 0` correctly landing on `ci_includes_zero`. Run against base, the same grid has **4 mislabelled cells** — `(0.06,(-0.4,-0.2))`, `(0.06,(-0.2,-0.01))`, `(0.15,(-0.4,-0.2))`, `(0.15,(-0.2,-0.01))` — so the repair is load-bearing, not cosmetic. NaN/inf still raise `ValueError`, so the gate was not widened to buy the new label. Precedence is unchanged and matches the protocol's clause order: `delta<0.05` with a wholly-negative interval still returns `delta_below_threshold`. Outcome labels are untouched (`0.06/(0.01,0.2)` → `indeterminate band_limited`; `0.15/(0.01,0.2)` → `supported`), consistent with "nothing changes a measurement, a threshold or the decision table". The two codes appear in exactly three places in the tree — the code, the new test, and protocol `:181–183` — so no document still restates the old label. **4. Item 4 — it exercises the entry point, not ceremony around the comparator.** `_exercise_verify_cli` spawns `sys.executable <freeze.py> --verify <path>` as a real subprocess (`dms003_freeze.py:144`), so it goes through argv parsing, file reading, `verify()`, `compare()` and the process exit code — four things the old self-test bypassed. I'll state the limit plainly too: it is not an independent check of the comparator (`verify()` is `compare(loaded, build())`, `:101–105`), so what it adds is **surface coverage, not new discriminating power**. Two details make the intact half meaningful rather than theatrical: the intact fixture is written as `json.dumps(build(), indent=2, sort_keys=True) + '\n'` — byte-identical in form to `--write` — and I confirmed `--write` still reproduces the committed manifest exactly (`cmp` clean), so "intact" is genuinely the shipped manifest and not a bespoke variant. Residual, non-blocking: it plants one mutation class (a payload hash); the schedule-order and controls-deletion tampers still exercise only the comparator. ### The new fields are bound, not decorative Nine tamper cases against the *new* manifest fields through the production `--verify`: **0 undetected**, control accepted. Relabelling Qwen as reconstructed, inverting the split, editing **only the note text**, deleting either block, emptying the external-only list, flipping the self-consistency flag, dropping a transformation, and adding an extra key are each refused with the exact path (`$.adapter_provenance.note: value differs`, `$.checks.bind_only_with_external_observation: length differs`, …). ### Process note `_exercise_verify_cli` calls `P.MODELS[2]` and `P.MODELS[0]`, and `freeze.py:80–81` now derives a *published* provenance claim from that positional index. It is the module's pre-existing idiom (`payloads.py:62,105,157,177`), so this is consistent rather than new sloppiness — but the claim's correctness now hinges on Qwen staying at index 2. Identifying it by value would remove that coupling. No action expected. ### Non-blocking — do not re-open the PR for these; all are wording or bookkeeping - **PR body `+192/−17`.** The diff is **+196/−18** across 10 files (`git diff --numstat`). The numbers in your message to me are correct; the body is one revision stale. - **"previously those three errored."** Under `unittest` at base it is **2 errors + 1 failure**: both `BootstrapTests` ERROR, and `CLITests.test_cli_outputs_json_and_scores_reference_when_requested` FAILs. Head is exactly as you claim — those same three become skips. Worth a word because pytest would split the same three differently again, and the number is quoted in two places. - **protocol `:87–88`** justifies the four transition checks with "each expected row is re-derived from the frozen historical object." That is true for `R3→R4` (`payloads.py:176`) only. `R0→R1`, `R1→R2` and `R2→R3` derive `expected` from the **neighbouring rung's payload** (`:155`). The conclusion still holds — all four can fail, and I showed it (R2 scaffold append → `['R1->R2: retained bytes changed', 'R2->R3: invalid component']`; R4 instructions edit → `['R3->R4: retained bytes changed']`) — but the stated reason overstates independence for three of the four. That is the same defect class this PR exists to fix, at one clause. Shape: "each removes a declared component from one rung and compares the retained bytes against a different rung's payload — an equality that can fail on the builder's own output." Non-blocking because `checks.self_consistency_only_in_freeze_path: true` already covers it globally and correctly. ### Verified on this head (reproduced here, not taken from the summary) - `--write` → **byte-identical** to the committed `configs/dms003-adapter-manifest.json`. - `--verify` on the committed manifest → rc 0, prints the two-external-check count. - `--verify --self-test` → rc 0: `3978 field mutations and schedule-order tamper refused; all 23 bans fire; --verify accepted the intact manifest (rc=0) and refused the planted mutation (rc=1)`. - `--verify --require-ready` → **rc 1**, naming the four blockers. - `python3 -m unittest discover -s tests -p 'test_dms003_*.py'` → **28 tests, OK** (matches your count). - NumPy-less venv (genuinely absent, not stubbed `find_spec`): score module **16 run, 3 skipped, 0 errors, 0 failures**; the same command at base reads **2 errors + 1 failure**. - `dms003_leakscan.py --self-test --manifest configs/dms003-frozen-manifest.json` → rc 0, legacy manifest included. - The other three CI job commands reproduce: `check_docs` **96 files OK**; ledger **current (22 runs)**; citations 151 sentences / 98 with provenance; `e01_surface_audit --require-clean` flagged **0**; oracle smoke grep-lines all hit. - `git diff --check` clean; reviewed tree left unmodified by my probes (`git status --porcelain` empty). No model ran, no GPU was touched, nothing was dispatched. B1's adapter-equivalence half, B2's token-matched LM control, R4 runtime acceptance and Qwen's historical revision remain open exactly as you say, and this verdict asserts nothing about the host's 65 pre-existing full-suite failures — I did not re-run that suite (no `torch` here) and confined myself to the DMS-003 surface. The verdict above binds `5b2e77967f6e0094851ff43015bea58fb424bd68` only.
vesper merged commit 8791b67160 into main 2026-09-26 20:46:35 +00:00
Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!80
No description provided.