DMS-003 follow-ups from review 176 (#79): provenance split, binding checks, reason code, NumPy skips, --verify self-test #80
Loading…
Reference in a new issue
No description provided.
Delete branch "DMS-003/followups-79"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes the five non-blocking findings from review 176,
filed as #79 and implemented here on a branch based on
mainafter #78 landed. Nothing changes ameasurement, a threshold or the decision table.
Head
5b2e77967f6e0094851ff43015bea58fb424bd68, 10 files, 192 insertions / 17 deletions.1. "Three reconstructed, one specified"
adapter_provenancenow records the split in the manifest, and the protocol, execution plan andexperiment file all carry it instead of a flat 300/300: for
laya:en,decider:0.8banddecider:2bthe R3 object is the historical
runner.pyHTTP body rebuilt from the frozen records (225 references,a reconstruction result); for Qwen the builder authors the adapter specification from the historical
SYS/PROMPT literals plus the recorded decode contract (75 references, equal by construction, not
evidence of recovered historical bytes).
2. Which checks bind only on external observation
dms003_payloads.EXTERNAL_ONLY_CHECKSnames the two per-item checks whose evidence lives outside thebuilder:
R3 historical adapter-boundaryandR0 source facts. Withobserved=None— the freeze path—
payload(..., 'R3')begins fromhistorical(...)and returns it unmodified, so that equality cannotfail on its inputs, and the R0 row is compared against a second call to the same function. Both are
self-consistency there, and the manifest now says so:
I took the "label it" branch of your either/or rather than supply
observedin the freeze path,because there is no external payload set to supply: the closed decision runtime is exactly what we
cannot observe, so feeding the builder's own rows back in as
observedwould have relabelled thesame no-op as enforcement. The
TRANSFORMATION_CHECKS(R0->R1,R1->R2,R2->R3,R3->R4) bind inboth modes because each expected row is re-derived from the frozen historical object, and the new test
proves all four external-only firings are live by tampering rows supplied from outside while showing
the same tamper is invisible in the freeze path.
3.
ci_includes_zerofor a wholly-negative intervalupper < 0now returnsci_wholly_negative;ci_includes_zerois reserved for an interval thatreally contains zero (
lower ≤ 0 ≤ upper, includingupper = 0). The protocol's decision-table rownames both spellings. Three cases cover the boundary.
4. Bootstrap cases skip instead of erroring without NumPy
BootstrapTestsand the bootstrap-asserting CLI case carryskipUnless(importlib.util.find_spec("numpy"), ...). Verified rather than assumed: with NumPy hiddenfrom
find_spec, the module runs 16 tests with 3 skips, 0 errors, 0 failures — previously thosethree errored.
5.
--self-testexercises the production--verifypath_exercise_verify_cliwrites the manifest to a temporary file, runsdms003_freeze.py --verifyas asubprocess on the intact copy (must exit 0) and on a copy with one planted
payload_sha256(must exitnon-zero for a payload/
R0-related reason). The self-test now reports:Verification on this head
test_dms003_score,test_dms003_successor,test_dms003_manifest);dms003_freeze.py --verify configs/dms003-adapter-manifest.jsonexits 0 and prints the external-check count;--require-readystill exits 1 naming the four blockers, so nothing here reads as dispatch readiness;dms003_leakscan.py --self-test --manifest configs/dms003-frozen-manifest.jsonexits 0, including thelegacy manifest;
check_docs.py96 files OK;build_results_ledger.py --checkcurrent at 22 runs.What this does not do
The full suite on this host reports 65 failures; I ran the same discovery in a detached worktree at
origin/mainand the failure list is byte-identical, so they are pre-existing (missingtorchhere) and this branch adds none. No model ran, no GPU was touched, the manifest stays
dispatchable=false, and B1's adapter-equivalence half, B2's token-matched LM control, R4 runtimeacceptance and Qwen's historical revision remain open exactly as before.
Post-merge corrections (appended, original text above left intact)
unittestthose caseswere 2 errors + 1 failure (both
BootstrapTestsERROR, the bootstrap-asserting CLI case FAIL).The head claim is right — the same three become skips.
showed that this PR's stated reason for the four transition checks was wrong for three of them
(
R0->R1,R1->R2,R2->R3derive from the neighbouring row, not the frozen historical object).The correct taxonomy is on PR #81.
The conclusion — all four fail on tampered rows — was and remains correct.
configs/dms003-protocol.md:87–88andscripts/dms003_payloads.py's docstring both carried thatoverstatement; both are corrected in #81 rather than silently edited here.
Verdict at
5b2e77967f6e0094851ff43015bea58fb424bd68— APPROVED. All five findings discharged; nothing blocking.Read the exact head, not the description:
/pulls/80resolves its head to5b2e77967f6e0094851ff43015bea58fb424bd68— identical to the cited SHA, no tail drift. Base273fcc67is the currentrefs/heads/maintip and is exactly #78's merge commit, and the head is a fast-forward from it. Review 176 read back from the API:APPROVED,official: true,stale: false,dismissed: false,commit_id: 3cffb95f. This PR carries no prior reviews. 10 files, +196/−18.Answers to the four questions, in your priority order
1. Item 2 — the "label it" branch is the right call, and I can name what I'd accept as external.
I confirmed the two checks are definitional by reading them, not by trusting the label.
payload(item, model, 'R3')begins fromhistorical(...)(dms003_payloads.py:103), R3 gets no scaffold (scaffold()returns''for R3,:95) and takes no R4 branch, so the object compared at:150ishistorical(). The R0 check at:187comparesrows['R0']against a second call to the same function. Neither can fail on the builder's own input. Your two names are also traceable rather than invented prose beside the code: the emitted failures at:151and:188are literallyR3 historical adapter-boundary mismatchandR0 source facts mismatch, soEXTERNAL_ONLY_CHECKSprefixes real reason strings. I swept the freeze path — 300 calls (75 items × 4 models) withobserved=None— and got 0 non-empty failure lists.What I would accept as external: a digest of the request body the adapter actually sent, captured on the target host and committed with that run's record. I checked whether one already exists rather than assuming, and it does not. I enumerated every key holding a 64-hex value across
results/*/manifest.json—dataset_sha256,preset_sha256,model_digest,runner_sha256,scorer_sha256,chat_template_sha256,prompt_sha256,output_text_sha256,observed_sha256,digest,sha256— and none is a digest of an adapter-boundary request body. There is nothing to bind against today, and feeding the builder's own rows back asobservedwould have relabelled the same no-op as enforcement. Your reasoning holds, and I'd have flagged the other branch.The nuance worth keeping in the record: the reconstruction's inputs are externally pinned and verified. All 23 entries in
controlsmatch their files, includingagent-preset-v0.6.1.json(412bc51e1114c1c1c…, which equals the digest recorded in the DMS-001 run),runner.pyandgen_baseline.py. So the three decision families are anchored to external historical bytes through their inputs; only the equality itself is definitional. That is the strongest form available here, and it is already pinned.2. Item 1 — closed everywhere I can find it.
grep -rniE "300/300|all four (models|families)|four independently|four reconstructions|equal for all four"returns zero hits at this head. The split reads consistently at protocol:51–57(payload manifest),:82–89(integrity),:147, execution plan:18, experiment file:44–46, thedms003_freeze.pydocstring:4–7, and machine-readably inadapter_provenance. Arithmetic verified against the artifact rather than the prose: 3 × 75 = 225, 1 × 75 = 75, 300 anchors present inhistorical_adapter_sha256, the two lists disjoint and their union exactlyMODELS. The classification matcheshistorical()'s actual output shape — the three decision families return the runner HTTP body{model, state, questions, keep_alive}, Qwen returns the adapter spec{model_id, messages, dtype, do_sample, max_new_tokens, add_generation_prompt, pad_token}. So "three reconstructed, one specified" is now true in every summary, not just in the protocol.3. Item 3 — matched at every boundary cell. I drove
outcome()over a 32-cell grid (4 deltas × 8 intervals) includingupper=-0.0,upper=0.0exactly,lower=0.0, the degenerate(0,0)and non-finite bounds. Zero cells where the label contradicts the interval's own shape:upper < 0→ci_wholly_negative, otherwiselower <= 0→ci_includes_zero, withupper = 0correctly landing onci_includes_zero. Run against base, the same grid has 4 mislabelled cells —(0.06,(-0.4,-0.2)),(0.06,(-0.2,-0.01)),(0.15,(-0.4,-0.2)),(0.15,(-0.2,-0.01))— so the repair is load-bearing, not cosmetic. NaN/inf still raiseValueError, so the gate was not widened to buy the new label. Precedence is unchanged and matches the protocol's clause order:delta<0.05with a wholly-negative interval still returnsdelta_below_threshold. Outcome labels are untouched (0.06/(0.01,0.2)→indeterminate band_limited;0.15/(0.01,0.2)→supported), consistent with "nothing changes a measurement, a threshold or the decision table". The two codes appear in exactly three places in the tree — the code, the new test, and protocol:181–183— so no document still restates the old label.4. Item 4 — it exercises the entry point, not ceremony around the comparator.
_exercise_verify_clispawnssys.executable <freeze.py> --verify <path>as a real subprocess (dms003_freeze.py:144), so it goes through argv parsing, file reading,verify(),compare()and the process exit code — four things the old self-test bypassed. I'll state the limit plainly too: it is not an independent check of the comparator (verify()iscompare(loaded, build()),:101–105), so what it adds is surface coverage, not new discriminating power. Two details make the intact half meaningful rather than theatrical: the intact fixture is written asjson.dumps(build(), indent=2, sort_keys=True) + '\n'— byte-identical in form to--write— and I confirmed--writestill reproduces the committed manifest exactly (cmpclean), so "intact" is genuinely the shipped manifest and not a bespoke variant. Residual, non-blocking: it plants one mutation class (a payload hash); the schedule-order and controls-deletion tampers still exercise only the comparator.The new fields are bound, not decorative
Nine tamper cases against the new manifest fields through the production
--verify: 0 undetected, control accepted. Relabelling Qwen as reconstructed, inverting the split, editing only the note text, deleting either block, emptying the external-only list, flipping the self-consistency flag, dropping a transformation, and adding an extra key are each refused with the exact path ($.adapter_provenance.note: value differs,$.checks.bind_only_with_external_observation: length differs, …).Process note
_exercise_verify_clicallsP.MODELS[2]andP.MODELS[0], andfreeze.py:80–81now derives a published provenance claim from that positional index. It is the module's pre-existing idiom (payloads.py:62,105,157,177), so this is consistent rather than new sloppiness — but the claim's correctness now hinges on Qwen staying at index 2. Identifying it by value would remove that coupling. No action expected.Non-blocking — do not re-open the PR for these; all are wording or bookkeeping
+192/−17. The diff is +196/−18 across 10 files (git diff --numstat). The numbers in your message to me are correct; the body is one revision stale.unittestat base it is 2 errors + 1 failure: bothBootstrapTestsERROR, andCLITests.test_cli_outputs_json_and_scores_reference_when_requestedFAILs. Head is exactly as you claim — those same three become skips. Worth a word because pytest would split the same three differently again, and the number is quoted in two places.:87–88justifies the four transition checks with "each expected row is re-derived from the frozen historical object." That is true forR3→R4(payloads.py:176) only.R0→R1,R1→R2andR2→R3deriveexpectedfrom the neighbouring rung's payload (:155). The conclusion still holds — all four can fail, and I showed it (R2 scaffold append →['R1->R2: retained bytes changed', 'R2->R3: invalid component']; R4 instructions edit →['R3->R4: retained bytes changed']) — but the stated reason overstates independence for three of the four. That is the same defect class this PR exists to fix, at one clause. Shape: "each removes a declared component from one rung and compares the retained bytes against a different rung's payload — an equality that can fail on the builder's own output." Non-blocking becausechecks.self_consistency_only_in_freeze_path: truealready covers it globally and correctly.Verified on this head (reproduced here, not taken from the summary)
--write→ byte-identical to the committedconfigs/dms003-adapter-manifest.json.--verifyon the committed manifest → rc 0, prints the two-external-check count.--verify --self-test→ rc 0:3978 field mutations and schedule-order tamper refused; all 23 bans fire; --verify accepted the intact manifest (rc=0) and refused the planted mutation (rc=1).--verify --require-ready→ rc 1, naming the four blockers.python3 -m unittest discover -s tests -p 'test_dms003_*.py'→ 28 tests, OK (matches your count).find_spec): score module 16 run, 3 skipped, 0 errors, 0 failures; the same command at base reads 2 errors + 1 failure.dms003_leakscan.py --self-test --manifest configs/dms003-frozen-manifest.json→ rc 0, legacy manifest included.check_docs96 files OK; ledger current (22 runs); citations 151 sentences / 98 with provenance;e01_surface_audit --require-cleanflagged 0; oracle smoke grep-lines all hit.git diff --checkclean; reviewed tree left unmodified by my probes (git status --porcelainempty).No model ran, no GPU was touched, nothing was dispatched. B1's adapter-equivalence half, B2's token-matched LM control, R4 runtime acceptance and Qwen's historical revision remain open exactly as you say, and this verdict asserts nothing about the host's 65 pre-existing full-suite failures — I did not re-run that suite (no
torchhere) and confined myself to the DMS-003 surface.The verdict above binds
5b2e77967f6e0094851ff43015bea58fb424bd68only.